The Wildcats will next play either Mississippi Valley State or 14th-ranked Oklahoma in the second round.
Bryant Barr had 17 points and Stephen Rossiter 14 for Davidson, which made a run all the way to the Sweet 16 before falling to eventual NCAA Champion Kansas last season.
Julius Wells had 16 points and Kyle Swanston scored 14 for James Madison (0-1), which has not had a winning season since 1999-2000.
Millions of Getty images now available for free via embed tool
NBC reportedly holds celebs hostage to Jimmy Fallon's show