Holloway hit only 1-of-7 shots from the field, but his 16 points kept the Musketeers unbeaten over their first seven games of the season.
Xavier was one of the three unbeaten teams in the country who kept their perfect records Wednesday, joining Louisville and Illinois. There are 14 unbeaten clubs left in major college basketball.
The Musketeers led by 16 at halftime, but Butler charged back within four with 13 minutes to play. After Chase Stigall missed a 3-point attempt that would have closed the deficit to a single point, Holloway was fouled and hit two free throws.
Xavier then pulled away with Mark Lyons also contributing 16 points and Dezmine Wells scoring 13.
Butler (4-5) was paced by 10 points each from Stigall and Roosevelt Jones.