Taylor hauled down 10 rebounds and Semaj Christon added 13 points for the Musketeers (16-11, 8-5 A-10).
Xavier's victory was its second in three contests.
Memphis (24-4, 13-0 C-USA) got 14 points and seven boards from Geron Johnson, but it wasn't enough to prevent their first loss since Dec. 15.
Joe Jackson reached double figures with 11 points for Memphis.
Wisconsin business offering 'therapeutic cuddling' forced to close
Puzzle-maker slips 'Murdoch Is Evil' into Rupert Murdoch's Sunday Telegraph