Taylor hauled down 10 rebounds and Semaj Christon added 13 points for the Musketeers (16-11, 8-5 A-10).
Xavier's victory was its second in three contests.
Memphis (24-4, 13-0 C-USA) got 14 points and seven boards from Geron Johnson, but it wasn't enough to prevent their first loss since Dec. 15.
Joe Jackson reached double figures with 11 points for Memphis.
LGBT community has 'bullied the American people': Bachmann
Attkisson leaves CBS News, reportedly over network's 'liberal bias'