Taylor hauled down 10 rebounds and Semaj Christon added 13 points for the Musketeers (16-11, 8-5 A-10).
Xavier's victory was its second in three contests.
Memphis (24-4, 13-0 C-USA) got 14 points and seven boards from Geron Johnson, but it wasn't enough to prevent their first loss since Dec. 15.
Joe Jackson reached double figures with 11 points for Memphis.
Swim Week Miami Beach 2014 [PHOTOS]