Malisse posted the lone break of the third set for a 7-5, 4-6, 6-4 win over Mayer. It was an upset in that Mayer is ranked No. 21 and Malisse comes in 67th, but Malisse is now 3-0 lifetime versus Mayer.
Malisse was forced into break-point situations five times in each the first two sets but had none to deal with in the third set.
He goes into the quarterfinals against qualifier Attila Balazs, who earned a 6-3, 6-4 second-round win over Potito Starace. Balazs finished off three of his four break opportunities.
The only other match Wednesday was a battle of qualifiers, with Daniel Brands topping Jurgen Zopp 7-6 (7-3), 6-3. Brands lost just 18 points on serve.