Second-ranked McIlroy had set himself up for the top ranking when he staged a furious rally to best world No. 3 Lee Westwood in Sunday morning's semifinal.
He needed only to down Mahan in the afternoon final to claim the top spot, but instead fell victim to the American, who earned $1.4 million for his fourth career PGA win.
Mahan will now likely move to No. 9 in Monday's new world rankings. He reached the championship round by eliminating Mark Wilson 2 & 1 in Sunday's other semifinal.
In McIlroy's victory over Westwood, the Northern Ireland native overcame a three-hole deficit after four holes, winning four of the next five to charge back into a 1-up lead. McIlroy moved 3-up with a birdie at 13, taking advantage of an errant Westwood tee shot.
The latter drained a 20-foot eagle putt at the par-four 15th, but later stumbled on No. 17 and conceded the stirring comeback win to McIlroy.
Boston schools pull out free condoms over wrapping complaints
Aaron Carter is still in love with Hilary Duff