The Rays lost their third straight and have dropped five out of six.
Trevor Cahill (9-9) gave up four hits over 7 1/3 innings and Oakland took a shutout into the ninth before the Rays scored four times, two of the runs being unearned.
Oakland piled up 16 hits, with Matsui providing three of them. He doubled in a run in the third and the Athletics then sent a dozen men to the plate in the fourth to turn the game into a rout.
Scott Sizemore tripled in two runs during the fourth and a single by Jemile Weeks brought in two more before Matsui hit his three-run blast to right.
Matsui added an RBI single in the fifth.
Tampa Bay starter James Shields (9-9) was charged with 10 runs on 10 hits in four innings.