The Rays had won the first two games of the season-opening series.
Baltimore's big inning gave it a 5-2 lead and the Rays closed the gap with single runs in the seventh and eighth.
Mike Gonzalez then pitched out of a bases-loaded jam in the ninth to earn his first save and preserve the victory for Brian Matusz.
Matusz (1-0), a rookie, gave up two runs and two hits with five walks in five innings. He struck out seven.
Tampa Bay was trying for its first 3-0 start since 2002. Jeff Niemann started for Tampa Bay, but left the game after 1 1/3 innings when he was hit on the right shoulder with a batted ball.
Mike Edstrom (0-1) took the loss after giving up three runs.
Celebrity Breakups and divorces of 2014 [PHOTOS]