Guyer was brought up from Class AAA Durham to take the roster spot left vacant when pitcher Jeff Niemann went on the 15-day disabled list. He did not take the bat off his shoulder while working a 3-1 count against Zach Britton and then hit the next pitch over the wall in left to give the Rays a 3-0 lead.
Tampa Bay got a two-run home run from Johnny Damon in the ninth en route to its second straight win.
James Shields (3-1) got the victory, giving up a run on three hits in 7 1/3 innings. He did not allow a hit until the fourth.
Britton (5-2) surrendered three runs in 5 1/3 innings.
The Rays were swept by Baltimore in a three-game series early in the season as part of an 0-6 start. Tampa Bay has rallied to stand only one game behind the division-leading New York Yankees in the American League East.
Notable deaths of 2014 [PHOTOS]
Julius Randle breaks leg during NBA debut