Jumbo Diaz (1-3) got the win thanks to a scoreless ninth, and Alex Colome closed it in the 10th to earn his 21st save despite allowing a solo homer to Mark Trumbo.
The score was tied at 2 when Adam Jones lined a double to right off Rays starter Jake Faria to start the seventh. Jones went to third when Souza mishandled the ball in the corner and scored on a Trumbo single.
Tampa Bay then tied the game at 3 thanks to a two-out rally in the ninth against Baltimore closer Brad Brach.
Pinch-hitter Shane Peterson walked, went to second on a balk. Pinch-runner Peter Bourjos moved to third on a wild pitch and then scored on Adeiny Hechavarria's fourth single of the night, which forced extra innings.
Faria allowed three runs on just those four hits in 6 2/3 innings in his fifth major league start.
Chris Tillman of Baltimore (39-40) saw his winless streak stretch to 10 starts despite a strong performance. The right-hander allowed two runs on seven hits in five-plus innings and left with the Orioles trailing, 2-1.
Joey Rickard (two RBIs) tied the game at 2-2 with a solo homer leading off the bottom half of the sixth, which set up the late-inning dramatics.
Ramos gave the Rays a 2-0 lead in the second with his first homer of the season, a two-run shot on a 1-2 pitch off Tillman. The catcher missed most of the first three months of the season while recovering from a torn ACL.
The Orioles cut that lead in half when Joey Rickard lined an RBI double to left-center in the third, their first hit of the game off Faria.
NOTES: The Orioles agreed to terms with their first-round pick in this year's draft, LHP D.L. Hall. MASN Sports reported that Hall got a $3 million signing bonus. ... LHP Zach Britton is set for one more rehab assignment (Monday at Triple-A Norfolk), and if all goes well, he could re-join the Orioles on July 5. ... Rays C Wilson Ramos is getting some playing time after missing most of the first three months (torn ACL). He got his second hit of the season, a two-run homer, in the second inning. ... Tampa Bay reversed the option of IF Daniel Robertson and instead put him on the 10-day disabled list due to neck spasms, retroactive to June 26.