Kelly Johnson and Shelly Duncan hit solo homers in each of the first two innings to give Tampa Bay the early lead, but Adam Jones countered with a two-run blast for the Orioles that tied it in the third.
Loney then made it a 4-2 game in the fourth with his key hit and he added an RBI single in the eighth.
Matt Moore (3-0) remained unbeaten with a 6 2/3-inning effort. He gave up two runs on five hits and struck out seven.
The loss was charged to Chris Tillman (0-1), who surrendered four runs on six hits in five innings.
Duggar sisters unveil Christian dating rules in new book
McPhee, Cokas 'working on their marriage' after affair