Baltimore had lost 30 of its last 37 meetings with the Yankees, including all six this season prior to Friday's game. The contest was delayed by rain at the start for almost 2 hours.
Reynolds hit his homer in the second and the Orioles built a 4-0 lead on an RBI double by Derrek Lee in the fourth and a solo homer from Lee in the sixth.
Baltimore took a 4-1 advantage into the ninth and New York quickly cut its deficit to two when Robinson Cano and Nick Swisher delivered doubles to open the inning.
Gregg gave up a one-out walk to Jorge Posada, but a groundout and a strikeout ended the game and gave him his 17th save.
The win went to Jeremy Guthrie (5-14), who gave up a run on four hits in seven innings.
A.J. Burnett (8-9) struck out 10, but surrendered four runs on five hits in eight innings to take the loss.
Ohio crash that killed two caught on camera [VIDEO]
Wisconsin business offering 'therapeutic cuddling' forced to close