The winning run was an unearned one for the Orioles, who salvaged one win in a three-game series at Yankee Stadium.
Matt Angle reached base on a one-out fielding error by New York shortstop Eduardo Nunez, stole second and came in with the go-ahead run on Reynolds' single to shallow left off Hector Noesi (2-1).
Nolan Reimold socked a two-run homer and drew four walks and Robert Andino added three hits for the O's.
The Baltimore bullpen tossed five innings of hitless relief, with Pedro Strop (1-1) collecting the victory thanks to 1 1/3 frames of shutout ball.
Alex Rodriguez and Jesus Montero each knocked in a pair of runs in the loss, which ended a six-game winning streak for the American League East-leading Yankees.