Pittsburgh tied the game on an eighth-inning home run by Pedro Alvarez and the Pirates missed a chance to take the lead in the ninth when New York center fielder Juan Lagares made a spectacular catch to rob Andrew McCutchen of an extra base hit that would have brought in a runner from first.
New York then struck in the bottom half of the inning against Jason Grilli (0-1) to win the opener of a four-game series.
Byrd reached to open the inning on an infield single and moved to second on a sacrifice from Andrew Brown.
That brought up Baxter, pinch hitting for Legares, and he drove an 0-1 pitch to center for the game-winning hit.
Bobby Parnell (4-0) picked up the victory as the fifth New York pitcher by working the top of the ninth.