The Capitals' triumph came in the opener of the Eastern Conference quarterfinal series, which will continue on Saturday with Washington hosting Game 2.
Ovechkin, who scored 32 goals in the 48-game regular season, produced a power-play goal 6:59 into the middle period to create a 1-1 tie.
Marcus Johansson put Washington in front with 5:39 left in the second and Jason Chimera tallied 46 seconds later.
The second-period outburst came against Henrik Lundqvist, who made 27 saves.
Braden Holtby, in his first season as a No. 1 goaltender, stopped 35-of-36 shots to get the win. Carl Hagelin scored New York's only goal with 15 seconds left in the opening period.
After making only seven appearances during the regular campaign last season, Holtby wound up starting 14 games in the playoffs and posted a 7-7 record.
This is the fourth time in five years the Capitals and Rangers have met in the postseason. New York won a seven-game series in the conference semifinals last year.
Ukrainian protestors topple Lenin statue [VIDEO]
'SNL': 'Anchorman 2' cast, One Direction sing 'Afternoon Delight' [VIDEO]