New York thus evened the Eastern Conference quarterfinal series at 2-2, every game having been won by the home team. Washington will host Game 5 on Friday.
The teams battled to a 2-2 tie through two periods, Washington coming back from a two-goal deficit with a pair of scores during the latter stages of the middle stanza.
Girardi then scored 59 seconds into the third and Derek Stepan gave New York a two-goal lead 5 minutes later.
Lundqvist stopped 27-of-30 shots while Braden Holtby made 30 saves for the Capitals.
'Happy' fan videos make Pharrell cry [VIDEO]
Pregnant Mila Kunis wins 'Best Villain' at MTV Movie Awards