The Rangers surrendered an early 2-0 lead and were trailing 3-2 with 1:11 left in regulation when New Jersey's Travis Zajac sent the puck over the glass for a delay-of-game penalty.
On the ensuing power play, Chris Kreider backhanded a shot past Martin Brodeur with 22 seconds left to force overtime.
In the extra period, New York's Ryan Callahan was whistled for a high-sticking double-minor and Gelinas responded on the man-advantage by unleashing a 55-foot shot that eluded Lundqvist for the game-winner.
Michael Ryder, Zajac and Cam Janssen each scored and Brodeur finished with 21 saves as the Devils broke a three-game losing streak.
Mats Zuccarello supplied a goal and an assist and Brad Richards also scored for the Rangers, who dropped a third straight to New Jersey.
Lundqvist ended with 20 saves.
Ray Liotta sues skin care company over use of likeness
Millions of Getty images now available for free via embed tool