The Rangers surrendered an early 2-0 lead and were trailing 3-2 with 1:11 left in regulation when New Jersey's Travis Zajac sent the puck over the glass for a delay-of-game penalty.
On the ensuing power play, Chris Kreider backhanded a shot past Martin Brodeur with 22 seconds left to force overtime.
In the extra period, New York's Ryan Callahan was whistled for a high-sticking double-minor and Gelinas responded on the man-advantage by unleashing a 55-foot shot that eluded Lundqvist for the game-winner.
Michael Ryder, Zajac and Cam Janssen each scored and Brodeur finished with 21 saves as the Devils broke a three-game losing streak.
Mats Zuccarello supplied a goal and an assist and Brad Richards also scored for the Rangers, who dropped a third straight to New Jersey.
Lundqvist ended with 20 saves.