Parise provided a helper on Travis Zajac's sixth goal of the playoffs 12 minutes into the first period, giving the Devils a 2-0 advantage.
The left wing added a power-play tally 2:41 into the third period to put New Jersey ahead 3-0 and later fired home an empty-netter for his sixth marker of the post-season.
Bryce Salvador also found the net for New Jersey, which rebounded from a 3-0 loss in Game 3 to draw even in the series.
Martin Brodeur turned aside 28-of-29 shots to earn the victory.
Henrik Lundqvist made 26 saves and Ruslan Fedotenko netted a third-period goal for the Rangers, who will host Wednesday's Game 5 at Madison Square Garden.