The Flyers twice came from behind and forced a 3-3 tie when Chris Pronger scored on the power play with 1:12 to play in the second period.
New Jersey, however, dominated the third and bounced back from a 2-1 setback suffered in the opener of the Eastern Conference quarterfinal series on Wednesday.
Zubrus scored on a slap shot with 4:04 remaining in the third period and Kovalchuk put the puck into an empty net with 33 seconds left.
Zach Parise, Colin White and Andy Greene also scored for the Devils. The goal by Parise came shorthanded just 2:45 into the game.
Martin Brodeur made 26 saves in recording the 99th playoff victory of his 16-year career. The Devils have won the Stanley Cup three times and Brodeur has been in goal for all of those championships.
Brian Boucher gave up four goals on 32 shots.
Philadelphia will host the third game of the series on Sunday.