The Bruins were trying to sweep the Rangers in the Eastern Conference semifinals, but gave up a third-period lead and saw their advantage in the series reduced to 3-1.
Boston will try again to advance to the conference finals when New York hosts Game 5 on Saturday.
Only three teams in the history of the NHL have come back to win a playoff series after losing the first three games. Boston was one of those victims, having been knocked out in 2010 after taking a 3-0 lead against Philadelphia.
New York did not lead in Thursday's game until Kreider scored his first goal of the playoffs by tipping in a shot from Rick Nash.
It appeared this would be New York's final game of the season after Nathan Horton and Torey Krug scored second-period, power-play goals to give the Bruins a 2-0 advantage.
Carl Hagelin, however, brought the Rangers within one midway through the second period and Derek Stepan scored on a wraparound shot 1:15 into the third.
Boston took the lead again on Tyler Seguin's goal with 11:54 to go in regulation only to have Brian Boyle forge a 3-3 tie with a power-play goal exactly halfway through the third.
New York played without center Brad Richards, who was a healthy scratch after scoring only one goal in 10 playoff games this season. He has been a disappointment after signing a nine-year, $58 million contract before the start of the 2011-12 season.