The Bulls scored only 11 points in the fourth period and missed nine consecutive attempts from the field down the stretch, but pulled out the win when Rose launched a shot over two defenders from the baseline that rattled home.
Chicago bounced back from a season-opening loss to Miami while the Knicks missed a chance to open the campaign with two straight victories.
New York made less than a third of its shots in the first period, turned the ball over six times and fell behind by 10 points in a poor start on the road.
The Knicks still trailed by 10 heading into the final period, but the Bulls went cold from the field and New York came back to tie it at 80-80 when Iman Shumpert made two free throws with 1:15 remaining.
New York's Tyson Chandler was fouled with 10 seconds to go while fighting for a rebound and he made 1-of-2 from the line to put the Knicks in front.
That set up the shot by Rose, which he put up over both Chandler and Raymond Felton.
Rose, playing in his second game after missing the entire 2012-13 season, led Chicago with 18 points but made only 7-of-23 from the field and suffered four of the Bulls' 18 turnovers.