Trailing 2-0 after the first period, the Islanders roared back in the middle frame with three goals from Colin McDonald, Tavares and Brad Boyes to grab the lead.
The Rangers' Carl Hagelin, however, notched the equalizer midway through the second, and after a scoreless third period and overtime, the game went to shootout.
Islanders goalie Evgeni Nabokov made a pair of stops while Frans Nielsen and Tavares beat Martin Biron in the tiebreaker, elevating the Islanders to their first win at Madison Square Garden in nine visits.
Nabokov finished with 36 saves to claim the victory.
Dan Girardi and Marion Gaborik each scored and Biron turned aside 28-of-31 shots for the Rangers, whose three-game winning streak was snapped.