With the score tied 1-1 and 2:24 left in regulation, Kelly got his second goal of the game when Jarkko Ruutu gained control of the puck behind the New York net and whipped a quick pass to the left of crease, where Kelly one-timed it home for the decisive score.
Kelly also tallied an unassisted, shorthanded goal in the first period and added an empty-netter for Ottawa, which broke a three-game losing streak.
Pascal Leclaire was tough in goal, stopping 25 shots to earn only his second win of the season for the Senators.
Brandon Prust scored and Henrik Lundqvist stopped 22-of-24 shots in New York's loss.
Notable deaths of 2014 [PHOTOS]