The Rangers rallied from a two-goal deficit to tie it, but eventually saw a four-game winning streak come to an end. New York had been trying to become the first team in the Eastern Conference to record its 10th home win.
Ottawa had lost four in a row, but outlasted the Rangers thanks in part to 34 saves from Robin Lehner. This was Lehner's third start since being called up by the Senators and the other two had resulted in overtime losses.
Four goals were scored in the first period during a span of just over 11 minutes.
Zack Smith and Patrick Wiercioch gave Ottawa a 2-0 lead, but Rich Nash and Brad Richards delivered the goals that tied it.
Almost 40 minutes went by before either side could muster another goal and it finally came on a wrist shot by Silfverberg for his fourth tally of the season.
Henrik Lundqvist gave up the three Ottawa goals on 39 shots.