The Sabres snapped a two-game skid, getting two goals from Jason Pominville and a 41-save effort from Jhonas Enroth.
Columbus has won only once in its first 10 games this season.
Kaleta's game-winning score was his first goal of the season. He scored just four times in 51 games with the Sabres last year.
Pominville gave Buffalo a 2-0 lead 4:51 in the second period, but Columbus came back to tie it on goals by Vaclav Prospal in the second and Fedor Tyutin in the third.
After Kaleta's go-ahead goal, Pominville gave the Sabres some insurance by scoring on the power play with 2:24 to play.
Steve Mason made 22 saves to take the loss.
Selena Gomez drops F-bomb, walks off stage during Jingle Ball performance
'SNL': 'Anchorman 2' cast, One Direction sing 'Afternoon Delight' [VIDEO]