The Sabres snapped a two-game skid, getting two goals from Jason Pominville and a 41-save effort from Jhonas Enroth.
Columbus has won only once in its first 10 games this season.
Kaleta's game-winning score was his first goal of the season. He scored just four times in 51 games with the Sabres last year.
Pominville gave Buffalo a 2-0 lead 4:51 in the second period, but Columbus came back to tie it on goals by Vaclav Prospal in the second and Fedor Tyutin in the third.
After Kaleta's go-ahead goal, Pominville gave the Sabres some insurance by scoring on the power play with 2:24 to play.
Steve Mason made 22 saves to take the loss.
Celebrity Breakups and divorces of 2014 [PHOTOS]