Atkinson, participating in his first shootout, moved to his backhand after a fake and got it past the Flames goaltender in the fourth round of the tiebreaker.
He finished with 28 saves after relieving starter Curtis Sanford and Nikita Nikitin scored for Columbus.
Sanford was assisted off the ice in the first period after suffering an unspecified lower body injury.
The Flames, meanwhile, sustained their second straight loss as they struggle to make the Western Conference playoff field. They remained in 11th place in the West, 11 points out of the post-season field.
Matt Stajan scored and Kiprusoff made 17 saves for Calgary.
NBC reportedly holds celebs hostage to Jimmy Fallon's show
Ray Liotta sues skin care company over use of likeness