Atkinson, playing in the 100th game of his NHL career, scored between the pads of Martin Brodeur after assisting on the Blue Jackets' only goal of regulation.
That goal came from Artem Anisimov 7:34 into the first period and stood up until early in the third. Adam Henrique delivered the game-tying goal for the Devils, who lost their second straight.
Curtis McElhinney had to stop just 16 shots to claim the win, the third in a row for Columbus. Brodeur made 29 saves.
Millions of Getty images now available for free via embed tool
Aaron Carter is still in love with Hilary Duff