James also provided eight rebounds and six assists for the Heat, who won their fifth straight game.
Miami was trailing 119-116 with 1:08 left in overtime when Chris Andersen executed a layup, cutting its deficit to one.
He connected on both free throws to put the Heat ahead by a point.
Korver missed a layup attempt over the rim on Atlanta's final push.
Jeff Teague put in 26 points and Millsap ended with 25 and 10 rebounds for the Hawks.
Al Horford contributed 21 points and 11 rebounds.
Ray Liotta sues skin care company over use of likeness
Aaron Carter is still in love with Hilary Duff