James also provided eight rebounds and six assists for the Heat, who won their fifth straight game.
Miami was trailing 119-116 with 1:08 left in overtime when Chris Andersen executed a layup, cutting its deficit to one.
He connected on both free throws to put the Heat ahead by a point.
Korver missed a layup attempt over the rim on Atlanta's final push.
Jeff Teague put in 26 points and Millsap ended with 25 and 10 rebounds for the Hawks.
Al Horford contributed 21 points and 11 rebounds.
Easer Egg Roll brings thousands to White House
Turkey considering to use pistachios to heat country’s first eco-city