Axelrod described Obama as "unsmiling and slightly irritated" while he was on the phone with the Republican.
"'You really did a great job of getting the vote out in places like Cleveland and Milwaukee,' in other words, black people,'" Obama said, summarizing Romney's words. "That's what he thinks this was all about."
Romney, who served one term as governor of Massachusetts, announced last week that he has decided against making a third try for the presidency.
On May 17, he will be in New Hampshire to speak at St. Anselm's College in Manchester and receive an honorary degree. The college, strategically located in the state with the first presidential primary, was the site of campaign debates in 2008 and 2012 and the place where Romney announced in 2012 that Rep. Paul Ryan of Wisconsin as his vice presidential choice.
Axelrod also revealed that Obama complained when he and Vice President Joe Biden were both in the Senate that "he can just talk and talk." He said the president joked when he was trying to persuade Sen. Olympia Snowe, R-Maine, to support his healthcare plan that he and Michelle would move into an apartment and let her have the White House if she came through.
A newspaperman turned political consultant, Axelrod first met Obama in 1992 and became a key campaign adviser in 2008 and 2012. He was a senior adviser at the White House from 2009 to 2011.