Carr is a former New South Wales premier.
Before his appointment was announced, there had been speculation Defense Minister Stephen Smith would get the Foreign Ministry post, the Sydney Morning Herald reported.
Gillard said Smith, who had been foreign minister before Rudd took the job, would stay on as defense minister and had much work ahead of him.
The prime minister retained her party leadership after defeating Rudd's challenge in a 71-31 vote earlier this week.
Emergency Management Minister Robert McClelland, a Rudd supporter, was dropped from the new Cabinet, the report said.
Carr, who has been out of public life for six years, said Gillard made him an offer he could not refuse.
''You don't choose the moment; very often the moment chooses you,'' he said.
Carr, praising his predecessor, said he would seek Rudd's advice, the Australian Broadcasting Corp. reported.
Rudd said Carr is an "excellent choice" and regards him as a good friend.
Three other ministers who had supported Rudd survived the reshuffle and retained their jobs.