After taking the oath at Government House, Rudd addressed Parliament, paying tribute to Gillard, who is retiring from politics, the BBC reported.
Describing politics as a "very hard life," Rudd called on lawmakers "to be a little kinder and gentler with each other."
Anthony Albanese was sworn in as deputy prime minister and Chris Bowen replaced Wayne Swan as treasurer. Six of Gillard's Cabinet members resigned after the party vote; further shake-ups are expected.
The party leadership change comes ahead of a Sept. 14 election, which polls suggest Labor will lose.
Opposition leader Tony Abbott called on Rudd to clarify whether the election would be advanced.
"I congratulate the prime minister on his restoration to high office," Abbott said. "May he elevate that office ... by telling the Australian people when will they get the chance to decide who the prime minister of this country should be and who should form the government of this country?"
Rudd said he would "identify a date for an election," hinting he wasn't bound by the date set by Gillard.
"There's going to be an election, it will be held consistent with the constitution and ... there's not going to be a huge variation one way or the other," Rudd said.
Rudd returned to lead Australia's government three years after he was toppled by Gillard in a Labor leadership vote.
Speaking after the vote Wednesday, Gillard said she was proud of her achievements in what "has not been an easy environment to work in" because of the minority in Parliament and divisiveness within the Labor Party, Britain's The Guardian reported.
Rebuking accusations that she played the "gender card" calling out sexism, she said her position as the first female incumbent "explains some things and it is for the nation to think in a sophisticated way about those shades of grey."