Gillard said she would not contest the vote, after which Deputy Prime Minister Wayne Swan and Senate leader Stephen Conroy, along with Joe Ludwig and Craig Emerson, resigned their Cabinet posts, the Australian Broadcasting Corp. reported.
The vote was the third time Labor leadership butted heads since Gillard ousted Rudd in 2010.
Gillard called for the ballot to end the long-running dispute over whether she or Rudd is better capable of guiding the party into an election that polls suggest will result in Labor's defeat, The Wall Street Journal reported.
Both Gillard and Rudd pledged that if they lost the leadership vote they would leave Parliament before the election, scheduled for Sept. 14.
Earlier Wednesday, opposition leader Tony Abbot called on Gillard to advance the election to Aug. 3 to prevent Labor Party divisions from hurting the country.
"The poison inside the Australian Labor party is paralyzing government in this country," said Abbott, leader of the Liberal National coalition.
When he announced his intention to take on Gillard, Rudd told reporters the party is set for a "catastrophic defeat" without a change at the top, stressing the need for better management of the economy as the country's mining boom nears its peak.
"The nation needs strong, proven national economic leadership to deal with the formidable new challenge that Australia now faces, with the end of the decade-long China resources boom and its impact on Australian jobs and living standards into the future," Rudd said. "This is a massive new challenge."