The demand is contained in a U.N. report that found Australia guilty of nearly 150 violations of international law over the indefinite detention of the 46 refugees, The Age newspaper reported.
The report is one of the most damning U.N. assessments of human rights in Australia, Michael Gordon, political editor of the Age, said.
The federal government also should provide the 46 people with rehabilitation and compensation, the report concluded.
Under Australia's treaty obligations, the government has 180 days to act on the committee's recommendations and to prevent similar incidents, The Age reported.
The report comes at a politically sensitive time for both the government and the main opposition, as they face off in a national election with voting set for Sept. 7.
Labor Party Prime Minister Kevin Rudd is seeking a third term against the opposition Coalition led by Tony Abbott of the Liberal Party. Opinion polls have put them close, with the Coalition starting to pull ahead.
One of the more contentious issues on the campaign trail is Australia's treatment of an ever-increasing number of would-be asylum seekers arriving by rickety boats in territorial waters after paying people smugglers for their journeys.
They come from as far as Pakistan, India and Sri Lanka, traveling overland through Southeast Asia to board boats in countries including Indonesia and Malaysia.
Labor's proposed solution is to send the asylum seekers directly to detention camps in Papua New Guinea and the Micronesian island nation of Nauru, with no hope of entering Australia under refugee status.
Abbott announced a plan to appoint a senior military commander as head of a force to go after people smugglers and boat arrival operations.
Abbott's Operation Sovereign Borders plan is for Australia's 12 agencies involved in border protection to come under the command of a single three-star military commander.
Meanwhile, refugees keep arriving by their hundreds, with many -- women and children -- drowning after their vessels sink.
The U.N. report, which focuses on 46 asylum seekers, says the government didn't demonstrate that full detention was warranted versus less restrictive prohibitions on the refugees freedom, the report said.
Ongoing detention of the 46 refugees -- mostly Sri Lankan Tamils -- is "cumulatively inflicting serious psychological harm" and in breach of the International Covenant on Civil and Political Rights, the U.N. report said.
The U.N. commission investigation followed a complaint in August 2011 by Professor Ben Saul of the Sydney Center for International Law.
"It's a major embarrassment for Australia, which is a member of the Security Council and often criticizes human rights in other countries," Saul told Fairfax Media. "Australia should do the right thing by respecting its international obligations and treating the refugees decently."
The report is another blow by the United Nations against Australia's refugee processing system.
The U.N. High Commissioner for Refugees said in July that despite improvements, conditions at its offshore processing center on Manus Island in Papua New Guinea -- housing 250 people -- didn't meet international standards.
"Living conditions are still harsh, processing remains slow and asylum-seekers are growing despondent over the lack of certainty about their future," UNHCR spokesman Adrian Edwards said.
The agency's second report on the Manus Island Regional Processing Center followed a monthlong visit, during which the team observed cramped living quarters, especially for single men housed in canvas tents.
"The mission found improvements since its last visit in January, but current arrangements still do not meet international protection standards for the reception and treatment of asylum-seekers," Edwards said.
Improvements included the transfer of all children and their families to Australia, progress toward establishing a legal framework for processing and some improvement in the physical facilities at the processing center.
"However, UNHCR's inspection also revealed continued and worrying shortcomings," Edwards said. "Freedom of movement is still extremely limited in what continues to amount to an environment of open-ended, mandatory and, in UNHCR's view, arbitrary detention."