Over the next four years, Australia will send the boat people to Malaysia as soon as they arrive in Australian waters and are picked up by Australia's coast guard and navy.
For every person transferred to Malaysia, Australia will take five refugees from Malaysia, Prime Minister Julia Gillard and the Minister for Immigration and Citizenship Chris Bowen said in a joint written statement.
This will mean an additional 4,000 refugees being moved to Australia, the statement said. But Australia will take only those refugees already in Malaysia and none of the boat people who will have been transferred there under the agreement.
The deal is part of a "regional cooperation framework" that the Australian government said is aimed at putting out of business notorious people smugglers operating across Asia.
"Australians are hospitable people and we believe in honoring our international protection obligations," the statement said.
"But do not underestimate our resolve to break the people smugglers' business model and the trade in human misery that they rely on. This landmark agreement will help take away the product people smugglers are trying to sell -- a ticket to Australia.
"The key message this will deliver to people smugglers and those seeking to make the dangerous sea voyage to Australia is: Do not get on that boat. Under this arrangement, if you arrive in Australian waters and are taken to Malaysia you will go to the back of the queue."
Successive Australian governments have urged Asian governments to crack down on people smugglers who charge thousand or dollars for passage on unseaworthy boats. But few countries have prosecuted smugglers, although some have been given jail sentences by Australian courts.
The government's critics and human rights groups were quick to condemn the agreement, claiming Malaysia's record on human rights is poor and the country hasn't signed the U.N. refugee convention.
"If you're looking for a country in the region that has some of the harshest policies toward refugees and asylum seekers, then you couldn't really go past Malaysia," Amnesty International Australia's refugee coordinator Graham Thom said.
But Malaysia's high commissioner in Canberra, Salman Ahmad, said Malaysia will treat the refugees well.
We have our own reasons for not being a signatory to the convention but this does not mean we don't treat refugees with respect and dignity," Ahmad told Sky television news.
He said it is a "one-off arrangement" that could be a model for agreements with other countries in the Asia-Pacific region.
"If these things work and become effective in tackling this problem, we hope other countries will also join in," he said.
The Gillard government, as previous governments, is wrestling with an annual influx of thousands of boat people. Australia's Department of Immigration said 134 boats carrying 6,535 people arrived in 2010.
So far this year, 16 boats carrying around 920 people have been intercepted by Australian authorities.
Detention centers on the mainland and also the main center on Australia's Christmas Island are full, or nearly full, and the government is keen to build detention centers.
In 2008 the government abandoned the controversial Pacific Solution, a 2001-07 policy in which asylum seekers were transferred to detention camps on small islands, including Papua New Guinea's Manus Island.
The policy appeared to work, with arrivals falling from 5,516 in 2001 to one in 2002. But problems ensued at some centers, most of which kept people under high-security confinement.
On Manus Island, hunger strikes, riots and a malaria outbreak happened in the early days of the center's operation. In 2002 more than 40 asylum seekers tried to escape by swimming out to sea.
The Australian government also confirmed in its statement that Australia is in formal discussions with Papua New Guinea to reopen the Manus Island center.