Observers who attended the 88-nation talks in Agadir, Morocco, said the collapse of the negotiations meant whale hunting could continue in a more disorderly and undisciplined manner than now.
The International Whaling Commission pushed for a compromise deal that could allow some hunting under a quota system.
Despite a global ban announced in 1986, Japan, Iceland and Norway continue killing whales. At least 1,500 were killed last year, most of them by crews on Japanese vessels, though some analysts said the numbers could be higher.
Hunting nations use loopholes in the ban to continue with catches, citing scientific research as the reason for killing whales.
In South America, previous bans on hunting have led to growth in whale stocks and promoted tourism focused on whale watching in Argentina, Brazil and Ecuador.
The worst killings, however, continue in a southern region designated as a whale sanctuary.
In addition to hunting, a further environmental threat to whales was highlighted when in May a dead sperm whale was found floating 77 miles south of the oil spill in the Gulf of Mexico. The discovery of the dead whale revived concerns about the future of whales in the southern waters.
Latin American nations and the European Union sought an early end to hunting while U.S. negotiators pushed for a compromise that could save the talks. Faced with hardened positions in the EU and South America, the pro-hunting nations quickly moved away from a compromise.
The failure means there is greater risk of less regulated whale-hunting than before the IWC congress, analysts said.
Argentine representative Susana Ruiz Cerutti said the draft proposal didn't meet the needs of Latin American countries, which pushed for an early ban. She told the BBC the plan would have legitimized "scientific" whaling by Japan in the Southern Ocean and failed to specify substantial cuts in catches.
U.S. commissioner Monica Medina said the breakdown was the fault of no particular party but other delegates blamed pro-hunting delegates.
The divergence of views was highlighted when Japanese representative Yasue Funayama, a junior agriculture and fisheries minister, said she saw IWC as a "resource
management organization. Opponents of the continuing hunt said they favored a stronger IWC playing the role of a conservation body.
Anti-whaling delegates accused Japan of vote-buying in a bid to garner support for the Japanese position.
Analysts said the collapse of the talks put into question IWC's future as a platform for negotiating workable deals. Although environmentalist groups hailed the failure of a compromise effort, moderate representatives who backed a compromise on lower catches by Japan, Iceland and Norway said the failure meant the group was headed for a more acrimonious future, with less possibility of whaling nations honoring the ban.