The forum in Phnom Penh and led by Cambodia ended in accusations of foot-dragging by some of member states over how to deal with Chinese territorial claims, especially regarding the Scarborough Shoal.
China isn't one of ASEAN's 10 members, which are Viet Nam, Philippines, Malaysia, Cambodia, Thailand, Brunei, Laos, Singapore, Indonesia and Myanmar.
Many members were unhappy about Cambodia's refusal to mention Scarborough Shoal in a final communique, which resulted in no final document, the first time in ASEAN's 45 years.
"The chair (Cambodia) has consistently opposed any mention of the Scarborough Shoal in the joint communique and announced that a joint communique can't be issued," Philippines Foreign Affairs Secretary Albert del Rosario said after attending the forum.
Cambodian Foreign Minister Hor Namhong said all ASEAN members were responsible for the failure to issue a final joint statement, a report by the BBC said.
"I requested that we issue the joint communique without mention of the South China Sea dispute ... but some member countries repeatedly insisted to put the issue of the Scarborough Shoal," Hor said.
"I have told my colleagues that the meeting of the ASEAN foreign ministers is not a court, a place to give a verdict about the dispute," Hor said.
Del Rosario warned China's "creeping imposition of its claim" over the entire South China Sea, "could further escalate into physical hostilities which no one wants."
"This puts in greater jeopardy the remarkable economic dynamism of our region which was made possible by the relative peace and stability that has prevailed in the past years," del Rosario said.
Del Rosaria also sounded a conciliatory note, saying the disagreement over including a Scarborough Shoal note didn't signal the breakup of ASEAN, a report by the Philippines's TV5 said.
"I don't think we should even think that this is the beginning of a tear in the organization," he said. "I think it just presents a bigger challenge for us to continue to build on what we stand for -- leadership, centrality and solidarity," del Rosario said.
The Scarborough Shoal issue is emblematic of the evolving and increasingly confrontational South China Sea territorial disputes, mostly involving China.
Scarborough Shoal -- also called Panatag Shoal and Bajo de Masinloc -- covers less than 60 square miles and has a highest point of around 10 feet above sea level.
The shoal is more than 400 miles off the Chinese coast but 150 miles off the coast of Zambales, a province on the western shore of Luzon Island, the largest and most northern Philippines island.
Ownership of the shoal, as for other disputed South China Sea territories, is important because legitimizes a country's access to natural resources including oil and gas on the seabed and fishing rights in the area.
Manila and Beijing have been at loggerheads over the shoal for several months, each claiming the other is driving the dispute toward a naval confrontation.
Another possible naval flash point is the Spratly islands and reefs group.
As well as Vietnam and China, ownership of various Spratly islands and reefs -- some visible only at low tide -- are disputed by Brunei, Taiwan, Malaysia and the Philippines.
The Spratly dispute has erupted into open military confrontation on occasions, such as the brief 1988 Johnson South Reef skirmish between China and Vietnam in which about 70 Vietnamese military personnel were killed.
Del Rosario reiterated his country's desire to have territorial conflicts resolved through a U.N. maritime treaty signed by the Philippines, China and other governments.
China's stance is for the issues to be resolved bilaterally.