An assessment released by the department listed seven problem areas identified during the exercise.
"The results were somewhat humbling," a private sector analyst involved in planning the drill told United Press International.
"But that's what it was for," the analyst added, "You want to learn where you need to make improvements."
"If the results are too good, you haven't challenged yourself enough. A report that says 'everything's fine' might be comforting, but it's not in anyone's interest. It's a whitewash."
The simulation, dubbed TOPOFF 2, for top officials, was staged on May 12-16 this year. Twenty five federal, state and local agencies took part in a mock terrorist attack involving a radiological or "dirty bomb" explosion in Seattle, and a covert biowar attack that released pneumonic plague in several locations in the Chicago metropolitan area.
The report said that the color-coded national threat alert system, which was notionally raised to its highest level -- "red" or "severe" -- for the first time ever during the exercise, needed "refinement."
Simply put, many first responders and other state and local officials just did not know what they were supposed to do when the so-called Homeland Security Advisory System threat level was raised.
"There was ... uncertainty regarding specific protective actions to be taken by specific agencies under a HSAS severe threat code red," the report said. It went on to propose the development of a "comprehensive operational framework that jurisdictions at all levels could use to help define their response plans at each HSAS threat condition."
Another problem identified by the report was poor communications infrastructure, especially in the public health sector in the Chicago area, where 64 hospitals took part in the exercise, during which volunteers playing patients poured in complaining of symptoms of a mysterious respiratory illness.
"The lack of a robust and efficient emergency communications infrastructure was apparent," says the report, pointing out that with hospitals relying on telephones and faxes for communications, some quickly found their lines overwhelmed. Repeatedly tabulating faxed casualty reports by hand was slow and "increased the potential for error."
"It was terribly twentieth century," said the analyst, pointing out that Chicago was widely considered to have one of the most robust health systems in the country.
In one Illinois location, the report said, ham radio operators where the only means of communication.
The document released Friday is a short public summary of a longer classified report, which UPI was not able to obtain. But interviews with a number of participants -- and a review of the unclassified summary -- paint a picture of an often chaotic, sometimes uncontrolled situation, with different agencies and jurisdictions giving different, and at times conflicting, advice to the public.
In Chicago, for instance, authorities appeared confused about who should get treatment, and the locations in which people might have been exposed.
"Inconsistent information was given by different jurisdictions as to who should seek prophylaxis and when, as well as the locations of the suspected plague release sites," says the report.
In Seattle, there was confusion about the size and shape of the area affected by the radiation released by the dirty bomb, which -- in a real emergency -- would make decisions about where to safely deploy responders impossible to make.
Planners said some degree of chaos was inevitable given the scale of the drill.
"The exercise was designed to simulate an unprecedented combination of circumstances," said the analyst. "Under the pressure of rapidly evolving events, people were forced in many places to improvise."
David Heyman of the Center for Strategic and International Studies -- a think tank which has been involved in a number of similar exercises -- told UPI that the biggest question raised by the exercise was the issue of preparedness.
"How do we know when we're prepared?" he asked. 'Where's the bar?"
He said exercises like TOPOFF 2 identified areas where first responders or other local agencies were less able to cope, but there was no absolute standard against which their readiness could be measured.
"There's no mechanism in place to tell us when we're ready," he said. "It's a big gap."
White House spokesman Trent Duffy told UPI that a presidential directive published Wednesday was designed to close that gap.
The directive instructs homeland security secretary Tom Ridge to draw up a "national preparedness goal," which would "establish measurable readiness priorities and targets."
"It will also include readiness metrics ... including standards for preparedness assessments and strategies, and a system for assessing the Nation's overall preparedness to respond to ... terrorism."
"This directive reflects everything we've learned about preparedness since Sept. 11," said Duffy.
But Heyman added, "even if we have standards, there's no oversight or enforcement mechanism to ensure everyone is meeting them."
The directive orders cabinet officials to use their funding discretion to encourage local jurisdictions to meet the standards laid down by Ridge.
TOPOFF 2 was the largest exercise of its kind ever staged in the United States. Over 8,000 officials took part in the $16 million exercise.
After the "live play" of the first two days of the simulation, the rest of the week was spent on a tabletop exercise examining the after-effects and the impact of such an attack
Details of more than 800 separate developments that made up the exercise -- referred to by the event planners as "stimuli" -- were contained in a 200-page script. Representatives of all the agencies involved -- sitting together in a northern Virginia command center -- fed the "stimuli" back to their colleagues in the field, who then reacted as they would do in a real emergency.
2014: The Year in Music [PHOTOS]