Panetta said that while a U.S. commando team was positioned in Sicily, the distances and relatively brief time frame of the incident made it unlikely they could have arrived in time to turn back the terrorist attack that left four Americans dead.
"By the time we got the information as to what, in fact, was taking place there, just distance alone made it very difficult to respond quickly," Panetta told CNN's "State of the Union." "That's just the nature of dealing with the Middle East."
The Obama administration has come under harsh and ongoing criticism from Republican lawmakers for supposedly failing to provide adequate security for U.S. diplomats in Libya, and for failing to respond quickly and forcefully to the raid.
Panetta told CNN he would likely testify before Congress about Benghazi after he steps down as defense secretary later this winter.
He said it all boiled down to the quantity and quality of the intelligence the United States had at its disposal in the run-up to the September attack. "There are areas in the Middle East where we do not have the kind of intelligence we should have in order to give us a heads up about these kinds of attacks," he said. "That's a reality."
Gen. Patrick Dempsey, outgoing chairman of the Joint Chiefs of Staff, agreed the flying times alone were enough to make a military rescue mission a futile gesture. He also took issue with the idea the consulate was besieged for some seven hours while Washington was caught flat-footed.
"It wasn't a seven-hour battle. It was two 20-minute battles separated by about six hours," said Dempsey, who also faces congressional testimony in the near future. "The idea that this was one continuous event is just incorrect."
Dempsey added: "Our closest armed aircraft happened to be in Djibouti. The distance from Djibouti to Benghazi is the distance from Washington, D.C., to Los Angeles."