"As we determined the details of what took place there and how that attack took place, it became clear that there were terrorists who planned that attack," Leon Panetta said Thursday in Washington.
Panetta, in the administration's latest assessment of the Sept. 11 attack, said investigators hadn't determined who was involved in the assault on the U.S. Consulate in Benghazi in which Ambassador Christopher Stevens and three other diplomatic employees were killed, CNN reported.
"There's a lot of different kinds of terrorism in that part of the world," Panetta said.
FBI investigators were still in Tripoli, waiting to travel to Benghazi and go to the consulate, which is unsecured. The lack of security was the main reason the investigators hadn't traveled to Benghazi, a U.S. official told CNN.
Libyan officials have said they detained dozens of people for questioning.
In the days following the assault, U.S. administration officials gave conflicting assessments of what led up to it. Some officials said the violence erupted spontaneously as protesters demonstrated against a U.S.-made film denigrating the Muslim Prophet Muhammad.
Republican lawmakers have sharply criticized the administration's response and assessments.
Asked when he concluded terrorists orchestrated the attack, Panetta said, "It took a while to really get some of the feedback from what exactly happened at that location."
A senior U.S. official told CNN evidence obtained within 24 hours of the attack indicated extremists either affiliated with or inspired by al-Qaida were behind the assault.
U.S. officials have said they had no "actionable intelligence" before the attack, but intelligence agencies had indications that groups in eastern Libya were trying to "coalesce," Gen. Martin Dempsey, chairman of the Joint Chiefs of Staff, said Thursday.
"There was a thread of intelligence reporting that groups in the environment in eastern Libya were seeking to coalesce but there wasn't anything specific and certainly not a specific threat to the consulate that I am aware of," Dempsey said.
The United States was temporarily removing staff members from its embassy in Tripoli for security reasons, a State Department official said.
A statement on the embassy's website warned citizens to avoid areas of Tripoli and Benghazi where anti-extremist demonstrations were expected Friday.