However, investigations of the sexual assaults can and should be improved, he said on NBC's "Today" show.
Panetta's comments came after six basic training instructors at Lackland Air Force Base in San Antonio, Texas, were sentenced on sexual misconduct charges involving at least 38 women.
More than 3,000 service members reported they were sexually assaulted in 2011. Only about 240 of the cases were prosecuted.
The Defense Department acknowledges the difficulty in prosecuting such cases may stop many victims from coming forward. The actual number of assaults, the Pentagon believes, is closer to 19,000.
"It's -- I mean, it's an outrage that -- that we aren't prosecuting people who are involved here," Panetta said.
The secretary said while the cases may be difficult to bring to trial, it can be done.
"We need to improve the investigations. And we need to ensure that we have prosecutors who are willing to bring these cases to court and make sure that these people don't get away," he said.
Panetta said the Lackland situation was isolated, adding "but the reality is, when you've got 19,000 sexual assaults taking place, this is something that's happening elsewhere, as well."
Five of the Lackland instructors charged have either pleaded guilty or been convicted. Their sentences have ranged from 30 days to 20 years in prison.
2014: The Year in Music [PHOTOS]