Panetta and other leaders at the U.S. Defense Department have been critical of political debates surrounding the billions of dollars in defense spending that may be cut in the so-called sequester.
Panetta said he was pleased that U.S. President Barack Obama expressed concerns over the matter during his State of the Union address Tuesday.
"Members of Congress need to understand that they were elected to protect the public, not to hurt the public," he said.
Overseas, NATO officials have expressed their own concerns that weak economies might not be able to support defense programs meant to ensure national and international security.
"This is not a game," Panetta said. "This is reality."
Panetta was speaking in what he said he believed was his final news conference at the Defense Department. Obama nominated former U.S. Sen. Chuck Hagel, R-Neb., as his replacement.
Republican critics have threatened to hold up Hagel's nomination vote on issues ranging from last year's terrorist attack in Libya to Hagel's stance on Iran.