General Manager Marty Hurney told the newspaper about Trgovac's decision from the site of Senior Bowl in Mobile, Ala.
The Observer had reported Wednesday that Trgovac, who has been the Panthers defensive coordinator since 2003, was planning to remain with the Panthers and sign a new contract.
"We offered (Trgovac) a contract and told him we wanted him back, but he informed us he wanted to pursue other options," Hurney said.
Defensive tackles coach Sal Sunseri took a college job at Alabama and linebackers' coach Ken Flajole became the defensive coordinator for the St. Louis Rams this week.
Before those defections, quarterbacks coach Mike McCoy left to be offensive coordinator for the Denver Broncos and secondary coach Tim Lewis took a similar job with the Seattle Seahawks.
Fox has not named any replacements.
Swim Week Miami Beach 2014 [PHOTOS]