Gannon, who played under Gruden from 1999-2001, told 95.7 The Game that he spoke with the team's new head coach on Wednesday and declined an opportunity to be the Raiders' quarterbacks coach.
The 52-year-old Gannon initially said he would consider the role and even attended Gruden's introductory news conference on Tuesday.
"I really came to my senses probably on the flight up there (Tuesday) to go to the press conference and just realized that to work alongside Jon Gruden it takes a special person," Gannon said. "It takes a person with great energy, passion and commitment and, more importantly, time. I didn't think it would be fair to myself, my family or even to Jon not to be all in in with him."
Gannon, who is currently a CBS analyst, said that he still plans to be involved with the Raiders "in some smaller capacity," perhaps by mentoring fellow quarterback Derek Carr.
"It's not like Derek is a guy that needs some special treatment," said Gannon, who was the NFL MVP in 2002.
"He's a tough guy, he's worked hard. He's developed into a really good player in this league. If you really lift up the hood and study him, he's a really good player."