The 66-year-old Letterman announced his intent to retire from the Late Show on April 3. The late night host has been with the show for over 20 years, and will most likely depart in 2015.
Rumors that Colbert might replace the host began almost immediately after Letterman's announcement. Stewart was also thought to be a potential replacement, but the 51-year-old Daily Show host says the job is not for him.
"I don't have that gear, I don't think," Stewart tells Vulture. "I just don't really have it anymore. I really like what I do, not that Stephen doesn't. But he has a real opportunity to broaden out in a way that I don't."
Stewart has nothing but good things to say about Colbert, and thinks he would make a fine replacement for Letterman.
"[Colbert] is a uniquely talented individual," the host continues. "He's wonderful in Colbert Report, but he's got gears he hasn't even shown people yet. He would be remarkable."
"He's got some skill sets that are really applicable, interviewing-wise," Stewart adds. "But also he's a really, really good actor and also an excellent improvisational comedian. He's also got great writing skills. He's got a lot of the different capacities. Being able to expand upon [those] would be exciting."
Colbert is under contract at Comedy Central until the end of 2014. Sources say the host would leave behind the conservative persona he assumes for Colbert Report if chosen to replace Letterman.