The British television personality recently announced he is leaving "Idol" at the end of this season to star on and produce a U.S. version of the British competition series "The X Factor."
"You're never going to pick anyone as good as you are, are you?" the New York Daily News quoted Cowell as telling reporters Thursday when asked who should take over his "Idol" job. "That's why I don't think anybody (at Fox) really asks me that question."
Pop star Madonna, radio star Stern and gossip blogger Perez Hilton have been rumored as being in the running to replace Cowell.
"You've just got to know what you're talking about," Cowell said, adding he is "fairly certain" Stern hasn't been approached by "Idol's" producers to serve as judge. "We have to have people on these shows who actually know what they're talking about, rather than just guessing. They've really got to have experience, so that they can not just criticize, but offer constructive advice as well."
Cowell described the casting of Madonna as unlikely, largely since the show requires a serious time commitment and the singer is busy recording, touring and caring for her four children.
"Perez would be funny," Cowell admitted. "He's got good taste in music, he's got a personality, that could work. ... The good news for (the producers) is that there's no shortage of candidates."
Celebrity Families of 2014 [PHOTOS]