"It will happen to me one day. Someone will say, 'It's time for you to move on to another show,' and I'll go, 'Fine,'" Cowell told the London Sun. "I wouldn't point fingers at anybody -- these things happen."
Cowell's friend and fellow "X Factor" judge Louis Walsh and host Kate Thornton were fired recently, which prompted another judge, Sharon Osbourne, to call Cowell "a coward" for not delivering the pink slips himself.
But Cowell contends the firings were not his decision.
"Richard Holloway, the executive producer, said, 'I've known Louis for longer than you actually -- 20 years -- and if I'm going to call Kate I'm going to have to tell Louis,'" said Cowell. "If it had been my decision I would have told them directly. That's the way I am."
Still, he said, he should have called his pal, adding, "He has every reason to believe he's been stabbed in the back."
Cowell said he tried to call Thornton but she hasn't returned his call.