Cole, who had been a judge on the original British version of the TV talent show, was scheduled to be a judge alongside Paula Abdul, Simon Cowell and L.A. Reid when the American edition of the show begins this fall.
TMZ said the show's producers feared Cole's British accident would be difficult for U.S. viewers to understand. They also thought Cole lacked chemistry with Abdul.
Singer Nicole Scherzinger is expected to take Cole's place as judge, the entertainment news Web site said.
Gal Gadot cast as Wonder Woman for 'Batman vs. Superman'
Man spent 15 hours in jail for plugging electric car into an outlet at a school