Cole, who had been a judge on the original British version of the TV talent show, was scheduled to be a judge alongside Paula Abdul, Simon Cowell and L.A. Reid when the American edition of the show begins this fall.
TMZ said the show's producers feared Cole's British accident would be difficult for U.S. viewers to understand. They also thought Cole lacked chemistry with Abdul.
Singer Nicole Scherzinger is expected to take Cole's place as judge, the entertainment news Web site said.
Celebrity Breakups and divorces of 2014 [PHOTOS]