Abbott, the first black female member of Commons, said in a recent Twitter post: "Dubious of black people claiming they've never experienced racism. Ever tried hailing a taxi I always wonder?"
Some drivers said they were angered by the suggestion they ignore and drive past passengers who are black.
Steve McNamara of the Licensed Taxi Drivers Association said a "substantial proportion" of London's 9,000 drivers are minorities.
"At worst she is racist and at best she is stupid in making comments like that, McNamara said. "Either way, she should go."
Abbott initially said her comments were taken "out of context."
On Thursday, the Labor Party issued a statement from Abbott, apologizing "for any offense caused."
Deputy Prime Minister Nick Clegg called her words "stupid and crass."
NBC reportedly holds celebs hostage to Jimmy Fallon's show
Ray Liotta sues skin care company over use of likeness