Abbott, the first black female member of Commons, said in a recent Twitter post: "Dubious of black people claiming they've never experienced racism. Ever tried hailing a taxi I always wonder?"
Some drivers said they were angered by the suggestion they ignore and drive past passengers who are black.
Steve McNamara of the Licensed Taxi Drivers Association said a "substantial proportion" of London's 9,000 drivers are minorities.
"At worst she is racist and at best she is stupid in making comments like that, McNamara said. "Either way, she should go."
Abbott initially said her comments were taken "out of context."
On Thursday, the Labor Party issued a statement from Abbott, apologizing "for any offense caused."
Deputy Prime Minister Nick Clegg called her words "stupid and crass."