Hoffa used the derogatory term during his remarks to introduce Obama at a Labor Day rally in Detroit.
Saying the Tea Party launched a "war on workers," Hoffa said: "President Obama, this is your army. We are ready to march. ... Let's take these son-of-a-bitches out and give America back to an America where we belong."
Hoffa said he stands by his remarks, ABC News said Tuesday.
Amy Kremer, who leads the Tea Party Express, called on Obama to "condemn this inappropriate and uncivil rhetoric," USA Today reported.
"Jimmy Hoffa's remarks are inexcusable and amount to a call for violence on peaceful Tea Party members, [including] many Teamster members," Kremer said in a written statement.
Tea Party activists also expressed displeasure for Vice President Joe Biden, who told a Labor Day rally in Cincinnati that they were "the only folks keeping the barbarians from the gates."