Duterte made the comment during an oath-taking ceremony for new appointees in Manila's Malacañang Palace.
"The U.S. Special Forces, they have to go. They have to go in Mindanao. There are many whites there," Duterte said before speaking on the threat of Abu Sayyaf, which he recently vowed to "erase" and "devour."
"If they see an American, they would kill him. They would demand ransom then kill him. Even if you're a black or white American as long as you are an American," Duterte said of Abu Sayyaf.
The Philippine Constitution prohibits foreign troops from taking part in direct combat operations. U.S. troops in the Philippines have assisted through training and sharing information.
Duterte briefly met with U.S. President Barack Obama last week during the Association of Southeast Asian Nations summit in Laos. The exchange was not scheduled due to a previous comment by Duterte in which he referred to Obama a "son of a bitch."
Duterte on Monday said he could not bring up the subject of removing U.S. troops from the Mindanao Island because he does not want further problems with the United States.
"I just could not say it because I don't want a rift with America. The situation there will be more volatile. If they are seen there, they will really be killed," Duterte said.