Eight people were arrested and six officers were injured, the Sydney Morning Herald reported.
Police estimated about 500 people took part in the protest, which included a confrontation outside a building that houses the U.S. consulate.
Demonstrators carried signs with messages such as "Behead all those who insult the prophet," as they protested a film produced in the United States that depicts the Prophet Muhammad as a womanizer and pedophile. Outrage over the film led to demonstrations -- some featuring violence -- at U.S. embassies and consulates in dozens of countries this week.
Abdullah Sary, one of several people who addressed the protesters, said the demonstration had intended to be non-violent, but that people became angered by the police use of gas and dogs.
Protesters attacked a man from outside the group when he shouted "Shame on you for promoting murder in the name of religion."
Police pulled him away from the area.
Keysar Trad, founder of the Islamic Friendship Association of Australia, condemned the protest.
"It's counterproductive and does nothing to uphold the prophet's honor as they claim," he said. "In fact it does the exact opposite."