The operation followed Palestinians stoning Jewish worshippers at Judaism's Western Wall, which is below the mosque, Ha'aretz reported.
Meanwhile, a suicide bomber riding a bicycle blew himself up near the greenhouse area of Kfar Daron in the center of the Gaza Strip. The suicide bomber died from the explosion, but no one else was hurt, the Jerusalem Post reported.
The Jerusalem clash followed a week of Palestinian protests in the West Bank against a barrier Israel is building in and around occupied territory.
Officers tossed stun grenades after Muslim worshippers "started rioting" at the end of Friday prayers near the holy site Muslims call al-Haram al-Sharif and Jews revere as the Temple Mount.
"Hundreds of Muslims threw rocks and rioted," police spokesman Gil Kleiman said. "Police entered the Temple Mount."
An ambulance driver said four Palestinian demonstrators were injured but none seriously, and police said three officers were slightly hurt.