The Nebi Akasha mosque, in an ultra-Orthodox Jewish neighborhood, was abandoned during the 1948 Arab-Israeli war and recently turned into a municipal storage facility, Haaretz reported.
Jerusalem Mayor Nir Barkat said such actions will not be tolerated.
"We must show zero-tolerance for any kind of violence and maintain the coexistence in the city," Barkat said.
Neighborhood residents denounced the perpetrators.
"These are crazy people who are set to ignite a religious war in the city," one man told Ynetnews.com. "I hope the Arab population understands that this is a minority. There are a lot of hooligans here."
Graffiti spray-painted on the walls of the mosque defamed Islam and Arabs. Authorities also found the phrase "price tag" -- an apparent reference to a dissident Jewish group that has been attacking government and military as well as Palestinian targets.
Police are investigating.
Wisconsin business offering 'therapeutic cuddling' forced to close
Britney Spears debuts 'Perfume' video