The Nebi Akasha mosque, in an ultra-Orthodox Jewish neighborhood, was abandoned during the 1948 Arab-Israeli war and recently turned into a municipal storage facility, Haaretz reported.
Jerusalem Mayor Nir Barkat said such actions will not be tolerated.
"We must show zero-tolerance for any kind of violence and maintain the coexistence in the city," Barkat said.
Neighborhood residents denounced the perpetrators.
"These are crazy people who are set to ignite a religious war in the city," one man told Ynetnews.com. "I hope the Arab population understands that this is a minority. There are a lot of hooligans here."
Graffiti spray-painted on the walls of the mosque defamed Islam and Arabs. Authorities also found the phrase "price tag" -- an apparent reference to a dissident Jewish group that has been attacking government and military as well as Palestinian targets.
Police are investigating.