The town's residents demonstrated during the demolition and threw stones at police, forcing them to use tear gas to disperse the crowd, Ynetnews.com said.
Many police remained in the town fearing riots may erupt after the structure was pulled down, the Web site said.
During the morning five local residents were detained, the site said.
Police Commissioner Dudi Cohen praised the police forces calling the operation "complex and sensitive."
"We acted with resolve to enforce the rule of law and relayed a message that Israel Police will not ignore illegal activity while remaining sensitive to the Muslims' feelings," Yohanan Danino, police chief of the southern district, told reporters.
Danino said all legal channels were exhausted before the decision was made to demolish the mosque, which was constructed without the necessary permits on public property, the Web site said.
Mayor Faiz Abu-Seheban said the mosque's demolition is a "crime committed by the State and government against an entire public. I am shocked and furious."
He said the town's residents would declare a general strike in protest.