Seven people were critically wounded.
The attack at Dogon Dawa, initially blamed on the Islamic extremist group Boko Haram, is now believed to have been by robbers seeking revenge for arrests made earlier by vigilantes, the Nigerian newspaper Leadership reported.
Witnesses said 10 gunmen entered Dogon Dawa early Sunday and went to the home of the vigilante chief, Ahmed Aliyu Makama, and killed him and his son.
The attackers then opened fire on a group of worshippers leaving a mosque.
The gunmen killed eight people in the mosque and others during a house-to-house search for vigilantes.
Police officials said a vigilante group from the village had gone to another village, about 20 miles away, Saturday and arrested four people suspected of being members of the armed group.
In a separate incident Sunday, a bomb exploded along a road in Maiduguri in Borno State near the Cameroon border.
Lt. Col. Sagir Musa, a spokesman for the Joint Task Force, said the bomb targeted JTF forces but went off without injuring anyone.