The fighting has been taking place in Banki, near the Cameroon border, and many Cameroon villages have been deserted, Voice of America reported.
Residents near the border said Nigerian troops fired indiscriminately at fleeing militants, leading to widespread panic in the area.
Both the infantry and air force were used in the fight, and houses and property were destroyed.
The Boko Haram members attempted to escape into Cameroon with their firearms, but the Cameroon military resisted.
Cameroon Defense Minister Edgar Alain Mebe Ngo'o said Thursday the government has deployed troops to fight Boko Haram members who may cross the border.
Boko Haram is responsible for several recent attacks in Borno state in recent months.