The air and ground assault killed at least 13 of the 200 militants seeking to reclaim part of the island for a Filipino sultan, The New York Times reported.
Last month an estimated 200 people from the southern Philippines landed in a remote area of Saba state on Borneo Island to re-establish a historic claim over the area by the Sultanate of Sulu that ruled the area for centuries.
Borneo Island is divided among Brunei, Indonesia and Malaysia.
The leader of the southern Philippine militant group Moro National Liberation Front said some members of his group were taking part in the fighting and more were planning to join them.
Sporadic fighting was continuing in remote coastal areas of Sabah, in the north of the island.
"The mopping and searching will cover a wider area given there are signs the intruders moved to another location," Chief Ismail Omar of the Malaysian National Police said Wednesday.