Malaysia reported that the Filipinos were surrounded near Lahad Datu, Sabah, by Malaysian security forces and ordered to surrender.
Malaysian police Inspector-General Tan Sri Ismail Omar said, the "situation is under control and the people have nothing to worry."
He said the armed Filipinos, believed to be members of "a militant group" in Mindanao, arrived on the shore of Lahad Datu by boat, The Philippine Star reported Thursday.
Malaysia and the Philippines since August 2010 committed themselves to enhancing maritime border security and to address illegal immigration.
Malaysian Defense Minister Datuk Seri Ahmad Zahid Hamidi said after meeting with Philippine Defense Secretary Voltaire T. Gazmin that Malaysian security forces were monitoring the waters east of Sabah to maintain security and prevent illegal immigrants from crossing the border.
Filipino militants have demanded official recognition as members of the Sultanate of Sulu, which carries influence among some southern Filipino Muslims, along with an assurance from Malaysia that their members who enter Borneo wouldn't be forcibly returned to the Philippines.
The impasse sparked one of the biggest security scares in recent years in Sabah, which is less than an hour by speedboat from southern Philippine provinces that have suffered a Muslim separatist insurgency for years.
The intruders landed in Sabah's largely rural, coastal district of Lahad Datu, Omar said, in the aftermath of "troubles in the southern Philippines."
Security on the Malaysian maritime border with the Philippines increased problems for the Malay Sabah province, where tens of thousands of Filipino have tried to immigrate over the past several decades.
Not all Filipino immigrants are seeking a better life; a decade ago, gunmen from Mindanao slipped twice into Sabah and abducted people, including tourists from a diving resort, for a ransom. Last November two Malaysians were abducted from a plantation in Lahad Datu and Malay authorities say they were subsequently transferred to Mindanao.
Philippine Department of Foreign Affairs spokesman Raul Hernandez told journalists in Manila that Filipino defense and security officials were in communication with their Malaysian colleagues over the encounter.
Malay Prime Minister Najib Razak, who was visiting Sabah when the incident occurred, said "The government is choosing to handle the issue through negotiation and to get the group to leave peacefully to prevent bloodshed."
The identity of the Filipino militants has yet to be determined, with some media reports speculating that they belong to various Filipino Muslim guerrilla factions fleeing from recent violence there but some Malay officials have speculated that they might in fact be personal security guards for a Muslim royal family in the southern Philippines who apparently failed to inform Malay authorities that they intending to travel to Sabah.