A 15-member Thai delegation and representatives of the rebel Barisan Revolusi Nasional -- National Revolutionary Front -- and several other insurgent groups met in Kuala Lumpur as a first step toward more detailed negotiations.
Paradorn Pattannathabutr, secretary-general of Thailand's National Security Council and who is attending the talks, said he was optimistic discussions will lead to "an atmosphere that yields solutions or yields progress that would result in solutions."
But he said that establishing a peace process would take time, the Bangkok Post newspaper said.
"Today's main focus (of the talks) is to reduce violence," Paradorn said.
"We will focus on building mutual trust and good relations. I'm confident that they will communicate our message to their militants but because the BRN is a large organization we have to give them some time."
Ahmad Zamzamin, a former senior aide to Malaysian Prime Minister Najib Razak, is helping direct the first formal peace talks to end the decades-long conflict.
The Bangkok Post report didn't confirm which rebel leaders from the BRN are attending the talks, although one of the attendees is believed to be Hassan Taib, who has been involved in setting up the meeting.
Fighting between government forces and rebels has claimed at least 5,300 lives in the Muslim-majority provinces of Yala, Pattani and Narathiwat, an area that borders Malaysia.
The provinces are under emergency law in an attempt by police, the military and the paramilitary Royal Thai Rangers to stem violence against Buddhist monks, school teachers and village officials as well as security forces.
More than 60,000 soldiers, including the controversial Rangers, a paramilitary group originally set up by the central government in the 1970s to fight communist insurgents in the mostly isolated jungles of southern Thailand, are on station.
The latest fatal attack on government forces killed three rangers and wounded five when a roadside bomb exploded in a remote district in Narathiwat.
The 15-pound bomb in a gas cylinder was planted near a power pole and detonated remotely.
Local army commander Lt. Gen. Udomchai Thammasaroraj said he believed the attack was intended to disrupt the peace process, the Post report said.
"We suspect this was the work of local militants who want to discredit the peace talks under way in Kuala Lumpur," he said.
While all sides hope the talks in Malaysia lead to more detailed negotiations within Thailand, a report by the BBC last month said there are questions over the authority of rebel leaders who would be signing any peace deals.
Taib, who abandoned the armed struggle against the Thai state in the 1990s, signed the talks agreement with Pattannathabutr earlier this year.
The agreement is significant because it commits the Thai government to recognizing the rebels as a negotiating partner with political demands that must be accommodated, the BBC report said.
But Taib is from one of the many factions of PULO -- the Pattani United Liberation Organization -- which has been more of less dormant in recent years, although Taib signed the deal for talks as leader of the umbrella group BRN.
Part of the problem has been a lack of clarity by rebel groups over their political demands, which have ranged from autonomy for Muslim-majority areas and provinces to complete political separation.
Clashes between security forces and rebel groups happen daily.
But only the most deadly skirmishes are reported and gain attention internationally, such as the early February storming by rebels of a government military base in Narathiwat.
Soldiers killed 16 rebels before driving off the attackers during the night-time raid.
No government troops were killed but several days earlier five soldiers were killed by suspected militants in a bomb attack in neighboring Yala province.