Prime Minister Najib Razak's governing Barisan Nasional -- National Front -- party won 133 of the 222 seats in Parliament but the result was quickly decried by opposition leaders.
It was the National Front's 13th consecutive general election victory since Malaysia gained independence from the United Kingdom in 1957.
The Front has held as much as 90 percent of parliamentary seats in the country of 28 million people with Islamic, Chinese, Indian and Western cultures.
But the party -- a loose coalition of more than 13 small and regional political parties that was formed in 1973 -- lost its two-thirds majority of seats in 2008 and failed in Sunday's polling to recoup the loses.
The election result brought a swift condemnation from the major opposition coalition of three parties, Pakatan Rakyat -- People's Alliance -- led by Anwar Ibrahim.
Ibrahim defied police warnings that his protest in the MBPJ Stadium in Kelana Jaya near Kuala Lumpur was illegal.
During the night-time protest, he called for rotating demonstrations around the country.
Ibrahim, 65, reportedly told the crowd that "BN has robbed the rights of the people," Malaysia's national news agency Bernama reported. "We will prove that they lied in 30 parliamentary seats."
But Election Commission Chairman Tan Sri Abdul Aziz Mohd Yusof said the "unhappy candidate" could apply for a review of the result by making a petition after the result was published officially by the commission.
"After the result is gazetted, he has 21 days to file a petition and if there is proof and the judge admitted the existence of corruption, fraud or illegal acts, the High Court judge will declare the election result invalid.
"Only then a re-election will be called. A rally such as the one tonight won't change anything," Abdul Aziz told Radio24.
Aziz also rejected opposition claims that the commission helped parties to win, Bernama reported.
A report by the BBC said up to 40,000 protesters attended the stadium rally, with many dressed in mourning black.
Their complaints ranged from the use of indelible ink in polling booths. Although it is meant to prevent multiple voting, it could be washed off easily.
Also, there were many accounts of a government scheme to fly tens of thousands of "dubious," possibly foreign, voters to vote in some constituencies, the BBC report said.
In a White House statement, U.S. President Barack Obama congratulated Razak on his election victory and the people of Malaysia, who turned out in record numbers to cast their votes.
The statement praised the parties of the opposition coalition on their campaigns that helped to underpin democracy in their country.
But the White House also said "it is important that Malaysian authorities address (polling) concerns that have been raised."