The soldiers, calling themselves the National Committee for the Establishment of Democracy, said the government and other institutions of power had been dissolved after an attack on the presidential palace in Bamako. They said a curfew was in place and the constitution had been suspended.
President Amadou Toumani Toure was reported to be safe and not in the custody of the rebels, the BBC reported Wednesday.
A spokesman for the soldiers said the mutiny was motivated by complaints the government was not giving troops sufficient arms to fight a rebellion being waged by ethnic Tuaregs in the West African country. "Considering the incapacity of the regime in effectively fighting against terrorism and restoring dignity to the Malian people, using its constitutional rights, the armed forces of Mali, along with other security forces, have decided to take on their responsibilities to put an end to this incompetent regime of President Amadou Toumani Toure," spokesman Amadou Konare was quoted as saying by CNN.
Heavy gunfire was reported in Bamako, the capital, between mutinous soldiers and troops loyal to the government.
The United States issued a statement strongly condemning the military coup. "We call for the immediate restoration of constitutional rule in Mali, including full civilian authority over the armed forces and respect for the country's democratic institutions and traditions," the White House said.
"The United States stands by the people of Mali and the legitimately elected government of President Amadou Toumani Toure."
The African Union and United Nations also expressed support for the democratically-elected president.
Fighting in Mali attributed to Tuareg rebels erupted in January when the rebels returned from fighting alongside forces loyal to former Libyan leader Moammar Gadhafi.
The conflict, following years of relative peace, rekindled ethnic tensions between northern and southern Malians.
The U.S. State Department warned last week of a threat of attacks and kidnapping of westerners traveling in and around Mali. The warning stated armed groups in the region were engaged in ongoing battles with the Malian military.
"The conflict has sparked civil disturbances and attacks against ethnic minority groups in Bamako and other cities in the south," the warning read.