Mali in January called for military support from former colonial power France. Malian forces needed help to take on al-Qaida and foreign militants that took control over parts of the country following a 2012 coup.
Rice said action in Mali outside normal peacekeeping duties wasn't something for the United Nations to consider.
"The United States and most members of the Security Council take the view that robust counter-terrorism operations in the far north of Mali are not properly or reasonably a U.N. mission," she said.
Malian Ambassador to the United Nations Oumar Daou said he supported a so-called parallel force "whose objective will be to carry out large-scale anti-terrorist and combat operations."
The United Nations approved a similar mission for the Democratic Republic of Congo. Daou said the Malian government was convinced that a parallel force would help with national stability and reconciliation.
The Economic Community of West African States estimates that any U.N. stabilization mission in Mali would require more than 11,000 troops and 1,400 police officers. ECOWAS said several of its member states have come forward to express interest in supporting the mission.