A team from the Durrell Wildlife Conservation Trust has named the mongoose-like creature Durrel's vonstira after Gerald Durrell, the trust's founder, the BBC reported.
With its marshy home under threat from invasive species and pollution, team members say the animal may be one of the world's most threatened mammals.
After a first sighting in 2004, one of the creatures was captured in 2005 for detailed measurements and blood and tissue samples, which were sent to the Natural History Museum in London along with one dead specimen.
Museum zoologists compared it to its closest relative, the forest-dwelling brown-tailed vontsira, and confirmed it was a new, separate species.
"It was indeed a distinct new species and the specimen we have in the museum is now recognized as the holotype (the specimen from which the species takes its name) so it is available to scientists for research in the future," the museum's Paula Jenkins said.
A discovery of a new mammal species is not common and finding a new carnivore species is "particularly unusual," she said.
"Durrell's vontsira is incredibly rare," she said.
"We know of only two animals in the wild. It has only been found in the wetlands of [Lake] Alaotra in Madagascar, so it lives in a very small area and is consequently vulnerable to the pressures on this threatened habitat."