Annan told reporters after weekend talks he had decided to leave the matter with the head of the African Union, former Tanzanian president Benjamin Mkapa, the Johannesburg Star reported.
"You do not need two mediators," Annan said.
There had been diplomatic speculation the United Nations was preparing to offer anything from a plan for Zimbabwe's economic recovery to an exit plan for Mugabe in exchange for amnesty against prosecution for malfeasance.
Mugabe's early years in power saw the country improve rapidly, but conditions have since degraded, with agricultural production plummeting amid government seizures of white-owned farms.
2014: The Year in Music [PHOTOS]