The prisoners include former Taliban Interior Minister Khairullah Khairkhwa and senior Taliban military commander Noorullah Noori, The Guardian said, citing sources familiar with the talks in the United States and in Afghanistan.
Washington may release former Taliban Deputy Defense Minister Mohammed Fazl -- a "high-risk detainee" held at the U.S. military prison Guantanamo Bay, Cuba, since Jan. 11, 2002 -- but hand him over to another country, possibly Qatar, a wealthy Persian Gulf emirate, the newspaper said.
Fazl is alleged to be behind the killing of thousands of Afghan minority Shiite Muslims between 1998 and 2001.
Other prisoners considered for release include former top Taliban intelligence official Abdul Haq Wasiq and major Taliban financier Mohammed Nabi, The New York Times reported, without saying the agreement in principle had been reached.
The White House had no immediate comment on the Guardian report.
When asked about the prisoner-release possibility Tuesday, White House spokesman Jay Carney said: "As far as releasing Afghan prisoners, we're not in a position to discuss ongoing deliberations or individual detainees. But our goal of closing Guantanamo is well established and widely understood."
It was unclear early Wednesday if the deal would include having the Taliban release U.S. Army Pfc. Bowe Bergdahl, 25, kidnapped in Afghanistan in June 2009. He is the only U.S. soldier held by the Islamist militant and political group.
"To take this step, the [Obama] administration has to have sufficient confidence that the Taliban are going to reciprocate," former Obama administration Afghan peace-process adviser Vali Nasr told The Guardian.
"It is going to be really risky," said Nasr, a Tufts University international politics professor and senior foreign-policy fellow at the non-partisan Brookings Institution think tank. "Guantanamo is a very sensitive issue politically."
Former European Union Afghan envoy Michael Semple, now a Harvard University human rights fellow, told The Guardian he thought releasing the prisoners was politically feasible for President Barack Obama, even in an election year.
"The prospect of ending a costly war in Afghanistan is sufficiently attractive for the Obama administration to move forward with it," Semple said.
Obama declared in June the United States had largely achieved its goals in Afghanistan and would withdrawal U.S. troops from the country by 2014.
"Even if all five of these people they release went straight back to Quetta [a Taliban stronghold city in Pakistan near Iran and Afghanistan] to rejoin a fight, it wouldn't make any real difference," Semple told The Guardian.
A Taliban statement Tuesday said that along with a preliminary deal to set up the office in Qatar, the group asked that Taliban detainees in Guantanamo be released. The statement did not give details about the prisoners the Taliban wanted freed or when the Qatar office would be opened.
The Obama administration reached a tentative agreement with the Taliban in November under which five Afghans detained at Guantanamo would be transferred to Qatar in exchange for the militant group's public renunciation of international terrorism. The deal collapsed after Afghan President Hamid Karzai rejected the terms, The Washington Post reported.