The decision came Monday after what Karzai described as the end of a two-month grace period he and U.S. President Barack Obama agreed to for completing the full transfer of the prison's control to Afghan forces, The New York Times reported.
Karzai met Sunday with top officials for a status meeting and issued a statement that said the U.S. breached a memorandum of understanding on handling prisoners.
"The meeting began by hearing reports by the attorney general and the military police commander [which said] that even prisoners who were presumed innocent by the court are still being held," the statement said.
U.S. forces also violated earlier agreements by housing newly captured prisoners at the Parwan detention center, the American-built detention center at Bagram Air Base, it said.
U.S. officials have said the memorandum of understanding left the handling of new prisoners captured by U.S. forces open to negotiations.
Talks stalled because of disputes about the release of some prisoners American forces have refused to let go, the Times said.
The status quo was "a serious breach of the memorandum of understanding," Karzai said in the statement, urging Afghan officials to take "urgent measures to ensure a full Afghanization of the prison affairs and a complete transfer of its authority."
A U.S. official told the Times the matter is an "issue of sovereignty" for Afghanistan and "a matter of security" of the coalition troops for U.S. Gen. George Allen, commander of U.S. and NATO troops in Afghanistan now in Washington, D.C., after a scandal erupted over emails he sent to a Florida woman .
"[You] can't just bring these guys in and let them go," the official said.
2014: The Year in Music [PHOTOS]