The men were all residents of Britain but are not British citizens, The Guardian reported.
The United States reportedly offered to return nine men from the U.S. base in Guanatanamo Bay, Cuba, in June. The conditions included that the men would not be allowed to leave Britain, would be monitored and would be banned from participating in any advocacy of violent or extremist activity.
The head of counter-terrorism at the Home Office said he could not justify the intelligence resources that would have been required for detainees who are not British citizens, the report said.
"They do not pose a sufficient threat," he said.
The only one in the group the government is considering accepting is Bisher al-Rawi, a native of Iraq who first came to Britain in 1985. A businessman, Rawi had reportedly been cooperating with British intelligence before he was arrested in Gambia.
The families of the detainees have appealed the government's decision.
2014: The Year in Music [PHOTOS]