U.S. officials have determined the five, Chinese Uighurs, pose no terror threat and should be released from detention in the U.S. facility at Guantanamo Bay, Cuba.
But the Uighurs have refused resettlement to the Pacific island of Palau, saying they have no cultural ties there, and Congress has forbidden the release of any prisoners into the United States, where they would like to go at least temporarily, SCOTUSBLOG.com reported.
Justice Stephen Breyer, joined by the three other members of the court, wrote separately to agree with the rejection of review. Breyer said there was no obstacle to the men's release.
A federal judge ruled earlier that the five had the power to get out of Guantanamo anytime they wanted, they just had to accept resettlement to Palau.
Lawyers for the five detainees insist the case is about constitutional issues, but the U.S. government says the case is about their wanting to be released into the United States.
At least four Uighurs at Guantanamo have been sent to Bermuda, while at least six have accepted an invitation to move to Palau, which also offered to take in six of seven other Uighurs at Guantanamo. Five Uighurs have been sent to Albania.
The Muslim Uighurs were arrested in Afghanistan in 2002 after the U.S. sent troops there and were accused of being members of a group being trained in terror tactics.
2014: The Year in Music [PHOTOS]