The rights group said around 4,000 prisoners are in government custody without charge. Another 4,000 or so are held by paramilitary groups that have no legal authority to hold prisoners.
Lawmakers this week threw their support behind a unity government presented by Libyan Prime Minister Ali Zidan. He told the rights group that, as a former rights activist, he was committed to upholding the rule of law.
Eric Goldstein, deputy Middle East director at Human Rights Watch, said Zidan's government has an opportunity to honor its pledges by reviewing prisoner issues.
"The new government can start on the right foot by addressing the detainee crisis," he said in a statement from New York.
Former Libyan intelligence chief Abdullah al-Senussi was extradited from Mauritania to Libya in early September. He, along with Gadhafi's son Saif al-Islam Gadhafi, is wanted by the International Criminal Court on charges of committing crimes against humanity during last year's civil war.
Fatou Bensouda, chief prosecutor at the ICC, told the U.N. Security Council recently that Tripoli was advocating for its right to try former regime officials on war crimes charges in national courts.
"I take full responsibility for the safety and security of all Libyans equally, including those who sided with the former regime and their families," the prime minister told the rights group.
Both men are in custody in Libya.