Amnesty International, in a 27-page paper published Thursday, highlights what it describes as a pattern of beatings and ill treatment of captured soldiers who fought in support of Moammar Gadhafi.
Hassiba Hadj Sahraoui, Amnesty International's deputy director for Middle East and North Africa, said in a statement, there is a risk that arrests and torture might become ingrained in Libya.
"We understand that the transitional authorities are facing many challenges but if they do not make a clear break with the past now, they will effectively be sending out a message that treating detainees like this is to be tolerated in the new Libya," she said.
Gadhafi's regime collapsed in August after rebel forces seized control of Tripoli. Gadhafi is wanted by the International Criminal Court for crimes against humanity committed since February.
Amnesty International said it documented the arbitrary arrest of around 2,500 people since late August. At least two guards told the rights group they beat detainees to get "confessions" out of them more quickly.
The rights organization called on the Transitional National Council to make sure it was operating according to the rule of law.
"The authorities cannot simply allow this to carry on because they are in a 'transitional' phase," noted Sahraoui. "These people must be allowed to defend themselves properly or be released."