The detentions, torture or killings of inhabitants of Tawergha are widespread and sufficiently organized to be considered crimes against humanity, said Human Rights Watch.
Analysis of satellite images show "systematic" destruction of large parts of the town since fighting stopped in mid-2011. A report by HRW said the destruction was an apparent attempt to prevent inhabitants from returning.
HRW blamed most of the damage on armed groups from Misrata, a town less than 20 miles south of Tawergha that was heavily targeted by Gaddafi during the 2011 uprising. Many residents of Tawergha were supporters of Gaddafi and were believed to have been involved in the fighting in Misrata.
About half of Libya's 60,000 internally displaced people are from Tawergha.
The rights group said Libyan authorities should promptly investigate individual Tawerghans alleged to have committed serious crimes, such as unlawful killings and rapes, during the conflict.
HRW also criticized foreign governments who intervened during the revolt against Gadhafi but failed "to challenge effectively the ongoing abuses against Tawerghans and others."
It called on the United Nations to condemn crimes against humanity committed against Tawerghans.