Human Rights Watch said despite the National Transitional Government's commitments to stop abuses, it documented killings, torture and forced displacement by militias of innocent civilians or people suspected of sympathies with the late leader Moammar Gadhafi.
Human Rights Watch said it wants the U.N. Human Rights Council to be more forceful in securing a meaningful response from the new government.
Libya's friends, especially those that supported the NATO military intervention last year, should approach Libya at the highest levels of government and insist on continued monitoring and involvement by the Human Rights Council, Human Rights Watch said.
Tripoli has resisted initiatives for independent monitoring of the promised return to normalcy, surrender of weapons by militia groups and lawful treatment of Gadhafi loyalists and suspected sympathizers of the former regime.
"Libya's unwillingness to support outside monitoring by the council suggests it has something to hide, rather than the transparency that is needed after four decades of dictatorship and eight months of war," Human Rights Watch Geneva Director Julie de Rivero said.
"Outside monitoring will help the central government rein in abusive militias and build the rule of law."
Critics say a robust response from the EU and NATO is also required to convince Tripoli it needs to do more to rein in the militias.
British Foreign Office Minister of State David Howell told Parliament that Libya faced "a lot of hard work ahead, including disarming militias, restarting the economy and building government institutions, but in our view Libya's future is potentially brighter than it was a year ago. It will take time."
A U.N. Human Rights Council report cited arbitrary arrests, torture and killing of Gadhafi loyalists and civilians.
The widespread and systematic nature of the abuses indicates that crimes against humanity have been committed, the report said.
Human Rights Watch said it had documented torture, killings and forced displacement of more than 35,000 people.
If Tripoli fails to act, Human Rights Watch said, the International Criminal Court still has jurisdiction to act against abuses in Libya.
"Libya's local and national authorities should know that the International Criminal Court can still investigate and prosecute serious crimes taking place in Libya, even after Gadhafi has gone," de Rivero said.
Amnesty International urged the United Nations to appoint a special rapporteur to monitor the evolving human rights situation in Libya.
It said there is urgent need also to establish vetting measures to ensure those responsible for serious human rights violations or abuses are not integrated into the Libyan security, military and police forces or other similar bodies.
Analysts said EU leaders appeared reluctant to displease the transitional government amid continuing worries over finding substitute sources for Iranian crude oil.
Libya's oil exports are expected to return to full pre-war levels next month, Libyan oil industry sources said.
Sanctions banning oil imports from Iran are to set to take effect in July but Tehran already has cut off supplies to several European buyers and found new markets for its oil.
Libya plans to export almost 1.4 million barrels of oil per day in April. If that level is achieved, Libyan exports will exceed deliveries in February 2011 before the NATO-led intervention ousted Gadhafi.
Key EU members including Germany and Italy are anxious to continue a softer approach toward Tripoli, analysts said.
Meanwhile, an uncertain security situation has led to cancellations of business tours to Libya and European businesses have scaled down expectations of an early return to normal operations in the North African country.