Speaking to the BBC in an interview, Gilani said the two sides need to trust each other.
Bilateral relations, already rocky, have plunged since a NATO airstrike last month in Pakistan's Mohmand agency in which 24 Pakistani soldiers died. To express its displeasure, Pakistan has closed its routes to supply coalition forces in Afghanistan, and kicked the United States off the Shamsi airbase in Balochistan province.
"Yes there is a credibility gap, we are working together and still we don't trust each other," Gilani told the BBC.
"I think we have to improve our relationship so that ... we should have more confidence in each other."
The BBC quoted Gilani as saying Pakistan may continue to block NATO convoys into Afghanistan for several weeks. He also refused to rule out closing Pakistan's airspace to the United States, the British news network said Sunday.
On other issues, Gilani denied rumors President Asif Ali Zardari had suffered a stroke and that the army was trying to oust him. Zardari is recuperating in a hospital in Dubai, and Gilani said he would need two weeks' rest before returning home.