The Thursday attack occurred outside the airfield in southern Kandahar hours after the U.S. official had departed from the area, CNN reported.
CNN quoted Panetta as saying one of those killed was an American. The other two were Afghan civilians, the report said.
The attack in the evening targeted an MRAP, or a mine-resistant ambush protected vehicle, as it was entering the base. It also wounded 18 other Afghan civilians, the report said.
CNN said the Taliban claimed responsibility, with its spokesman Qari Yousuf Ahmadi saying in an e-mail that the attack was carried out by a "brave Taliban fighter." However, the message did not say if the attack was connected to Panetta's visit.
A U.S. official traveling with Panetta, however, said the secretary was never in danger.
Panetta arrived in Kabul Wednesday and The New York Times said his visit to Kandahar Thursday was not announced prior to the trip.
The Times reported Panetta, during a news conference with Afghan President Hamid Karzai, said the bombing killed one American and wounded three others.
In other developments, the Times quoted Afghan officials that Asadullah Khalid, Afghanistan's spy chief, was transported to the United States for treatment after being wounded in an assassination attempt last week. Khalid reportedly suffered abdominal wounds.
Panetta announced that U.S. President Barack Obama had invited Karzai to come to Washington early next month to discuss a "shared vision of Afghanistan beyond 2014," the Times reported.
U.S. and NATO forces are scheduled to end their combat operations by the end of 2014. After 2014, there could a small U.S. military presence to train and support Afghan security forces.
"We are committed to respecting the sovereignty of Afghanistan," Panetta was quoted as saying. "We believe that we can achieve the kind of agreement that would be acceptable to both countries."
Agreeing, Karzai said any agreement that respects "Afghan sovereignty and Afghan laws" is one he can present to his public "with ease."
2014: NFL Cheerleaders [PHOTOS]
Syrian Al Qaida group executes Lebanese soldier