Afghan security forces shot and killed two attackers in a gunfight after the Monday explosion, CNN reported, quoting Gen. Hamid Wardag, a commander of the local Afghan National Army.
Three security guards and a civilian were killed in the blast, the report said. Five more people were injured.
It was not clear how many insurgents were involved in the gunfight but at least two were killed.
The U.N. refugee agency is also located near the blast site.
The IRD is involved in helping rehabilitate areas hit by the war.
The Kandahar region has had an escalation of violence lately.
On Oct. 22, three U.S. troops died in an explosion set off by an improvised explosive device.
On Saturday, a suicide car bomber rammed the vehicle into a NATO armored bus in Kabul, killing 17 people including 13 foreigners, most of them Americans, and four Afghans.