He said the Taliban's unwillingness to meet with a Pakistani delegation influenced his decision.
"At this point, we're inclined not to go, but we'll keep communicating," he said in response to question about whether he would go later.
"This official invitation has measured significance. It is some attempt to reach out and to dialog. However, questions still remain. A delegation from Pakistan today have sought to meet with the Taliban, aiming to convince them to turn over Osama bin Laden, open the camps for inspection, and release the eight Christian aid workers, two of whom are Americans, whose parents we talked with on yesterday (Thursday). It appears that the Taliban were not receptive to this message so far," his reply to the Taliban said.
"This is a disappointing sign. As President Bush and Secretary of State Colin Powell have stated, they (the Taliban) must turn over the suspects and they must free the captives and join the global war on terrorism," it said. "The alternative is war and bloodshed and the Afghan people have already suffered much too much."
He said there are 500,000 orphans and several million widows in the country, which has a badly damaged infrastructure and millions of people attempting to leave.
Jackson, who is attending a convention in central Flolrida, said the clergy worldwide can play an important role in promoting peace and in the humanitarian dimension of the conflict. He said he would like to see the suspects go before a world court.
He said the United States could have responded to the terrorist attacks of Sept. 11 with blind revenge, but "Instead, we saw disciplined presidential leadership. We chose restraint and reason and coalition building. You cannot fight this terrorist war unilaterally."
Jackson has previously led religious delegations on humanitarian missions to free people held hostage. During the Reagan administration, he visited Syria and brought home a U.S. Navy lieutenant.
He was able to persuade Saddam Hussein to release 600 women who were being held in Iraq. He also met with Slobodan Milosevich in Yugoslavia and facilitated the release of three soldiers.
In each case, the U.S. government recommended that he not go, but did not stop him.
The aid workers held in Afghanistan are being tried for preaching Christianity in the Muslim country.
Celebrity Couples of 2014 [PHOTOS]