Carter will leave Sunday and return to the United States Friday after visiting with Castro, other government leaders and Cubans in all walks of life, including dissidents. He said he plans no negotiations although he believes it is time to open the two countries to each other.
U.S. reaction to the trip has been mixed.
The Miami Herald reported Tuesday that two House members from Miami -- Reps. Lincoln Diaz-Balart and Ileana Ros Lehtinen, R-Fla., tried to stop Carter from going by sending a strongly worded letter to the White House earlier this year.
The letter the White House said in part: "We write to request that you deny permission for Mr. Carter to visit the Cuban dictator. While U.S. law authorizes the granting of licenses by the Treasury Department to U.S. officials and member of Congress to visit Cuba on official business for the U.S. government, it does not do so for former presidents seeking to appease anti-American dictators."
The letter also described Carter as "directly responsible for having brought to power the terrorist regime of the Ayatollah Khomeini in Iran."
The letter to President Bush was dated March 22 and noted it wasn't addressed to Treasury Secretary Paul O'Neill whose department is in charge of enforcing the embargo against Cuba "because of Mr. O'Neill's publicly professed opposition to your policy in Cuba."
The previous week, O'Neill had told a congressional committee he would rather use resources to track down terrorists than violators of the Cuban embargo.
The letter was sent before Carter received his license to travel to the communist country. The Bush administration has since expressed hope the former president would bring a strong human rights message to Cuba.
The influential Cuban American National Foundation of Miami met with Carter last week to offer advice. The organization also committed to support lifting the embargo if Cuba agreed to free elections within 12 months, along with other concessions, said Jorge Mas Santos, chairman of the militant group.
There was also speculation Sunday that the release of political prisoner Vladimiro Roca 70 days before the end of his five-year prison term was a goodwill gesture by Castro to Carter. But Roca said he had been told two years ago he would be released 70 days early.
He was imprisoned for his part in publishing a pamphlet calling for more human rights and free elections in Cuba.
President Castro invited Carter to come to Cuba in October 2000 when both were attending the funeral of former Canadian Prime Minister Pierre Trudeau. Carter received a formal invitation from Castro in January. The former U.S. president received the travel license from the Treasury Department April 5.
Swim Week Miami Beach 2014 [PHOTOS]