Ed Parker, the lead of the Virgin Money South Pole Allied Challenge, called off the race after five days.
"We have had a tricky couple of days. The weather remains good but the terrain is very difficult, far harder than we were anticipating and because of various factors, I have decided to suspend the race," he said on the Walking with the Wounded website Saturday.
Parker said he is determined that all the members will make it to the pole together, but the teams will not race and will journey the last 70 miles together.
The participants are set to reach the pole by Friday or Saturday, CNN reported.
Prince Harry, the British royal who served in Afghanistan, has been an active member of the organization and took part in the race to the North Pole before withdrawing to attend his brother's wedding.
"True Blood" star Alexander Skarsgard is an honorary member of the U.S. team and Dominic West, who starred in "The Wire," joined veterans from Australia.