Wolfson suffered from colon cancer and Alzheimer's disease, the Miami Herald reported. He died Sunday on the 30th anniversary of his marriage to Patrice Jacobs, daughter of a prominent racing trainer.
A millionaire by the time he was 28, Wolfson was first involved in construction and shipbuilding. In 1960, he founded Harbor View Farm in Florida's Marion County and bred generations of thoroughbred horses, including Affirmed, winner of the 1978 Triple Crown.
In the 1960s, Wolfson was jailed for almost a year for selling unregistered stock. He always believed he had been railroaded.
"I didn't use any fictitious names in selling the stock. No Swiss banks,"' he said. "When I made the sale, I reported it to the SEC. I'd have had to be an idiot to do anything willfully wrong."
After his release, he became a crusader for prison reform.
Swim Week Miami Beach 2014 [PHOTOS]