Picower, who lived in Palm Beach, Fla., before his death from natural causes in 2009, withdrew $3.4 billion from his account with Madoff over the years after investing $650 million, The Miami Herald reported Tuesday.
Madoff has said several of his investors and his bank must have known or at least been suspicious that he was running a Ponzi scheme.
The recently released "The Wizard of Lies; Bernard Madoff and the Death of Trust," by New York Times reporter Diana Henriques, however, says Madoff claims he once gave Picower a small portion of a withdrawal request with no explanation.
William Zabel, Picower's attorney, said the claim was false.
"Jeffry Picower was neither complicit in nor did he know of Madoff's Ponzi scheme," Zabel said.
Madoff, he said, "is pointing the finger at someone who was never charged with wrongdoing, no longer is living and can't defend himself."
The Picower estate gave up $7.2 billion in a settlement pursued by court-appointed trustee Irving Piccard on behalf of victims of the fraud, which lost an estimated $65 billion.
Madoff was sentenced to 150 years in prison for perpetrating the fraud.
Madoff has also said the reason investors are settling charges with investigators is because they are guilty of at least benign complicity in the scheme. However, the book also says Madoff's credibility on any claim is pretty thin, the Herald said.
Celebrity Couples of 2014 [PHOTOS]