There was no word on the exact cause of his death, CNN reported Wednesday.
Born in the Virgin Islands, Griffith moved to New York at the age of 19 and had his first big breakthrough several years later when he won a Golden Gloves title.
He made headlines in 1962 for his pummeling of Benny Paret after Paret called him a derogatory name in Spanish. Paret died of his injuries 10 days later.
A 2005 documentary "Ring of Fire" recalled that bout and how it haunted Griffith for years but he kept on fighting.
By the end of the 1960s, Griffith had won two middleweight championships in addition to his titles in the lighter welterweight division.
Griffith retired with a record of 85 wins, 24 losses and two draws, the International Boxing Hall of Fame reported. He was inducted into its ranks in 1990.
Megyn Kelly: Santa Claus and Jesus are both white men
Susan Sarandon: I was stoned at almost every awards show