White died in Denver Monday morning at 84 following complications with pneumonia. The Supreme Court said it would announce funeral arrangements as soon as possible.
White was the last of the living retired justices.
He served 31 years on the court, from 1962 to 1993. Only eight justices served longer in the two centuries of the court's history.
He didn't much like the press, and hated his college football nickname of "Wizzer."
A waitress once asked him if he was "Wizzer White." White growled back at her, "I used to be."
He was also intensely private. He reportedly spent his 25th anniversary on the court shredding his private papers with his clerks.
Despite his gruff exterior, other justices saluted him as a warm human being.
"He was a good colleague and a great friend," Chief Justice William White said in a statement. "He came as close to anyone I have known to meriting Matthew Arnold's description of Sophocles: 'He saw life steadily and saw it whole.' All of us who served with him will miss him."
White was the only former professional football player to become both a lawyer and justice.
"Byron White was already a national hero to sports fans when I first met him in Pearl Harbor during World War II," Justice John Paul Stevens said Monday. "I knew immediately that he was the kind of person that I would want as a friend .... He was the kind of person for whom respect, admiration and affection continue to increase as you learn more about him. He was a true hero during his naval service, a brilliant student and law clerk, an outstanding member of the profession, both in private practice and as a public servant, and a great judge. He was also blessed with an exceptionally loving bride and a fine family of which he was justly proud. I will miss him."
"Justice White was an extraordinary man," Justice Sandra Day O'Connor said. " ... The people who reach the top in several fields are few, indeed. Justice White was one of the few and was an American original."
Justice Antonin Scalia remarked about White's "painfully firm handshake: You had to squeeze back hard or he would hurt you."
He added, "If there was one adjective that never could, never would, be applied to Byron White, it is wishy-washy .... His former colleagues have missed him since his retirement nine years ago; we will miss him more now. May he rest in peace."
Justice Anthony Kennedy was equally warm in his praise.
"The court must strive in all it is and all it does to reflect the integrity and the strength of a nation dedicated to the cause of freedom," Kennedy said. "Bryon White was himself a remarkable personification of these values and this purpose .... He honored the United States and its people by his splendid, vibrant human spirit."
"Justice White was a welcoming colleague and a solid friend," said Justice David Souter. "Like the others here, I will miss him."
"He was a great man, an outstanding member of the court and a wonderful friend," Justice Clarence Thomas said in his statement. "Virginia (Thomas) and I extend our heartfelt sympathy to (White's wife) Marion and her family. We will keep them in our thoughts and prayers."
Justice Ruth Bader Ginsburg, who filled White's seat when he retired, issued a statement several pages long.
Quoting the late Justice Potter Stewart, who knew him at Yale Law School, on White's academic excellence coupled with athletic brilliance, Ginsburg said White "was, in reality, both Clark Kent and Superman."
Justice Stephen Breyer, the most junior member of the court, had one of the briefest statements: "I was lucky to have come to know Justice White in his later years. Justice White was a great judge and a thoroughly decent man -- forceful, engaging and strongly committed to public service."
2014: The Year in Music [PHOTOS]