Silber's daughter, Ruth Silber-Belmonte, told The New York Times he suffered from kidney disease and died at his home. He was 86.
Silber spent 25 years as president of B.U. and seven more as chancellor before retiring in 2003.
A native of Texas, Silber graduated from Trinity University in San Antonio and received a doctorate in philosophy from Yale. He returned home to teach at the University of Texas in Austin.
During a year teaching in what was then West Germany on a Fulbright fellowship, Silber learned that his Presbyterian father was also a Jewish refugee from the Nazis and that an aunt died in the gas chamber at Auschwitz.
Back in Texas, Silber rose to be dean of the university's college of arts and sciences. He was an aggressive executive, replacing 22 department heads, and an outspoken liberal whose views on racial integration and other issues put him at odds with the board, which fired him in 1970.
In 1971, B.U., founded in the 19th century as a Methodist seminary, was going through financial troubles and was overshadowed by the more prestigious schools in the Boston area. As president, Silber increased enrollment by 50 percent while raising admissions standards and tuition and expanded the university's endowment and campus.
Silber also called police to break up student demonstrations, froze salaries and clashed with faculty, staff and alumni. He was also the highest paid university president in the country and received a controversial multimillion-dollar severance package after his retirement.
Silber is survived by six daughters, a son, 26 grandchildren and three great-grandchildren. His wife, Kathryn, who was his debating partner at Trinity, and a son predeceased him.
Swim Week Miami Beach 2014 [PHOTOS]