Rice, 44, just completed his third season at Rutgers, which included a three-game suspension without pay and $50,000 fine after school officials first saw the practice video last November.
"I can't say anything right now except that I'm sorry," Rice said after Rutgers announced his firing.
"I've let so many people down, my players, my administration, the fans, my family, who is sitting in their house just huddled around because of the fact that their father is an embarrassment.
"There's no explanation for what's on those films and there's no excuse for it, I was wrong."
A compilation of scenes from Rutgers practices, the 40-minute tape shows Rice using gay slurs to insult players, throwing basketballs at their heads, and shoving, kicking and grabbing them on the court.
ESPN's "Outside the Lines" obtained the video and showed it Tuesday, creating a media firestorm.
"Based upon recently revealed information and a reviews of previously discovered issues, Rutgers has terminated the contract of Mike Rice," the school said in a statement Wednesday.
The two-paragraph release included a statement from Rutgers Director of Intercollegiate Athletics Tim Pernetti, stating: "I am responsible for the decision to attempt a rehabilitation of Coach Rice. Dismissal and corrective action were debated in December and I thought it was in the best interest of everyone to rehabilitate, but I was wrong. Moving forward, I will work to regain the trust of the Rutgers community."
Rutgers President Robert Barchi issued a statement Wednesday saying Rice's "abusive language and actions are deeply offensive and egregiously violate the university's core values."
Barchi said the video "shows a chronic and pervasive pattern of disturbing behavior," and he concluded Rice cannot continue to serve effectively in a position that demands the highest levels of leadership, responsibility and public accountability."
"This was a regrettable episode for the university, but I completely support the decision to remove Coach Rice," New Jersey Gov. Chris Christie said in a statement Wednesday.
Pernetti, who had given Rice a vote of confidence last month, said Tuesday the three-game suspension and $50,000 fine would be the extent of the school's disciplining of Rice, The (Newark, N.J.) Star-Ledger reported. The suspension -- for what the school called at the time "a violation of athletic department policy" and use of "inappropriate behavior and language" -- cost Rice $74,905, about 25 percent of his pay for the 2012-13 season, the newspaper said.
The school required Rice to attend sensitivity training and anger management counseling, which was still in progress at the end of the season, and Rice asked that the school appoint someone not connected with the basketball program to monitor has behavior.
The tape is believed to have been shot by former director of player development Eric Murdock, whose contract wasn't renewed last year, the Star-Ledger said.
Murdock's attorney, Raj Gadhok, said Tuesday Murdock plans to sue the school for wrongful termination, ESPN reported.
"Mr. Murdock was terminated for having complained of and reported illegal conduct by Mike Rice," Gadhok said.
Rice compiled a 44-51 record at Rutgers, including a 15-16 mark in 2012-13. He previously coached at Robert Morris for three seasons and has a career coaching record of 111-61.
No plans have been announced regarding a successor to Rice.