Cleveland dropped a 105-98 decision to Charlotte in Wednesday's season finale, leaving the Cavaliers with a 24-58 record, which was the third worst in the NBA this season.
Cleveland dropped its last six games.
"I have tremendous respect for Byron professionally and a great deal of admiration for him personally," Cavaliers General Manager Chris Grant said in a statement. "At the same time, it is critical for where we are as a team to ensure that we capitalize on every opportunity for development and success and we have fallen short of that on the court."
The Cavaliers were 61-166 in three seasons with Scott as coach. He took over the team just after LeBron James left for Miami. The first season Cleveland was 19-63, including an NBA-record 26-game losing streak.
Scott, who played 14 seasons in the NBA and won three league titles with the Los Angeles Lakers, also coached at New Jersey and New Orleans. He is 416-521 overall as a coach and led the Nets to the NBA finals in 2002 and 2003.