Casey leaves the role just two days after he was named the NBA Coach of the Year by the league's 30 coaches.
"After careful consideration, I have decided this is a very difficult but necessary step the franchise must take," Raptors president Masai Ujiri said in a statement, according to ESPN and USA Today.
"As a team, we are constantly trying to grow and improve in order to get to the next level. We celebrate everything Dwane has done for the organization, we thank him, and we wish him nothing but the best in the future. He was instrumental in creating the identity and culture of who we are as a team, and we are so proud of that."
The Raptors won a franchise-record 59 games this season and finished as the top seed in the Eastern Conference. Toronto beat the Washington Wizards in the first round of the playoffs, before being swept by LeBron James' Cleveland Cavaliers.
Casey had a year remaining on his contract with the squad.
"Everybody expected more," Casey said after the season-ending series loss. "We expected more. I expected more. And we talked about all the things they want do this summer to get better. To be motivated to come back next year and get over the hurdle. And to continue to improve."
The firing came as a shock around the league for some of the NBA's highest-profile stars.
🤔🤷🏾♂️ https://t.co/Tr757oXpqx— DWade (@DwyaneWade) May 11, 2018
WOW WOW WOW oh WOOOOOWWWWWW. Well I hope the @Raptors are bringing in the ghost of Red Auerbach. Or maybe Pat Riley, Phil Jackson.. Masai Ujiri is as good as they come, he must be playing chess with us right now..— Reggie Miller (@ReggieMillerTNT) May 11, 2018
The Raptors have never advanced past the Eastern Conference finals and have only reached that point once in franchise history. Casey, 61, was hired in 2011. He owned a 320-238 record in seven seasons with the Eastern Conference squad. He posted a 21-30 mark in the postseason.