The 59-year-old Anderson was a former assistant coach to Nolan Richardson for 17 years, who guided the Razorbacks to a national championship in 1994. Anderson returned to Arkansas in 2011 after coaching at UAB and Missouri.
Anderson led Arkansas to the NCAA tournament in three of the last four seasons entering this year, but the Razorbacks struggled and fell to an 18-16 record this season. Arkansas failed to reach the tournament in four of his first five seasons as the team's head coach.
Anderson had a 169-102 record in his eight seasons at Arkansas.
"After a review of the program, including an analysis of the past eight years and a look forward, I have made a decision that a change in leadership will best position our men's basketball program for future success," Arkansas athletic director Hunter Yurachek said in a statement. "In the past eight seasons, we have won a number of games and have made appearances in both the NIT and NCAA tournaments.
"However, in my evaluation, we have not sustained a consistent level of success against the most competitive teams in the nation to enable us to compete for SEC and NCAA championships on an annual basis. That will continue to be the benchmark for our success throughout our athletic program."
The school said a national search for the team's next head coach will begin immediately. Associate head coach Melvin Watkins "will serve as a lead on men's basketball team related matters until a head coach is named."
One potential candidate for the vacant position is Houston's Kelvin Sampson, according to ESPN. Sampson worked under Yurachek at Houston. The Cougars have made the NCAA tournament in back-to-back seasons, and play Kentucky in the Sweet 16 on Friday.