Haley took the Chiefs to a 10-6 record and an AFC West Division title last season, but Kansas City slipped to 5-8 this year after a 37-10 loss Sunday to the New York Jets.
Kansas City went 4-12 in 2009, Haley's first season, and is 19-26 overall with him as coach.
"This was a difficult decision but one that we feel is best for the future of the Chiefs," Chiefs Chairman Clark Hunt said in a statement. "Although there have been bright spots at different points this season, we have not made meaningful progress and we felt that it was necessary to make a change."
Defensive coordinator Romeo Crennel was installed as head coach for the remaining three games this season. Kansas City has home games against Green Bay and Oakland before ending the season at Denver.
Crennel was the head coach for the Cleveland Browns from 2005-08 and posted a record of 24-40.
Haley, 44, was hired Feb. 6, 2009, after the Chiefs had a front office shake-up that led to the firing of Herm Edwards as head coach. It was Haley's first head coaching position. He previously was offensive coordinator with the Arizona Cardinals and also was on the Dallas, Chicago and New York Jets coaching staffs.