Frazier was relieved of his duties one day after Minnesota finished the 2013 season with a 14-13 win over Detroit that left the Vikings with a 5-10-1 record.
Minnesota was 21-32-1 with one playoff appearance under Frazier, who got the job after Brad Childress was fired 10 games into the 2010 season.
"We have tremendous respect and appreciation for Leslie and what he has done for the Minnesota Vikings," Vikings owner Zygi Wilf said. "He stepped in and established a strong positive culture here and he has been the consummate professional as our head coach and in this community.
"Making this change is difficult but what we determined is best for the organization."
Minnesota General Manager Rick Spielman said, "We believe a coaching change is needed to help build a successful team moving forward."
Frazier had been on the Vikings' coaching staff for four seasons, serving as defensive coordinator and assistant head coach, before he was named interim head coach in 2010. He was given the head coaching position Jan. 3, 2011.
The Vikings were 3-3 in 2010 when Frazier was the interim coach and 3-13 in his first full year as head coach. Minnesota made the playoffs with a 10-6 mark in 2012 but slumped in 2013.