Frazier finished his first season as the defensive coordinator of the Bills, helping the club record its first playoff appearance since 1999.
The 58-year-old has spent many years as an assistant -- including two seasons (2005-06) with the Colts under Tony Dungy as a defensive assistant, defensive backs coach and assistant head coach.
Colts owner Jim Irsay reportedly is a "big fan" of Frazier, per Rapoport.
Frazier also served as a head coach for the Minnesota Vikings (2010-13), posting a 21-32 mark.
In seven seasons at the helm of the Colts, Pagano posted a 53-43 record that included an appearance in the 2014 AFC Championship Game. Indianapolis has failed to make the postseason in three straight seasons.
From 2012-14, Pagano led the team to three consecutive 11-5 seasons and playoff appearances. Pagano coached 10 Pro Bowl players during his tenure with the Colts.