Frazier was fired by the Vikings last week after compiling a 21-32-1 record over three-plus seasons as their head coach, but was quickly tabbed by Smith to serve as his defensive corps chief.
Smith was named Thursday to replace Greg Schiano, who was let go after a 4-12 season.
"I have a tremendous amount of respect for Leslie Frazier and I consider myself very fortunate to have him on my staff," he said. "Leslie is one of the best defensive coaches in the NFL and he has a long track record of helping players reach their full potential."
Smith said Frazier "transformed Minnesota's defense into one of the league's elite units during his time there and I look forward to working with him to do the same here in Tampa Bay."
Frazier led the Vikings' defense prior to his promotion to head coach and also served in that position for Cincinnati from 2003-04.