Smith left St. Louis last week to take over the helm of the Chicago Bears.
Martz interviewed former Bears Coach Dick Jauron and former Green Bay Packers defensive coordinator Ed Donatell for the position before choosing Marmie, the St. Louis Post-Dispatch reported.
Marmie, 61, served as the Arizona Cardinals' defensive coordinator under Dave McGinnis since 2000 but was fired along with the entire staff at the end of the 2003 season.
"I worked with Larry for seven years," Martz told the newspaper. "He probably has had more of an impact on me professionally than anybody. I'm thrilled and excited that we have an opportunity to bring Larry to St. Louis."
Martz and Marmie first worked together in the mid-1980s on John Cooper's staff at Arizona State.