Pelini, 39, returns to Nebraska from LSU, where he was defensive coordinator for three seasons. He had the same position at Nebraska in 2003.
Callahan was let go Nov. 24 after a 5-7 season.
"We need a head coach with strong defensive credentials and great leadership," said Tom Osborne, Nebraska's interim athletic director and former coach.
"We were also looking for someone who can inspire confidence and get players to play with great effort. And, of course, we also wanted our new head coach to understand our traditions, including the importance of our walk-on program and the importance of football in this state."
Pelini's coaching experience includes stints on the defensive staffs of the San Francisco 49ers, New England Patriots and Green Bay Packers in the NFL.