Caldwell was let go by the Colts two weeks ago after a 2-14 season and posting a mark of 26-22 with one Super Bowl appearance in three years.
In Baltimore, he fills a vacancy that had been open since Jim Zorn left following the 2010 season.
Baltimore quarterback Joe Flacco threw for 3,610 yards with 20 touchdowns and 12 interceptions this past season as the Ravens won the AFC North. Offensive coordinator Cam Cameron handled the quarterback coaching duties.
Caldwell's resume includes a stint as head coach at Wake Forest.