Kubiak, 52, took the Texans to the playoffs in 2011 and 2012 before suffering a mini-stroke in November and was fired the next month as the club was mired in the longest losing streak in franchise history.
Houston finished 2-14.
"I want you to know for the Ravens, for our coaching staff, for our players, for our organization, for our fans -- it's a great day and I'm excited about it," Harbaugh said.
Kubiak will take over an offense that ranked 29th in the NFL last season as the defending Super Bowl champions failed to make the playoffs for the first time since 2007 with an 8-8 record.
He replaces Jim Caldwell, who left to become the head coach of the Detroit Lions.