The rookie signal-caller connected on 12-of-25 passes for 188 yards and a touchdown and didn't throw an interception for the Texans (9-3), who strengthened their hold on first place in the AFC South with the win.
Yates was starting behind center after both Matt Shaub and backup Matt Leinart went down with injuries.
Foster scored on a 1-yard run to cap a 19-play, 85-yard drive 6 minutes into the fourth quarter, breaking a 10-10 tie putting Houston ahead for good. He carried the ball 31 times for 111 yards.
Houston's defense intercepted Atlanta's Matt Ryan twice, limiting him to 20-of-45 passing for 267 yards.
Michael Turner also couldn't get on track for Atlanta (7-5), managing only 44 yards rushing.