The Ravens trailed 23-7 heading into the fourth period, but Taylor put together scoring drives of 40 and 68 yards to bring Baltimore within three points.
After the Ravens forced an Atlanta punt, Asa Jackson brought the kick back 78 yards for the winning touchdown with 6:18 to play.
Taylor, who has appeared in 10 regular-season games and thrown 30 passes during two seasons with the Ravens, completed 5-of-9 passes for 83 yards.
He hit Aaron Mellette with a 40-yard scoring throw and then found Tandon Moss with a 5-yarder.
Joe Flacco saw first-half action for the defending Super Bowl champion Ravens and went 7-of-9 for 118 yards, a touchdown and an interception.
Atlanta's Matt Ryan was 8-of-15 for 97 yards and a first-period touchdown.
Baltimore has won both its exhibition games and Atlanta is 0-2.