Delhomme completed 10-of-19 passes for 109 yards before leaving the game late in the third quarter with an elbow injury.
The victory was the fourth for Carolina (2-1) in its last five games against the Falcons (0-3).
"Any time you have a division game on the road it is a big game," said Panthers Coach John Fox. "It was a hard-fought game and we're glad we got the job done."
Joey Harrington was 31-for-44 for 361 yards passing and two touchdowns for Atlanta, but the Falcons were unable to mount enough defense in the second half.
Harrington hit Alge Crumpler for a 13-yard touchdown that put Atlanta ahead, 17-10, with 9:09 left in the third quarter.
Carolina responded with Delhomme’s 5-yard TD pass to Jeff King and Foster’s 10-yard TD run up the middle with 47 seconds to play in the third.
The two teams traded field goals in the fourth to close out the scoring.