Boasting an offense that ranked second in the league with 32 points per game, the Saints were held without a first down on their first six possessions of the second half.
But with his team trailing by four points, Brooks made all the big throws on a 10-play, game-winning march that took three minutes.
Brooks went five of seven for 81 yards on the drive, including a 38-yard completion to Stallworth to the Carolina 10. They hooked up on the winning score when Stallworth made a leaping catch in the left corner of the end zone.
"During the course of the game, you don't think about what they have done," said Brooks, who finished 20 of 37 for 259 yards.
"We just wanted to get back on top and win. We were really shut out in the second half. We had several options and Donte' made a great adjustment to make the catch."
"I was probably one of the last reads," Stallworth said. "That is why they drafted me, to make plays annd help this team win."
Rookie defensive end Charles Grant sealed the win by returning quarterback Rodney Peete's fumble 34 yards for a touchdown on the game's final play.
Deuce McAllister rushed for two touchdowns for the Saints (7-2), who tied the idle Tampa Bay Buccaneers for first place in the NFC South. New Orleans is off to its best start since 1992.
"I've got to give my team credit, it wasn't a pretty win," Saints Coach Jim Haslett said. "We didn't play smart in some areas. Any time you play on the road in a place like this with a tough defense, it is a tough game."
Returning after missing two games with a knee injury, Peete passed for 310 yards and two touchdowns, but could not prevent the Panthers (3-6) from suffering their sixth straight loss. Five of the losses have come in the final four minutes.
"All I can say it that it is very unfortunate we have had it happen more than enough this year," Panthers first-year coach John Fox said. "We are not making the plays down the stretch, particularly on defense when we need to make a stop. You got to be tough minded. You've got to finish. You've got to overcome things."
It looked as if the Panthers would not suffer another tough loss when Reggie Howard intercepted Brooks deep in his own territory on the Saints' final drive. But the play was nullified on an interference penalty on Howard.
The Saints also benefited from a holding penalty on cornerback Terry Cousin, nullifying a sack by rookie Julius Peppers.
Peppers had a sack earlier in the contest, giving him 10 on the season and moving him within four of Jevon Kearse's rookie record.
"The game is never over with us, things just didn't go our way again," Peppers said. "I was thinking we were going to win. It wasn't taken from us, we gave it away. It is hard and frustrating at the end of the game. We have to get tougher. We have to finish."
Helped by the defensive miscues, Brooks converted third and six with a 19-yard connection to Joe Horn. He also found tight end Boo Williams for 12 yards, moving the ball into Carolina territory before finding Stallworth down the sideline for 38 yards. Two plays later, he connected with the rookie for the winning score.
Carolina got the ball back after Stallworth's score, but Peete was hit by Darren Howard and Grant scooped up the ball.
Peete had a solid game in his return to the lineup, completing 23 of 40 passes while providing a spark to the Panthers, who had failed to register an offensive touchdown in their last two games.
Peete got the Panthers their first offensive touchdown in 13 quarters, capping an eight-play, 80-yard drive with a three-yard pass to tight end Jermaine Wiggins with 34 seconds left in the first half, pulling Carolina within 13-7.
Lamar Smith, who was held to minus two yards rushing on five attempts in in the first half, broke off a 59-yard run on the Panthers' first possession of the third quarter.
The Panthers had a first and goal at the two, but were stopped on three straight runs and Shayne Graham kicked a 19-yard field goal.
Peete moved the Panthers 61 yards in six plays on their later drive in the quarter, including a 35-yard connection to Kris Mangum to the one. This time, Smith powered over to give the Panthers a 17-13 lead.
But with their offense playing well and their defense dominating, the Panthers committed a blunder on special teams when Todd Sauerbrun had his punt blocked deep in his own territory by Steve Gleason.
Two plays after the block, McAllister scored on a four-yard run to give the Saints a 20-17 lead with 1:35 left in the third quarter.
McAllister, who entered third in the NFL in rushing, gained 71 yards on 12 attempts. He has 907 yards and nine rushing touchdowns in nine games.
It was defensive struggle in the fourth quarter until Peete engineered a seven-play, 79-yard drive. He kept the drive alive with a quarterback sneak on fourth and one from his own 42 and found Muhsin Muhammed for 42 yards to the New Orleans 15.
Peete capped the drive on the next play, finding Wesley Walls for a touchdown that gave the Panthers a 24-20 lead with 3:31 remaining. But the defense could not hold the lead.