Carpenter, who had four hits in the game, walked with one out in the 12th and advanced to third when Jon Jay singled. Holliday worked the count full against Bryan Morris (5-6) and then lined a hit into center that allowed Carpenter to trot home with the game-winner.
Kevin Siegrist (1-1) struck out two in one inning of relief to earn his first major-league win.
Pittsburgh scored four times in the top of the fifth inning and the Cardinals answered with five runs in the bottom of the inning. A home run by the Pirates' Russell Martin in the eighth tied the score 5-5.
Clint Barmes had a two-run home run for the big hit in the Pittsburgh fifth.
Daniel Descalso got the Cardinals on the board with a two-run single and then scored on Carpenter's triple. Jay followed with a score-tying single and Holliday doubled in Jay to put St. Louis ahead.
Carpenter, Jay and Holliday -- the top third of the St. Louis batting order -- combined for three runs, four RBI and eight of the Cardinals' 12 hits.