Lance Berkman drew a one-out walk from Jamey Wright (4-3), and Kelly was inserted as a pinch-runner. After Skip Schumaker struck out, Mike Carpenter sent a single into right field, but Kelly had to stop at second after tripping as he rounded the bag.
Furcal followed by lining at hit to left and Kelly scored easily when Dodgers outfielder Tony Gwynn Jr. overran the ball, handing the Cardinals a 2-1 lead in their four-game set against Los Angeles.
Fernando Salas (1-3) picked up the win after tossing two shutout innings for the Cards.
Los Angeles starter Aaron Harang held St. Louis to two runs on two hits over 7 1/3 innings.
Hanley Ramirez, obtained from the Miami Marlins in a major trade earlier Wednesday, went 2-for-4 with an RBI single in his Dodgers debut.
The second-place Dodgers remained 2 1/2 games behind National League West-leading San Francisco. The Giants fell to San Diego Wednesday.