Patrick Peterson's interception of Tampa Bay rookie Mike Glennon late in the game led to a score-tying TD strike from Carson Palmer (21-of-38, 248 yards, one touchdown, two interceptions) to Larry Fitzgerald.
After Arizona stopped the Buccaneers at their own 1-yard line on the next possession, the Cards took over with good field position and Feely cashed it in with his decisive boot.
Peterson then sealed the win with another pick-off as the Cardinals (2-2) recorded their first road triumph in more than a year.
The Buccaneers (0-4), meanwhile, remained winless after Glennon took over from the benched Josh Freeman behind center.
He finished with 193 yards on 24-for-43 passing with a touchdown, but was intercepted twice and lost a fumble in his first career start.