With both Tomas Vokun and Michal Neuvirth nursing injuries, Holtby was given the starting goaltending job despite having appeared in only 21 NHL games.
Holtby was up to the task through three periods, stopping 29 shots and battling last year's Stanley Cup Finals hero Tim Thomas to a draw.
Thomas fought off a threatening shot from the Capitals' Marcus Johansson 70 seconds into overtime and Boston quickly counterattacked with Benoit Pouliot firing a cross-ice pass to Kelly.
Kelly, speeding up the left wing, let his shot go from the outer rim of the faceoff circle and beat Holtby cleanly for the winning goal.
Thomas, who posted a 16-9 record last year during Boston's march to the Stanley Cup title, had to make only 17 saves Thursday.
The Bruins will host the second game of the series Saturday.
Senate Democrats to pull all-nighter on climate change
Dennis Rodman pledges to end trips to North Korea