It was a major success for the Americans, who haven't won an Olympic medal in men's soccer in 104 years.
The Japanese missed an open net twice and supplied a majority of the pressure but it was Holden who managed the lone goal. He took a cross from Marvell Wynne and rifled a low shot. Goalkeeper Shusaku Nishikawa got a hand on the ball but it trickled past him and into the net.
The United States, playing in Group B in Tianjin, will take on the Netherlands on Sunday in the second of three round-robin games. The Americans' hopes for a berth in the quarterfinals was helped along Thursday when the Netherlands had to settle for a 0-0 tie with Nigeria.
The top two finishers in each of the four groups make it to the quarterfinals.
In other action Thursday, Argentina defeated Ivory Coast 2-1 and Australia tied Serbia 1-1 in Group A; Brazil downed Belgium 1-0 and China tied New Zealand 1-1 in Group C; and Italy ran away from Honduras 3-0 while South Korea and Cameroon tied 1-1 in Group D.