The committee, meeting in Buenos Aires, first eliminated Madrid from consideration after the Spanish capital lost a first-round tie-breaker with Istanbul, the largest city in Turkey, 49-45, and then gave Tokyo the games over Istanbul by a 60-36 vote.
It will be the second Summer Games for Tokyo, which hosted the event in 1964.
"Tokyo presented a very strong technical bid from the outset -- and it needed to in competition with two such high-caliber bids from Istanbul and Madrid," outgoing IOC President Jacques Rogge said.
Tokyo entered the day as the slight favorite, but all three cities had drawbacks, analyst Mike Lee of Vero Communications told the BBC earlier Saturday.
Madrid is grappling with a sluggish national economy that could have affected its financing of the games. Istanbul has seen a rise in political dissent and doping scandals. Japan has had massive problems with the Fukushima nuclear power plant, damaged in the 2011 earthquake and tsunami.
Lee said those issues and others had put all three cities "onto the defensive for a large part of this campaign."
The IOC was also to select a new president this weekend.