Cleveland also had the fourth pick in the first round and used that choice on Texas forward Tristan Thompson.
The Cavaliers became the first team since the NBA draft lottery era began in 1985 to have two of the first four selections in the same year.
Cleveland won just 19 games during the 2010-11 season after James left in free agency to play for the Miami Heat. The Cavaliers had surpassed the 60-win mark in each of the previous two years.
Irving became the third Duke player to be selected No. 1 in the NBA draft, joining Elton Brand (Chicago, 1999) and Art Heyman (New York, 1963).
"I didn't have any doubts about going No. 1," Irving said. "I'm going to remember this for the rest of my life."
Despite having had a brief college career, Irving became the consensus choice as the best player in what was considered to be a weak draft class. A toe injury caused him to miss 26 games in his only year at Duke. He returned for the NCAA Tournament, during which Duke lost in the regional semifinals to Arizona.
The 6-8 Thompson averaged 13.1 points and 7.8 rebounds during his freshman year at Texas.
"One of my strengths and attributes is getting up and down the floor and playing with Kyrie is going to be exciting," Thompson said.
Minnesota used the second pick in the draft to select forward Derrick Williams of Arizona. Center Enes Kanter of Turkey went to Utah with the third pick. Kanter had enrolled at Kentucky, but was declared ineligible by the NCAA because he accepted payment from a team in Europe.
Lithuanian center Jonas Valanciunas went to Toronto as the No. 5 pick. Rounding out the Top 10 were forward Jan Vesely from the Czech Republic (to Washington), Bismack Biyombo of the Democratic Republic of the Congo (to Sacramento), Kentucky point guard Brandon Knight (to Detroit), Connecticut guard Kemba Walker (to Charlotte) and Brigham Young guard Jimmer Fredette (to Milwaukee).
Biyombo was reportedly chosen by the Kings in order to trade him to Charlotte. Fredette was picked by the Bucks after leading all of college basketball in scoring last season.
The first round ended with the champion Dallas Mavericks selecting Texas forward Jordan Hamilton, who was expected to wind up in Denver as part of a three-team deal that would send Portland's Rudy Fernandez to Dallas.
Of the 26 players taken in the first round, six played basketball last season outside the United States.