Orlando lost its 12th straight game and has won only twice since Christmas.
Irving led the way with 24 points, eight assists and six rebounds. Marreese Speights came off the bench to score 18 points for the Cavaliers, hitting all 12 of his attempts from the foul line.
It was Cleveland's third straight win.
Orlando closed within 104-101 when Nokola Vucevic made a running hook shot with 4:42 to play. Cleveland then ran off its seven straight points to take control.
Vucevic scored 25 points and grabbed 13 rebounds. Arron Afflalo added 23 for the Magic.
Swim Week Miami Beach 2014 [PHOTOS]