The dramatic victory came after Oklahoma State's Marcus Smart had missed chances at the buzzer in both regulation and the first extra period.
Kansas (22-4, 10-3 Big 12 Conference) is tied for the lead in the league race with rival Kansas State.
The Jayhawks took a 66-65 lead with 1:01 remaining in the second overtime on two free throws from Jeff Withey, but the Cowboys went back in front on a jumper by Markel Brown with 52 seconds left.
After Tharpe's jump shot lifted Kansas into the lead, Brown missed a fadeaway jump shot that could have given the Cowboys the win.
Travis Releford led Kansas with 18 points and Withey had 17 points with 14 rebounds.
Oklahoma State (19-6, 9-4) got 20 points from Brown and 16 from Smart.