Sofia Kenin, the 35th-ranked player, emerged with a 6-2, 7-5 win over No. 10 Williams. Katerina Siniakova advanced in the French Open with a 6-4, 6-2 upset victory over Osaka.
"I could see that she's not so confident like she was," Siniakova said of Osaka.
Osaka blamed her poor performance on fatigue. She also admitted to feeling pressured during the tournament.
"I think this tournament I have had a feeling that was different to the other grand slams, or every other grand slam that I have played, because usually I find it very freeing and fun, and this time around I was kind of tense the entire time," Osaka told reporters.
Osaka had won 16 straight Grand Slam matches.
Osaka hoped to become the first female player since 2015 to claim three straight Grand Slams. The last player to do so was Williams, who recorded her earliest loss at a major tournament in five years.
Williams sat out four Slams in 2017-18 while on leave to give birth. She has also battled a knee injury and an illness in recent tournaments.
"It's a lot of emotion," Kenin said in her on-court interview. "Serena is such a great player and a true champion."