Sixth-seeded Venus Williams dropped a fourth-round match against 10th-seeded Karolina Pliskova in which both players let match-point opportunities slip away.
Serena Williams polished off Russia's Yaroslava Shvedova 6-2, 6-3 for her 308th match victory in Grand Slam singles events. That established an Open-era record for matches won in majors, one more than Roger Federer's tally and two more than Martina Navratilova's total.
The younger Williams sister has been dominant through four rounds. She had not lost more than three games in any of the eight sets she has played in the event and has yet to lose a service game in the tournament.
This is the ninth consecutive time Serena Williams has reached the quarterfinals at the U.S. Open. Her quarterfinal opponent will be fifth-seeded Simona Halep, who advanced to the quarterfinals for the second straight year with a 6-2, 7-5 victory over 11th-seeded Carla Suarez Navarro.
"I had to run a lot and I tried to be more aggressive today," Halep said. "This was very special today to win. I want to enjoy today."
Venus Williams, 36, nearly made an amazing comeback in her 4-6, 6-4, 7-6 (3) loss to Pliskova, with the crowd clearly behind her.
"I was prepared that I'm going to play Venus -- it's going to be tough, because all the people are cheering for her," said the Czech player.
Pliskova led 4-2 in the third set, but Venus Williams rallied to win three straight games and held a match point with Pliskova serving at 4-5, 30-40.
Pliskova saved the match point with a swinging forehand volley winner.
"I really played the perfect point there," said Williams, "and she managed to stay alive."
Pliskova then had a triple match point of her own while serving at 6-5, 40-0. She double-faulted the first match-point opportunity away, and Williams parlayed that into five consecutive winning points to push the match to a tiebreaker.
"To be honest," Pliskova said, "it was really difficult ... when I lost my serve."
Pliskova dominated the tiebreaker and throught she had won the match with a winner at 6-2, but Williams challenged the call and the review showed that Pliskova's shot was out. Pliskova finished off Williams on the next point, however, to complete the see-saw match.
"In the 'breaker, I went for a little bit more," Williams said, "but I didn't put the ball in enough."
Pliskova next will play Croatia's Ana Konjuh, who upset fourth-seeded Agnieszka Radwanska of Poland 6-4, 6-4 late Monday night.
Konjuh, 18, never previously advanced beyond the third round of a major. She is three years removed from winning the U.S. Open junior singles title.
Rawanska, a 27-year-old who was a semifinalist at the Australian Open this year, couldn't solve her younger opponent.
"I was really trying. I was just too slow today," Radwanska said.