The No. 17-seeded Williams wasted no time early in the match, winning the first set in only 18 minutes.
Kanepi responded with a victory in the second set to force a decisive third. Williams, who had 18 aces in the match, took control of the final frame for the 6-0, 4-6, 6-3 win.
"It wasn't an easy match at all," Williams told reporters after the match. "[Kanepi] has had a lot of big wins in her career. I was just happy to get through it to be honest."
One of those big victories came in this year's U.S. Open, with Kanepi upsetting No. 1 Simona Halep in the first round of the tournament.
Williams, wanting to avoid the same fate as Halep, prevailed and will now confront No. 8 Karolina Pliskova in the quarterfinals. Pliskova defeated Australia's Ashleigh Barty 6-4, 6-4 on Sunday to advance to the next round.
The quarterfinal matchup between Williams and Pliskova will be a rematch from the 2016 U.S. Open, which saw Pliskova beat the American in the semifinals.
In other women's action, No. 19 Anastasija Sevastova upset No. 7-seeded Elina Svitolina in three sets to reach the quarters. She will attempt to dethrone No. 3 seed Sloane Stephens, who is the defending champion. Stephens defeated No. 15 Elise Mertens late Sunday to set up the quarterfinals showdown.
On the men's side of the tournament, No. 1 seed Rafael Nadal defeated Nikoloz Basilashvili 6-3, 6-3, 6-7 (6), 6-4 on Sunday at Flushing Meadows. Nadal will challenge No. 9 Dominic Thiem in the quarters, who upended No. 5 Kevin Anderson in the fourth round.
American John Isner outlasted Canada's Milos Raonic in five sets to advance. Isner will return to the U.S. Open quarterfinals for the first time since 2011 after the victory.
Isner will take on No. 3 seed Juan Martin del Potro, who defeated No. 20-seeded Borna Coric in straight sets late Sunday night.