The last visit for either woman was Sunday, Nov. 4, when Jill Kelley took a tour of the White House with her husband, children and twin sister Natalie Khawam, NBC News reported.
Petraeus resigned four days later, after the FBI revealed he was having an extramarital affair with his biographer, Paula Broadwell.
Broadwell visited the White House in June 2009 and June 2011, a White House official said.
On the first occasion, she met with a member of the National Security Staff about policy in Afghanistan and Pakistan. The second time she was one of about 20 people attending a broad briefing on Afghanistan and Pakistan.
Kelley, an unpaid social coordinator at MacDill Air Force Base in Tampa, Fla., made trips to the White House with her sister for breakfast in the cafeteria Sept. 28 and for lunch Oct. 24.
Kelley's visits were arranged by a member of the White House staff she met at MacDill.