The meeting came just hours after CBS announced it canceled Imus' radio show, "Imus in the Morning," effective immediately, CNN reported.
Imus touched off a firestorm of criticism, and apologized repeatedly, when he referred on the air to the Rutgers women as "nappy-headed hos."
New Jersey Gov. Jon Corzine offered the mansion as a "neutral" site for Imus to meet with Rutgers players, coaches and family members. CNN quoted sources inside the mansion said the meeting was "a long way from being over."
It was not known whether the women would accept Imus' apology.
Corzine was injured in an automobile collision near Atlantic City as he was returning to the mansion, and was not present for the meeting.