Although the couple took their romance public by walking the red carpet Monday night at a Speedo promotional event in London, UsMagazine.com said Phelps declined to answer questions about their relationship.
Phelps, 27, has earned 22 medals, 18 of them gold, during the past three Olympics.
Rossee, 25, has appeared in runway shows for fashion designers Michael Kors and Betsey Johnson.
Jordana Brewster on Paul Walker: 'He was an enormous presence in my life'
Gal Gadot cast as Wonder Woman for 'Batman vs. Superman'