The two raced last weekend at an after-hours pool party at the Wynn Las Vegas resort and casino -- a race the world record-holder in the 200-meter and 400-meter individual medleys easily won.
"We were at the same pool party and his people came over to my table and said Prince Harry wants to meet you," Lochte told NBC's "Today" show.
"I took off my shirt and jumped in, and we started racing," the gold medal winner said, adding swimming against the partying prince was "the coolest thing."
But Lochte, 28, told the program he was "kinda happy" he missed out on the prince's late-night naked party.
"After our race, we went our separate ways," he said.
The gossip Web site TMZ said Thursday the British royal family was "furious" over the photos the site published Tuesday night -- photos that quickly spread globally. The New York Daily News said the 27-year-old Harry, who returned to England Thursday, faced the "wrath" of Queen Elizabeth II.
The royal family moved to block British publication of the naked photos Wednesday, and no British newspaper published them.
The family wrote to the Press Complaints Commission, Britain's voluntary newspaper and magazine regulatory body, drawing attention to the privacy clause in the commission's Code of Practice, which the family said should ensure "reasonable expectations of privacy in a hotel room," a spokesman said.
The family also reminded individual newspapers of their own editorial codes of conduct and to respect the prince's privacy, the spokesman said.
Buckingham Palace and Prince Charles, Harry's father, did not threaten legal action if the photos were published, the spokesman told the British newspaper The Guardian.
An unsigned Guardian blog Thursday suggested Harry, who stripped on the Las Vegas Strip, should be stripped of his royal status -- or at least should abdicate.
"I bet Harry would jump at the chance," the blog said. "He could be like King Edward VIII and abdicate -- not from the throne in his case, but from being third in line to it.
"We'd no longer call him prince, and in return we'd give him back some of his privacy.
"He could get as drunk as he likes and strip of all over the place," the blog said.
King Edward VIII abdicated the British throne in 1936 to marry U.S. divorcee Wallis Simpson.
He famously said, "I have found it impossible to carry the heavy burden of responsibility and to discharge my duties as king as I would wish to do without the help and support of the woman I love."
TMZ reported Thursday when Harry began the party at his VIP suite at the five-star Encore Las Vegas luxury resort, casino and hotel connected to the Wynn, the young women he brought with him were not asked to surrender their cellphones.
And when the party fired up and several women started snapping photos, one of Harry's security guards feebly said, "Aww, come on, no photos."
The security guys were enjoying the party more than protecting the prince, sources told the Web site.
"Harry's team acted like a bunch of amateurs," a source said.