The agents, who had been sent to Colombia before President Barack Obama's arrival to provide security for the president's trip there for a summit, have been returned to the United States after the incident in Cartagena, the U.S. broadcaster said.
"One of the agents did not pay one of the prostitutes, and she complained to the police," said Ronald Kessler, a former Washington Post reporter and author of "In the President's Secret Service: Behind the Scenes With Agents in the Line of Fire and the Presidents They Protect."
The Washington Post, which first reported the story, said Kessler noted prostitution is legal in designated "tolerance zones" in Colombia but is considered inappropriate by the Secret Service. Kessler said several of the agents are married.
Edwin Donovan, an agency spokesman, said an unspecified number of agents had been recalled and replaced with other agents but would not provide details about the alleged misconduct.
Donovan said security for Obama had not been compromised by the incident.
Donovan, in a statement, said the matter has been turned over to the Office of Professional Responsibility, which serves as the agency's internal affairs unit.
"The Secret Service takes all allegations of misconduct seriously," Donovan said. "These personnel changes will not affect the comprehensive security plan that has been prepared in advance of the president's trip."
Obama is in Colombia for this weekend's Summit of the Americas, which focuses on economic policy and trade.