The agents and uniformed officers were placed on administrative lead Saturday after they reportedly consorted with prostitutes while performing advance work for the president's trip to the Summit of the Americans in Cartagena.
"I think it's been much more of a distraction for the press," White House spokesman Jay Carney told reporters. "Our focus and the president's focus continues to be on the meetings he's having."
The Wall Street Journal said five members of the U.S. military's Southern Command were also confined to quarters in the incident.
The Secret Service said publicly the flap had not compromised the tight security the president receives when travelling, "We regret any distraction from the Summit of the Americas this situation has caused," Assistant Director Paul Morrissey said in a written statement.
Obama may have actually welcomed some distractions once the summit began. Politico said the United States received some chiding for its international trade policies and counter-narcotics programs in Latin America.
Administration officials downplayed any apparent friction. "It's the Americas. Nobody is shy about making their voice heard," deputy national security adviser Ben Rhodes told reporters. "Anybody who watched the discussions … saw that there was a much broader basis for agreement among the leaders than there were issues of disagreement."