The incident, which occurred Saturday in Stockholm, stemmed from a rumor that grew following accusations the facility's residents had sexually molested girls at a public pool, the Swedish news agency TT reported Tuesday.
A week earlier, two staff members and 18 boys from the facility visited the pool. While there, five to seven of the boys allegedly molested three girls, ages 11 and 12, TT reported.
"They were touched on the back, legs, and behind. No clothes were removed," Diana Sundin, a police spokeswoman, said.
The boys were reported to police as sexual molestation suspects, TT reported.
But in the subsequent days, accusations against the boys morphed on Internet discussion web sites, TT said. Some sites called the incident a "gang rape of an 11-year-old girl, carried out by 20 men."
"The whole thing has been blown way out of proportion. No one was raped. No one has said anything like that during interviews or in the report," said police officer Mats Eriksson.
Police said about 30 people, mostly men, from the white-power group Nordic Youth arrived at the facility carrying signs, throwing eggs and screamed racial epithets. They left when authorities arrived.
"Outside the refugee facility they screamed 'blackheads' and 'go home immigrants' and other things," said Sundin.
Police said there are rumors the group plans to return, so surveillance around the facility has been stepped up.