The couple and a young child were on a free trip to the Musee d'Orsay with a charity that supports struggling families, the British newspaper The Daily Telegraph reported Monday.
The family was browsing a room with paintings by Van Gogh when guards told them they had to leave.
"I argued with the security man, telling him the family were all decent and properly dressed," said a worker with the charity Act for Dignity.
The director of the museum told French news sources he was saddened by the incident.
Claire Hedon, vice president of the charity, said she wrote to Aurelie Filippetti, the French culture minister, to complain about the incident.