The men, who are between the ages of 17 and 31, arrived at the home on Thursday night apparently looking for ghosts.
"They had been looking for ghosts, trying to summon spirits, beating on the floors," said Col. John Doran.
Built in the 1850s, the LeBeau Plantation House was once one of the largest plantations south of New Orleans. It has also been used as a hotel and illegal casino, among other things.
"We all heard the ghost stories while growing up," said St. Bernard Parish Sheriff Jimmy Pohlmann. "In combination with smoking dope in there, it appears it was intriguing to them."
Police suspect that Dusten Davenport, 31, became frustrated when no ghosts materialized, and after drinking and smoking marijuana, decided to burn the place down and began stacking wood to burn. He then convinced the other men to help him.
Davenport and four others were arrested on Friday on charges of arson, simple burglary and criminal damage worth more than $50,000. The sixth man was arrested on charges of accessory to arson and criminal trespassing and the seventh was arrested on a charge of accessory to arson.
Michelle Mahl Buuck wrote "The Historic LeBeau Mansion: A Forgotten Monument" about the purportedly haunted mansion. "There were lots of kids who would just go, perhaps because of the stories of it being haunted. They would take many nighttime excursions, going there just hoping to see a ghost," Buuck said.