The fire -- dubbed the Chariot fire -- started Saturday near Chariot Canyon, southeast of Julian, the Los Angeles Times reported.
The cause of the fire was not determined.
Cal-Fire said all evacuation orders related to the wildfire were lifted Thursday at the El Prado, Laguna and Horse Heaven campgrounds.
Most of the structural damage caused by the fire was at the Al Bahr Mount Laguna Shrine campground, the Times said.
More than 1,600 fire personnel battled the fire along with helicopters and air tankers, mostly near the Cleveland National Forest, the newspaper reported.