Spokesman Dave Foster says the headstones were placed along the banks of a small stream in the cemetery and officials feared removing them would damage the stream, The Washington Post reported Wednesday.
The Post alerted the cemetery's new managers to the headstones last week but officials are unable to explain why the old managers used them in the first place.
The new team has vowed to remove the headstones as soon as possible and dispose of them properly.
An Army investigation released earlier this month found Arlington's management system to be "dysfunctional."
More than 200 graves were mislabeled and at least four funeral urns were dumped in a dirt pile.
The cemetery's top two leaders have been reprimanded and replaced.
Celebrity Families of 2014 [PHOTOS]