The mystery began when David Robertson and his wife were walking on the east side of the city's landmark Mount Royal and found the old headstone, The Gazette newspaper said.
The stone commemorates Patrick McDonald who died in 1891, and his wife, Ann Murphy, who died in 1913.
Robertson notified the Roman Catholic Notre Dame des Neiges cemetery located on the west side of the mountain, which led them to discover the stone had been replaced with a new, much larger monument, the newspaper said.
Cemetery spokeswoman Christine Latour said rules had been breached.
"Someone ordered a new monument and did not ask us for a permit to take the old one down. You are only allowed to put up one monument on each plot," she said.
Meanwhile, Robertson said he'd like to know how the old stone ended up where he found it.
"No one could carry it. It would have to be transported and dumped," he told the newspaper. "You'd think the Catholics would take care of their own better."