The salvage effort was called a "one-step-at-a-time" operation.
Workers said they were slowly tearing down the blue, one-story house because of the instability of the ground, CNN reported Monday.
The giant sinkhole engulfed a bedroom with Jeff Bush inside Thursday.
Officials said the body of the 36-year-old Bush likely won't be recovered from the sinkhole that is more than 20 feet wide and more than 50 feet deep.
Crews started demolishing the house Sunday and expected to finish Monday. Once the debris is cleared, engineers will examine the sinkhole to determine the best way to fill it.
Authorities stopped searching for Bush after odds against survival became clear, CNN said.
"We just have not been able to locate Mr. Bush, and so for that reason the rescue effort is being discontinued," Hillsborough County Administrator Mike Merrill said Saturday. "At this point, it's really not possible to recover the body."
Once officials get a better view of the sinkhole, they can get a better idea of what to do next, Merrill said Sunday.
"This is one step at a time because we really don't know what we're dealing with here," he said.