County officials say they are taking steps to ensure none of the debris ends up on so-called "murderabilia" websites, The Cleveland Plain Dealer reported.
Michelle Knight, one of the former captives, arrived at the home carrying a bundle of yellow balloons she said represented other people still missing, WKYC-TV, Cleveland, reported.
She told their families not to give up hope.
Cheers erupted from a crowd as heavy machinery began knocking down the wood frame home.
Castro, 53, was sentenced last week to life in prison plus 1,000 years after pleading guilty to more than 900 charges of rape, kidnapping and aggravated murder.
Cuyahoga Land Bank acquired the property last week.
USA Today reported officials decided to demolish the house because "we didn't want some kind of gruesome, macabre shrine," said Joseph Frolik of the Cuyahoga County Prosecutor's Office.
Prosecutors said Castro cried when he signed over the deed, lamenting the "many happy memories" he had with the women while in the house.