Castro, 52, was charged Wednesday afternoon with four counts of felony kidnapping and three counts of felony rape. His brothers, Pedro and Onil Castro, were initially held on similar suspicions but will not be charged in connection to the case.
Although Castro's alleged crimes came as a surprise to friends and neighbors, he has been accused of violence on multiple occasions, USA Today reports.
He was arrested on domestic violence charges in 1993, but the grand jury opted not to indict him. And in 2005, Grimilda Figueroa, with whom Castro has three children, complained he had beat her and threatened her.
Figueroa claimed Castro broke her ribs, her nose twice, knocked out a tooth, twice dislocated her shoulder and caused a blood clot in her brain. She said he threatened to kill her and her two daughters, but charges were dropped when Figueroa's attorney failed to show up at the hearing.
Castro, who worked as a bus driver for the Cleveland Metropolitan School District, also had trouble with work. According to his personnel file, he was suspended for 60 days in 2004 after leaving a child on his bus and again in 2009 for making an illegal U-turn while children were on board.
He was twice reported to supervisors for cursing at kids, and suspended for two months for using the bus to go grocery shopping.
Castro was fired in October 2012 after leaving the bus unlocked and unattended for several hours.
Early police reports indicate chains and ropes were recovered from Castro's home where he held captive Amanda Berry, 27, Gina DeJesus, 24, and Michelle Knight, 32, along with Berry's 6-year-old daughter.
Castro's neighbors on Seymour Street have claimed they contacted police about a woman crawling naked in the backyard.
At a press conference Wednesday, Cleveland Mayor Frank Jackson said officials had "no indication that any of the neighbors, bystanders, witnesses or anyone else has ever called regarding any information, regarding activity that occurred at that house on Seymour Avenue."
Police visited the house once -- in 2004 -- after the incident with the child left on the schoolbus but left when no one answered at the door.