Prosecutor Victor Perez told a news conference he had signed criminal complaints against Ariel Castro for kidnapping and rape. He will appear in court Thursday, Perez said.
No charges will be filed against Castro's brothers, Pedro and Onil. Perez said there is no evidence they were involved in the alleged crimes.
Amanda Berry, 27, Gina DeJesus, 24, and Michelle Knight, 32, were kidnapped and held captive in a house owned by Castro. Berry managed to escape Monday with the help of a neighbor and the other two were freed by police.
Police said the kidnapping counts against Castro pertain to Berry, her 6-year-old daughter, DeJesus and Knight. The rape counts apply to the three adult women.
Police Chief Ed Tomba said the women were held captive separately and Monday was the first time in a decade they had a chance to escape.
Tomba said Castro is cooperating with authorities.
"We don't anticipate any other victims where he's a suspect," Tomba said.
Tomba said most of the evidence against Castro came from the victims.
Tomba declined to comment on reports the three women suffered numerous pregnancies and forced miscarriages.
"That will be something that will have to come out in court," he said.
He said more than 200 items were taken from Castro's home but declined to comment on whether a suicide note written by Castro was among the evidence.
Berry, who attracted a neighbor's attention Monday to free herself and two other women following a decade in captivity, and DeJesus Wednesday were released from the hospital where doctors checked their conditions.
Berry's sister Beth read a statement to hundreds of reporters and gawkers from the front lawn of a house, festooned with a sign reading, "Wish it, dream it, do it." Beth Berry urged people to respect the family's privacy and thanked them for their support.
A few hours later, a cheer went up as DeJesus returned to her home.
Sandra Ruiz, DeJesus' aunt, read a statement thanking police and others for their support and seeking privacy to let the family "heal."
"There are not enough words to say or express the joy that we feel for the return of our family member Gina," she said, asking the community to allow the justice system to take care of the suspects and not to retaliate against them or their families.
Earlier, police Chief Michael McGrath confirmed chains and ropes were found in the house where the three young women were held.
"We have confirmation that they were found, and there was chains and ropes in the home," McGrath said.
The women were allowed in the back yard "once in a while," he told NBC News.
He described the physical well-being of the women as "very good, considering the circumstances."
Charges were expected to be filed Wednesday, the chief said.
The daughter of primary suspect Ariel Castro is in prison in Indiana for trying to kill her 11-month-old daughter, The Indianapolis Star reported.
Emily L. Castro, 25, was sentenced in February 2008 to 30 years in prison after a judge found her guilty but mentally ill for attempting to murder the child by stabbing her in the neck four times in April 2007.
Cleveland officials said they would investigate criticism of how an emergency dispatcher handled the call that freed the three imprisoned women and also acknowledged authorities had visited the home twice previously but didn't go inside.
Public Safety Director Martin Flask said his department, which oversees Cleveland police, was looking into criticism the emergency dispatcher was rude toward Charles Ramsey, the resident hailed as a hero who called 911 after hearing captive Amanda Berry frantically screaming, "Help, help me out!" through the front door open just wide enough to fit a hand through.
Ramsey said when he called 911, the dispatcher didn't take him seriously.
Then after Ramsey kicked in the door and freed Berry, and she tearfully told the dispatcher herself who she was and described her alleged captor, the dispatcher hung up before police arrived, Flask said.
"While the call-taker complied with policies and procedures which enabled a very fast response by police, we have noted some concerns which will be the focus of our review, including the call-taker's failure to remain on the line with Ms. Berry until police arrived on the scene," Flask said in a statement.
"Please be assured that this matter will be investigated, and if necessary, appropriate corrective action taken," the statement said.
Berry, DeJesus and Knight were rescued Monday night after allegedly being held for a decade and were reunited with their families following examination at a Cleveland hospital.
A police source told WEWS-TV, Cleveland, the women had "multiple" pregnancies and one of the women had two to three miscarriages because she was malnourished.
Among the people rescued from the house was a 6-year-old girl believed to be Berry's daughter.
The dispatcher procedure issue came hours after officials acknowledged authorities were called to the house at least twice in recent years but never went inside.
"We are happy that they are returned to us," Mayor Frank Jackson told reporters. "But we have several unanswered questions. Why were they taken, how were they taken and how did they remain undetected in the city of Cleveland for this time?"
Officials said that would be investigated, too.
The home where the alleged imprisonment took place was owned by Castro, 52, a former school bus driver born in Puerto Rico who was arrested along with his brothers, Pedro Castro, 54, and Onil Castro, 50, officials said.
Officials said Department of Children and Family Services investigators went to the home in January 2004, when two of the girls were missing, because Ariel Castro had left a child on a school bus.
Investigators "knocked on the door but were unsuccessful in connection with making any contact with anyone inside that home," Jackson said at the news conference.
He said officials had "no indication that any of the neighbors, bystanders, witnesses or anyone else has ever called regarding any information, regarding activity that occurred at that house on Seymour Avenue."
Neighbors, however, told news media calls were made to police several years ago about a woman crawling around in the home's yard.
Ariel Castro also was interviewed by police in the 2004 instance, officials said. Castro said he had "inadvertently" left the child on the bus.
He was called before a disciplinary hearing in 2009 for negligence and disregard for the safety of passengers and was fired in November 2012 after another "demonstration of lack of judgment," school district records indicate.
Police said Knight vanished in 2002 at age 20. Berry disappeared a day before her 17th birthday in 2003 after calling her sister to say she was on her way home from her job at Burger King. DeJesus vanished at age 14 a year later on her way home from school.
They were discovered just a few miles from where they disappeared.