Police discovered the four men -- ages 80, 74, 65 and 50 -- in a converted garage attached to the home of Essie Mae Scranton Friday after receiving a 911 call, the Houston Chronicle reported.
Police said the men are believed to be homeless and were being held captive in the garage.
Police arrested Walter Jones, Scranton's grandson, at the scene and charged him with injury to the elderly by act and injury to the elderly by omission.
Investigators are looking into whether the men were being forced to hand over their government aid, including disability, Social Security and veterans' checks.
"What the men initially told us was they were lured there with the promise of cigarettes and food," Houston police spokeswoman Jodi Silva told CNN.
Silva said it was unclear how long the men had been held captive, but said one man said it felt like 10 years.
"Whether it goes into years or not is unknown," Silva said.
Silva said police were looking into whether four women found inside the house, not the garage, were also being held captive. Three of the four women suffer from physical disabilities and mental illness.
Duggar sisters unveil Christian dating rules in new book
Ohio bar shooting arrested, charged with murder