The New York Daily News said 42-year-old Patrice Lamumba Moore screamed for help repeatedly when the ceiling-high wall of books fell on him Saturday, but no one responded to his calls until Monday afternoon.
Neighbors explained the delay by saying Moore was always screaming and talking to himself.
The former book and magazine peddler said he has sworn off collecting.
"It nearly killed me," he said. "But I've retired from the books. I was a rapper once. I'll do that again."
Once fire officials rescued him, they ordered the building vacated, citing numerous safety code violations, in effect evicting Moore. He is upset that when the debris was removed to the dump, all of his clothing and other possessions were thrown out, too.
He is already a recipient of public assistance, and Moore said he hoped the social services department would help him get his books in order.