Officials said the mixed-use building collapsed early Thursday morning, trapping people inside for about four hours as they waited for heavy-duty equipment to arrive, the Nigerian Tribune reported Friday.
Among the victims were a pregnant woman, her husband and 18-month-old son, officials said.
Adeniran Isiaka, who lives in a nearby apartment building, told the Nigerian Tribune he heard rumblings and a loud bang.
"We were waiting to eat our early morning fasting food," Isiaka said. "Myself and a friend of mine decided to [play] cards to while away the time when we heard a rumbling sound and later a sudden loud bang The whole area was covered with dust from the collapsed building."
The Tribune said the first floor was a commercial-use area, and the next two floors were for apartments.
A health worker and a member of the Community Development Association told the newspaper on condition of anonymity that the late arrival of emergency crews contributed to the high death toll.
Witnesses said the building's owner fled the site.
Police arrested the spokesman for the National Emergency Management Authority, Ibrahim Farinloye, for allegedly insulting a police officer and telling the officer to stop sending residents away from the collapsed building, the Tribune reported.