Police and fire officials said the death toll could go higher as they feared there were more victims trapped beneath the debris of the Om Sakthi Fireworks factory on the outskirts of Sivakasi, the Press Trust of India reported.
Hundreds of workers were inside the factory when the first blast occurred, India's NDTV reported.
Witnesses said all 60 rooms of the building were gutted, including 20 where chemicals were stored.
A portion of the factory's manufacturing unit collapsed on top of many of the employees.
Fire and rescue services could not get near the factory initially because of fear more explosions would occur, PTI said.
Police said Om Sakthi Fireworks was involved in making sophisticated fireworks that are more dangerous than those that simply make loud noises.
Sivakasi is India's largest hub for the manufacture of matches and firecrackers.