Firefighters said they arrived at the Gloucester County house about 2:30 a.m., only to learn that one of three adults who had escaped the burning house had gone back to get the children and was still inside, The (Norfolk) Virginian-Pilot reported.
The two surviving adult relatives were taken to the hospital to be treated for injuries that were not thought to be life-threatening, Assistant Fire Chief C.W. Miller said.
The names of the deceased had not been disclosed, the newspaper said.
The deceased children were ages 7, 2 and 6 months, said Maj. Ryan Cookson of the Gloucester County Sheriff's Office.
The fire was believed to have been an accident, and was under investigation by the Sheriff's Office and Virginia State Police, the newspaper said.
The fire did not spread to any other structures and no firefighters were hurt in the course of their duties, the newspaper said.
Moore to attend retreat in to avoid Kutcher's wedding
'How to Train Your Dragon 2' releases 5-minute clip