The kittens were given away outside a Wal-Mart in McDonough, The Atlanta Constitution-Journal reported.
Three of the kittens in the litter and the mother cat were turned in last week. Two of those kittens and the mother tested positive for the potentially fatal disease, health officials said. The fourth kitten was turned in Saturday.
However, "The virus was in its incubation stage, and wasn't at the point where it was transmittable," said Vince Farah, a spokesman with the Henry County Animal Care and Control department.
The ordeal began when a woman took in a pregnant, stray cat that later gave birth. On Sept. 9, the woman and her friend brought the kittens to the Wal-Mart parking lot and gave them away.
A man who adopted one of the kittens brought it to the vet after it became ill and test results returned Sept. 19 showed the cat had rabies. Henry County officials scrambled to locate the other cats.
Rosie O'Donnell unveils nearly 50-pound weight loss
Disney's 'Jessie' to feature network's first engagement