The New York Post reported Friday that U.S. District Judge Alvin Hellerstein said he realized the families got a "disproportionately large" sum when the law firm representing them asked for a 25 percent cut instead of the standard 15 percent.
Hellerstein was the judge who originally approved the settlement.
"How can he take back our money?" asked Sara Golinski, the daughter of Ronald Golinski, who was killed in the Pentagon attack.
The judge said the Maryland law firm of Azrael Gann and Franz manipulated the settlement process by waiting until the last minute to settle and then demanding higher amounts than everyone else got.
Hellerstein said that the settlements all involve "modest wage earners at the Pentagon."