The trouble began Sunday at the high-security U.S. Penitentiary near Florence when a group of prisoners began openly celebrating the Nazi leader's birthday in the recreation yard.
U.S. attorney Troy Eid said it appeared that white supremacists began attacking minority inmates.
KMGH-TV in Denver said as many as 100 men were eventually caught up on the melee. Two were killed and five had to be hospitalized.
The prison is one of three separate facilities in a complex that includes the ultra-high security Supermax prison where prisoners such as unabomber Ted Kaczynski and terrorists Mohammed A. Salemeh and Richard Reid are locked up.
Ohio bar shooting arrested, charged with murder
LGBT community has 'bullied the American people': Bachmann