The U.S. Justice Department has accused the five-term Maricopa County sheriff of racial profiling.
Black community leaders joined Councilman Michael Johnson Thursday on the steps of the federal courthouse to call for an immediate change in leadership in the Sheriff's Office, The Arizona Republic reported.
Many of the speakers compared Arpaio to Birmingham, Ala., police officers who abused civil rights activists in the 1960s, the newspaper said.
Arpaio, 78, a former U.S. Drug Enforcement Administration agent known for his get-tough stance on illegal immigrants, has refused to resign.
"They're ganging up on me," Arpaio said. "They know I can win next year. They think they can drive me out. It's never going to happen."
The Justice Department has accused Arpaio of fostering discrimination against Latino residents through the sheriff's immigration-enforcement efforts and treatment of Hispanic inmates in Maricopa County jails.
Federal prosecutors say they can file a lawsuit under the federal Civil Rights Act if the sheriff refuses to cooperate or does not reach an agreement with federal officials.