The incident took place Jan. 13 when West's wife, Kim Kardashian, entered the office at the same time as the alleged victim, the Los Angeles Times reported Friday.
The alleged victim held the door open for Kardashian and "words were exchanged, and both parties report [the] victim used a racial epithet," the district attorney's office said in a statement.
Kardashian relayed the incident to West via cellphone, and West showed up at the office later to allegedly punched the victim an unknown number of times.
The district attorney rejected the case because West agreed to pay the alleged victim $250,000 and "no significant injuries were documented."
The district attorney said the alleged victim wouldn't speak to prosecutors after the civil compromise, TMZ reported.