Swank, actor Jean-Claude Van Damme, violinist Vanessa Mae and singer Seal were blasted by human-rights groups for attending the Oct. 5 event celebrating the birthday of a politician accused of torturing dissidents, killing opponents and encouraging the abuse of women.
The British newspaper reported Sunday Swank, a two-time Oscar winner, fired Jason Weinberg, her manager of eight years, last week due to the scandal that erupted after she attended the celebration.
Swank later apologized for being a guest at the event and vowed to donate her six-figure appearance fee to charity.
Two of Swank's representatives at the Hollywood talent firm Creative Artists Agency also are expected to be dropped from her team since they are the ones who directly booked the event, the newspaper said.
"Hilary values her liberal credentials and is close to Michelle Obama. She's really upset by what happened," a source close to the actress told The Independent. "It's partly her fault. But what can she do? She can hardly fire herself."