The recording artist's publicist issued a statement saying the single-vehicle accident was caused by "aggressive pursuit by the paparazzi."
However, INF owner Chris Doherty told TMZ all the photographers on scene of the accident work for him and didn't arrive until after the crash.
"No photographers were following him at the time he crashed," Doherty said. "He crashed his car and it's convenient for him to blame us. He needs to be careful with his defamatory statements."
Doherty also said police talked to his photographers at the scene and "appeared to be satisfied" they did not cause the collision.
Brown's representative told TMZ the singer is considering suing the photographers because of the crash.
No injuries were reported and the car was towed for repairs, police said.
Swim Week Miami Beach 2014 [PHOTOS]