TMZ Sports released a video of Smith on Sunday, which showed him kicking the man when he was on the ground and then punching him when he stood up.
The alleged vandal ran away after the physical altercation. Smith said in a video posted online that the man broke a window on his truck.
Smith noted he was parked in a residential area and not close to any stores where looting occurred during the protests.
"I just want you all to know right now, before you all see this [expletive] somewhere else," Smith said in the video. "One of these little [expletive] white boys didn't know where he was going and broke my [expletive] window in my truck. Broke my [expletive]. This was a residential area and no stores over here.
"So when the footage comes out and you all see it, I chased him down and whooped his [expletive]."
Smith said he wanted to make it clear that his retaliation against the man wasn't racially motivated.
"This ain't no hate crime. I ain't got no problem with nobody and nobody got no problem with me," Smith said. "It's a problem with the [expletive] system. That's it."
Smith, 34, played in 11 games for the Cavaliers last season. The 15-year NBA veteran hasn't been on a roster this season.
Los Angeles is one of many cities in the United States where protests have happened over the death of Floyd, a black man who died last week in Minneapolis after police officer Derek Chauvin placed his knee on his neck for several minutes. Chauvin was charged with third-degree murder after the incident.