The exchange occurred in the fourth quarter of the Lakers' 107-99 victory on Monday at State Farm Arena. Play was stopped for a foul with about eight minutes remaining when a woman stood on the side of the court and screamed in James' direction.
James also looked back and spoke to the woman and the people sitting next to her. Arena staff and officials then spoke to the courtside attendees and escorted them out of the area.
"At the end of the day, I'm happy fans are back in the building," James told reporters. "I missed that interaction. I need that interaction. I don't feel like it was warranted [for them] to be kicked out.
"There was a back-and-forth between two grown men. He said his piece and I said my piece. Then, someone else jumped into it and said their piece. I don't think they should have been kicked out."
James also said he thought the referees did what they had to do to make sure players could stay focused on the game. He also said the man he first exchanged words with "went out of bounds" with the comments he directed at him.
James had 21 points, nine assists and seven rebounds in the win. Anthony Davis led the Lakers with 25 points. Trae Young also scored 25 points and had 16 assists to lead the Hawks.
The Lakers host the Denver Nuggets at 10 p.m. EST on Thursday at Staples Center in Los Angeles.