Pitt, 49, told Esquire magazine he frequently forgets what someone looks like after he meets them and he doesn't recognize the person again when their paths cross later.
"So many people hate me because they think I'm disrespecting them," Pitt said in the interview, adding he briefly attempted to deal with the condition by asking people to remind him where they had met before.
"People were more offended," he said. "Every now and then, someone will give me context, and I'll say, 'Thank you for helping me.' But I piss more people off. You get this thing, like, 'You're being egotistical. You're being conceited.' But it's a mystery to me, man. I can't grasp a face, and yet I come from such a design/aesthetic point of view."
The father of six young children and the fiance of actress Angelina Jolie said he plans to get officially tested for prosopagnosia but until then avoids people he doesn't know well.
"That's why I stay at home," he said.
Pitt will soon be seen in the zombie apocalypse movie "World War Z."
Wikipedia said a person can be born with prosopagnosia or the disorder can be a result of brain damage. There is no cure for the condition but people who suffer from it can develop strategies to help them cope with it, the website said.