The 49-year-old actor, who currently has six children with fiance Angelina Jolie, admitted that having a big family is what he'd always wanted.
"I always thought if I wanted to do a family, I wanted to do it big. I wanted there to be chaos in the house. There's constant chatter in our house, whether it's giggling, screaming, crying or banging," he said.
"I love it. I hate it when they're gone. Maybe it's nice to be in a hotel room for a day -- 'Oh, nice, I can finally read a paper'. But then, by the next day, I miss that cacophony, all that life," he added.
In fact, Pitt is so content being a family man that he claims to never have known a better life.
"I have very few friends. I have a handful of close friends and I have my family and I haven't known life to be any happier," he told Esquire.
The reason behind the actor's lack of friends might be his self diagnosed prosopagnosia, a disease which prevents him from remembering new faces. He says the condition makes him seem "egotistical" and "conceited" because he can never remember people he's met.
"So many people hate me because they think I'm disrespecting them," Pitt says adding, "It's a mystery to me, man. I can't grasp a face and yet I come from such a design/aesthetic point of view. I am going to get it tested."
Pitt's full interview with Esquire is available in the physical issue of the magazine, out now.
Celebrity Families of 2014 [PHOTOS]