The 61-year-old actor reflected on his short-lived marriage to the 41-year-old actress in the November issue of GQ magazine.
"I never felt good enough for her," Thornton told the magazine, explaining that Jolie was involved with several social causes and frequently invited to celebrity events.
"I'm real uncomfortable around rich and important people," he added. "I like how I am."
Thornton and Jolie, who co-starred in the 1999 movie Pushing Tin, split in 2002 after two years of marriage. The actor had shared similar sentiments about the relationship on the ABC series Nightline in 2012.
"She has one way she wanted to live her life and I had another way to live mine and I was too insecure," he admitted. "I did feel like the Phantom of the Opera hiding in the catacombs. People have actually said that I didn't deserve to be with her."
Thornton and Jolie were known for their unconventional displays of affection, including wearing vials containing each other's blood. The actor told The Hollywood Reporter in 2014 that the romance was a "whirlwind."
"I mean it was a crazy time," he said. "She thought it would be interesting and romantic if we took a little razorblade and sliced our fingers, smeared a little blood on these lockets and you wear it around your neck."
Thornton has since married Connie Angland and shares 12-year-old daughter Bella with his wife. Jolie filed for divorce from third husband Brad Pitt in September after two years of marriage and a decade together.