Gossip mongers hounded the 27-year-old reality television star and sock designer after his plane from Paris landed at LAX on Saturday.
According to TMZ, Rob ditched the wedding because he felt insecure about his appearance in wedding photos.
"Rob did come to Paris but skipped [the rehearsal dinner at] Versailles and the wedding," a source told People. "He just didn't want to see a bunch of people both nights and really just is not feeling like himself."
Kardashian took to Twitter last week to deny rumors that he had been to fat camp, saying he was "aware" of his weight gain but was working hard in the gym.
He has since deleted all of his tweets posted before Memorial Day.