Allen, 72, sued the company for $10 million for allegedly using his image in ads without his permission.
The trial had been scheduled to start Monday with Allen expected to be the first witness to take the stand.
However, E! News said the settlement was announced shortly after Allen arrived at the Manhattan federal courthouse Monday morning.
"I am told the settlement of $5 million I am being paid is the largest amount ever paid under the New York right to privacy law," E! News quoted Allen as telling the media outside the courthouse.
He added he hopes the settlement "would discourage American Apparel or anyone else from ever trying such a thing again."