Sotomayor, who wrote the dissent in an April ruling that allowed Michigan to ban affirmative action, said race-based affirmative action programs helped people like her and President Obama rise above difficult childhoods to the very top of their fields.
And unlike using race, she said, the statistics show economic or geographic considerations don't work to improve diversity.
"It just doesn't," she told ABC's George Stephanopoulos on This Week Sunday. "If you start from the proposition that advantage inures to a background that's privileged -- and it does."
"Legacy admission is a wonderful thing, because it means even if you're not as qualified as others you're going to get that slight advantage."
"But what does 'qualifications' mean in an academic setting?" she continued. "A place like Princeton... could fill their entire beginning freshman class with students who have scored perfectly on undergraduate metrics. They don't do it because it would not make for a diverse class on the metrics that they think are important for success in life."
Sotomayor said she agrees with Obama that his children should not receive "special consideration" for their race, adding that race shouldn't be considered in a vacuum.
"But even privileged people will show you dramatic accomplishment that doesn't go just to grades," she said.