Any decision in the case likely would be applied nationwide.
The Supreme Court is closely divided between four liberals and conservatives, making Justice Anthony Kennedy a key swing vote, though even conservatives expressed skepticism of Prop 8 from the bench Tuesday.
In the Prop 8 argument, Kennedy expressed sympathy for the children of same-sex couples while questioning attorney Charles Cooper, who represented the private proponents of Prop 8, The Huffington Post reported.
"They want their parents to have full recognition and legal status. The voice of those children is considerable in this case, don't you think?" Kennedy asked.
Theodore Olson, a former Bush administration solicitor general, and U.S. Solicitor General Donald Verrilli, the Obama administration's top courtroom lawyer, argued on behalf of the challengers.
California voters approved Proposition 8, the California Marriage Protection Act, in a 2008 vote with slightly more than 52 percent for and nearly 48 percent against. Prop 8 says in part, "Only marriage between a man and a woman is valid or recognized in California."
A federal judge declared Prop 8 unconstitutional and a three-judge appeals court panel in San Francisco agreed 2-1.
Neither California's governor nor its attorney general is defending the law in court. ProtectMarriage -- its sponsor is a state non-profit, California Renewal -- is the official proponent of the proposition and has been allowed to defend it in the Supreme Court.
Last month, the Obama administration told the Supreme Court California's ban on same-sex marriage is unconstitutional.