Court papers say the 3,200-year-old Assyrian tablet was looted from the Vorderasiatisches Museum in Berlin at the end of World War II, The New York Times reported.
It remains unclear how the survivor, Riven Flamenbaum, who died in 2003, got the tablet. He was liberated from the Auschwitz concentration camp in 1945.
His children discovered the tablet, about the size of a passport photo, after Flamenbaum's death.
The German museum had sued seeking return of the tablet but a Nassau County (N.Y.) Surrogate Court judge ruled in favor of the Flamenbaum family in 2010.
The court noted the museum had never reported the tablet stolen and said it was not possible to determine how Flamenbaum had acquired it.
The decision by the appeals court in Brooklyn reversed the Nassau County court decision.
Raymond J. Dowd, the lawyer who represented the museum and has represented the families of Holocaust victims attempting to recover lost art, said: "The principle that property taken unlawfully should be returned is consistent with the rights of Holocaust victims. This precedent will help those seeking return of stolen works that are [in] museums not only in the U.S. but throughout Europe."
But Seth A. Presser, a lawyer for the family, said the appeals court decision "caused a remarkably inequitable result" and would be appealed.
Presser said the family did not plan to sell the tablet.
A daughter of Flamenbaum had told AOL News the tablet was all her father "had left from 'that bitter time' and he wished to hand it down to his children and future generations to serve as a reminder of the brutality and decimation of his family at the hands of the Nazis."
2014: NFL Cheerleaders [PHOTOS]
Ex-Navy SEAL probed over bin Laden revelations