The discovery was made two days after FBI and New York police investigators began going through Burke's home in Queens, the New York Daily News reported Thursday.
"Unknown remains have been recovered and we are examining them," city medical examiner's spokeswoman Ellen Borakove said.
Investigators have not said whose remains they have been seeking but they are related to a more than 30-year-old case.
Burke, of the Luchese crime family, died in prison of cancer in 1996. He was the alleged mastermind behind the heist of $6 million at Kennedy Airport in 1978.
The mobster allegedly killed a number of men who helped pull off the heist to keep them from talking to police, the newspaper said.
Burke's wife, Mickey, who no longer lives at the residence that was searched, said she has "no idea" for who or what investigators were looking.