When Ledger died in January at the age of 28, he left everything he had to his parents and sisters in a will written before he fathered his 2-year-old daughter Matilda with ex-partner Michelle Williams.
However, his father Kim's brothers are now criticizing Kim for handling their grandfather's estate so badly 15 years ago he was removed as executor.
"It plunged into enormous debt," Mike Ledger told People magazine. "Our only vested interest is to assure that Matilda is well looked after."
Although Kim Ledger recently vowed to see to it that Matilda "will be taken care of," his brother Mike said, "When you are talking about large sums of money like this, it should be an independent executor, but Kim hasn't chosen that way."
Kim Ledger's brother Hayden told the magazine Williams' father asked him if Kim Ledger was capable of managing Heath Ledger's estate.
"I just said, 'No, not if he handles it like ours,'" Hayden said.
Pregnant Mila Kunis wins 'Best Villain' at MTV Movie Awards
Charlize Theron not engaged to Sean Penn 'yet'