Netanyahu and Barak informed Maj. Gen Yoav Galant of their decision several hours after Attorney-General Yehuda Weinstein decided the state would not represent Galant before the High Court of Justice in a land dispute.
Weinstein said it was "doubtful whether it is morally appropriate to maintain the appointment," Ma'ariv said.
"In light of the situation after the attorney general's decision and the consequences arising, the prime minister and defense minister have no choice but to cancel the nomination and restart the process of choosing the next chief of staff," a statement said.
Barak said he planned to propose to the cabinet he appoint the deputy Chief of Staff Maj. Gen. Yair Naveh to become acting chief of staff for 60 days, the newspaper said.
Outgoing Chief of Staff Lt. Gen. Gabi Ashkenazi is due to wind up his stint as army chief on Feb. 14.
Last week, Israeli Comptroller Micha Lindenstrauss issued a report on alleged improprieties in Galant's handling of building permits for his home. The report also alleged Galant lied to the Israel Lands Authority about a plot of land.
Oscar Pistorius receives 5-year sentence