The report says Australia has proposals for 89 coal mines which could more than double annual output from about 430 million tons in 2011 to about 980 million tons by 2020.
China is seen as a key export market for those projects.
But the report, commissioned by HSBC's Climate Change Center of Excellence, points to the changing nature of China's outlook for coal and the impacts on resource-rich Australia, currently the world's largest exporter of coal to China.
"China's demand for coal is changing as a result of environment-related factors, including environmental regulation, developments in cleaner technologies, air pollution, improving energy efficiency, developments in gas markets and political activism," Ben Caldecott, co-author of the report was quoted as saying by Business Green.
"This could lead to less demand from China and lower coal prices, which would increase the risk that Australian coalmines, reserves and coal-related infrastructure become stranded assets," he said.
Moreover, the International Energy Agency in its Medium-Term Coal Market Report released Monday said tougher Chinese policies aimed at reducing the country's dependence on coal will slow the global increase in coal demand.
Although China accounted for about half of the total global coal demand in 2012, it was the country's second-lowest demand growth since 2011, IEA says. Still, the agency predicts China will account for nearly 60 percent of new global coal demand over the next five years.
The University of Oxford report recommends that investors in a dozen upcoming Australian coal projects "seek clarity" on the associated costs.
The report also warns that state governments will suffer if projects are abandoned or mothballed, because less production will reduce royalty payments.
The report comes after Australian Environment Minister Greg Hunt last week approved the expansion of Abbot Point port that will serve the Galilee Basin mines, turning it into one of the world's largest coal ports.
Environmentalists say the port expansion could significantly damage the Great Barrier Reef and critics of the project argue that it is likely unnecessary due to uncertainty facing the coal sector.
"It really is bizarre that approval to dredge an export channel through the Great Barrier Reef would be granted before we even know if there will be any coal mines that will use it," Richard Denniss, fossil fuel analyst at the Australia Institute, was quoted as saying by the Guardian newspaper. "It's like icing a cake before it's been cooked and when it seems increasingly unlikely anyone will even want to eat it," he said.