The meeting is set for next week in Switzerland.
CNN reported Sunday that the United States and other countries have withdrawn from the conference over a document said to single out Israel. Australian Foreign Minister Stephen Smith said Israel was being unfairly targeted.
"Regrettably, we cannot be confident that the Review Conference will not again be used as a platform to air offensive views, including anti-Semitic views," he said.
Canada, Israel, Italy and Sweden have already announced they are boycotting the conference, the news network reported.
The boycott by the Obama administration has drawn criticism from the Congressional Black Caucus, which said it was "deeply dismayed" by the decision.
"Had the United States sent a high-level delegation reflecting the richness and diversity of our country, it would have sent a powerful message to the world that we're ready to lead by example," a statement said.
Celebrity Families of 2014 [PHOTOS]