The announcement came during the two-day China visit of Japanese Prime Minister Yoshihiko Noda, The New York Times reported.
Japan also would seek to acquire yuan-denominated Chinese bonds, which the Times said would allow it to add yuan reserves to its foreign-exchange holdings.
China currently is the world's largest creditor to the United States with most of its foreign exchange reserves of $3.2 trillion held in U.S. dollars. Japan has the world's second-largest reserves with $1.3 trillion.
The Times said the currency deal would help the two to move away from using the U.S. dollar, even as China seeks to make its yuan more widely used.
The U.S. dollar currently is the world's reserve currency. The Chinese yuan is not yet fully convertible.
"Chinese officials have made it clear that they believe the international economy is too heavily dominated by the dollar," Professor Charles Kupchan at Georgetown University in Washington told the Times. "They believe, as part of China's rise, that the international system should move to a more balanced structure."
He said the yuan is not yet ready to compete as a global reserve currency, but the agreement with Japan "could be a baby step in that direction."
As China and Japan are neighbors with likelihood of one becoming the main trading partner of the other, it makes sense for them to trade directly without using dollars, said Professor Jeffrey Bergstrand at the University of Notre Dame.
The Times said the move could help the yuan appreciate against other currencies. China is seen as artificially keeping the yuan undervalued, which helps it run surpluses against its trading partners.