All of Japan's 54 nuclear power stations were shut down after the March 2011 earthquake and tsunami that devastated a four-generator facility in Fukushima.
The protesters were upset over the government's decision to reopen two reactors at the Oi power plant on the western Sea of Japan coast, the Kyodo news agency reported.
Some of the demonstrators carried signs calling for the resignation of Prime Minister Yoshihiko Noda over the order to restart nuclear power production, the South Asian News Agency said.
The country had a 30 percent reliance on nuclear power for its needs, but has seen a spike in electrical demand with summer weather.
There were no immediate reports of arrests as the protesters surrounded the parliamentary Diet building, the reports said.
Anti-nuclear protests have been staged on a near-weekly basis in Tokyo in recent months, but Sunday's was the largest as the two reactors were brought back into production.