Yoshihiko Noda, a lawmaker with Japan's main opposition party, denounced Abe for his decision to commute from his private residence in Tomigaya, Shibuya, on Friday night, when North Korea launched what may have been an intercontinental ballistic missile.
Noda said Abe lost precious minutes immediately after North Korea's late Friday provocation -- time that would not have been spent in a chauffeured vehicle if he chose to live in the official residence.
"I cannot believe [Prime Minister Abe] still commutes from his private home to the prime minister's house," Noda, a former prime minister, told reporters. "Even though he is the chief commander of the self-defense forces he is extremely heedless of crisis management."
Abe's choice of residence is not new. The prime minister never fully moved to his official residence in Tokyo since assuming office in December 2012, Kyodo News reported Monday.
Pyongyang test-launched its latest missile at 11:42 p.m. Friday.
Abe left his house at 12:13 a.m. and arrived at central command, the prime minister's residence, at 12:22 a.m.
For 40 minutes, Japan's forces were not able to respond to the provocation.
The reasons for Abe's choice are unclear.
Kyodo reported Abe has previously said living at the official residence would not allow him to "separate his work and his life."
But the prime minister's residence was also the site of assassinations on Feb. 26, 1936, when a group of young Japanese military officers killed several top officials during an attempted coup.
In some circles, the residence is rumored to be haunted by a ghost in military uniform, according to Japanese media.
"There was enough warning North Korea would launch a missile," Noda said. "It is a big problem. [Prime Minister Abe] must leave his private residence."