Early results indicated Abe's Liberal Democratic party suffered heavy losses in the House of Councilors in Sunday's election, the Mainichi Daily News reported.
"We tried our best and felt we made some progress, so the results are extremely disappointing," Abe said. "I must push ahead with reforms and continue to fulfill my responsibilities as prime minister."
The House of Counselors has limited power compared to that of the lower House of Representatives. But the Liberal Democratic Party faced losing control of the upper chamber for the first time since the party was founded more than 50 years ago, Kyodo News Service reported.
Abe's popularity has plummeted since he became prime minister 10 months ago. His party has been hit by a series of scandals that have led to the suicide of one minister and the resignation of two others.
The results of two by-elections for the lower house will not affect the LDP's significant majority there.