The no-confidence vote came amid conflicting accounts of government responsiveness to the rebel seizure and sale of Libyan National Oil Corporation crude oil. The government claimed on Monday it took control of the tanker, Morning Glory, carrying the stolen oil, but the rebels rejected the government's claim.
Zeidan had been politically weakened for months following infighting with Islamists.
One man standing outside the building where the lawmakers voted Tuesday said, "We have nothing personal against Zeidan, but his government has failed the people."
Following the vote, Defense Minister Abdallah al-Thinni was sworn in as interim prime minister until a replacement is named in two weeks' time.