Interior Minister Gen. Ghulam Mujtaba Patang, who was in charge of the police force, was ousted by a vote of 136 to 60, The Wall Street Journal reported.
Patang, who was appointed less than a year ago, defended his work during his time in office.
"The Ministry of Interior was like a scrapped car with a broken engine," he said. "They put me in this car, I changed the drivers, but the car didn't start because the engine was not working."
Before the vote, Patang told the lower house of Parliament that 2,750 police officers had been killed in action during the previous four months. By contrast, 355 soldiers in the better-equipped Afghan army had been killed in the first three months of the Afghan year.
The veteran police officer will retain the office until President Hamid Karzai names a replacement. The president must nominate someone within 30 days.
Patang claimed one of his deputies had urged lawmakers to remove him.
His ouster comes ahead of presidential elections next spring and the complete pullout of Western troops by the end of 2014.
Notable deaths of 2014 [PHOTOS]
More remains found at MH17 crash site in Ukraine