At least one guard was killed and four people were injured in the attack, which lasted nearly two hours, officials said.
The fighters, who helped out former dictator Moammar Gadhafi were seeking unpaid stipends promised under a payment plan that was suspended last month, the Tripoli Post reported. The rebels are also seeking foreign medical treatment for those injured in last year's uprising
Prime Minister Abdurrahim al-Keib was in the building at the time of the attack but was uninjured, an adviser told the BBC.
A government spokesman said the former rebels were armed with light to medium weapons, including machine guns and rocket launchers.
Witnesses said the fighters were part of a militia from Yafran, which is populated by the Berber ethnic minority.
The BBC said it is the second attack on the building in the last two months. Libya has been trying to get thousands of militia members to disarm following last year's uprising.