The sheriffs were on hand at the capitol for a meet-and-greet, something they take part in every year, KTUL-TV, Tulsa, Okla., reported.
"Everybody in that building knew who we were," Colbert said. "One of the senators ... complained because we were armed in the building."
The sheriffs were told they either had to disarm or leave the building.
"So we all packed up and left," Colbert said.
He said he's not sure which senator made the complaint.
"We're the people that protect these people," he said. "If the state troopers are satisfied that we're OK in that building I'm pretty sure that the legislators should have something else to worry about such as the economy or something."
Colbert said some lawmakers have called to apologize for the incident.