The president presided at a naturalization ceremony for 76 new citizens.
Several people yelled out criticism during his speech in Charlottesville, Fox News reported. One man shouted "This man is a fascist," while another screamed curses.
"We believe in free speech in America," Bush ad-libbed.
The protesters were removed by police.
In the ceremony at Monticello, Bush paid tribute to Jefferson, the nation's third president, and to the immigrants taking the oath of citizenship.
"We honor Jefferson's legacy by aiding the rise of liberty in lands that do not know the blessings of freedom, and on this Fourth of July we pay tribute to the brave men and women who wear the uniform of the United States of America," he said. "We also honor Jefferson's legacy by welcoming newcomers to our land, and that is what we're here to celebrate today."