The 5-2 vote was held Monday night, the Asheboro Courier-Tribune reported.
The novel, published in 1952, won the National Book Award and was ranked 19th on the Modern Library's list of the 100 greatest English-language novels of the 20th century. But one board member, Gary Mason, said he found no "literary value" in it.
"Invisible Man," written in the first person, is the story of a black man born in the south. As he tells his tale, he is living underground in a forgotten section of basement in a New York City apartment building.
The book was assigned this year as one of three Randleman High School juniors could read during the summer. The book became controversial when a parent, Kimiyutta Parson, complained, saying the language and sexual content were not appropriate for teenagers.