Wilkerson, a retired Army colonel, made the observation Friday during an interview on MSNBC about a remark former New Hampshire Gov. John Sununu made on CNN Thursday, when he said Powell endorsed Obama for president because both men are black.
Sununu, a co-chairman of Republican nominee Mitt Romney's campaign, backed off the comment Friday.
"My party, unfortunately, is the bastion of those people -- not all of them, but most of them -- who are still basing their positions on race," Wilkerson said. "Let me just be candid: My party is full of racists."
Wilkerson called Sununu's comment "an unfortunate slip of words."
"The real reason a considerable portion of my party wants President Obama out of the White House has nothing to do with the content of his character, nothing to do with his competence as commander-in-chief and president, and everything to do with the color of his skin," he said. "That's despicable."