However, 80 percent of those polled said they oppose a congressional proposal to cite Jackson as a "global humanitarian."
The responses of African-Americans on both questions were significantly different than those of other races, Zogby said.
There were also differences based on political ideology, with Democrats and liberals more likely to favor the resolution and not see the sexual allegations as part of Jackson's legacy.
The Zogby Interactive poll included the responses of 4,517 adults from July 10-13. The survey has a margin of error of plus or minus 1.5 percent.
The pop icon was dogged by child molestation allegations throughout his life, but never convicted of a crime against children.
Asked what should be Jackson's lasting legacy, 24 percent of all adults polled said his humanitarian accomplishments, 12 percent said the allegations of child molestation, 36 percent said both, 23 percent said neither and 6 percent said they were not sure.
Meanwhile, 62 percent African-Americans said humanitarian accomplishments, 3 percent said the allegations, 23 percent said both, 7 percent said neither and 4 percent said not sure.
Thirty-nine percent of liberals said humanitarian accomplishments, 3 percent said the allegations, 36 percent said both, 16 percent said neither and 6 percent said not sure.
Of those respondents who describe themselves as moderates, 34 percent said Jackson's humanitarian accomplishments, 8 percent said the allegations, 29 percent said both, 25 percent said neither and 5 percent said not sure.
Amongst conservatives, 8 percent said his humanitarian accomplishments, 20 percent said the allegations, 40 percent said both, 26 percent said neither and 6 percent said not sure.
Zogby said when the percentages of those choosing both statements as part of Jackson's legacy are combined with those agreeing with each statement, 60 percent cite his humanitarian work and 48 percent cite the allegations.
For African-Americans, combined totals are 85 percent for his humanitarian work and 26 percent for the allegations.
As for political ideology using the same combined percentages, 75 percent of liberals cite Jackson's humanitarian work, compared to 48 percent for conservatives.
2014: The Year in Music [PHOTOS]