Fifty-four percent of residents said in The Washington Post poll that Gray should resign, while 37 percent said he should not and 9 percent had no opinion.
The poll comes after three campaign associates pleaded guilty to felonies, including charges linked to secret payments to an opponent and a "shadow campaign" to get Gray elected.
Gray, who is African-American, also has lost support among his political base, with 48 percent of African-Americans saying he should resign. Nearly 40 percent of those who voted for him in 2010 said they think he should quit, the poll found.
The mayor has said a federal investigation into his campaign has had no effect on how his administration is running the city. But only one in three respondents said they approve of the job he is doing as mayor and 44 percent said the city would be better off if he left office.
Though Gray has not been accused of wrongdoing, only 22 percent said they view him as honest and trustworthy.
The telephone poll of 1,002 Washington, D.C., residents was conducted Sunday through Tuesday and has a margin of error of 4 percentage points.