"I am so tired of the personal attacks and I've answered all those questions," Spitzer, who has been living apart from his wife, said at a campaign stop outside a New York subway station in response to a reporter's question.
He would not say "no" when asked two more times, saying voters didn't want to hear gossip about his personal life.
"The public cares about what I did in government. That's what I'm going to be talking about and that's what the public is going to be voting on -- based on what I did in terms of trying to clean up Wall Street, which got a lot more attention than the other things we did," he said.
Spitzer, 54, and his wife, Silda Wall Spitzer, 55, have been living separately, at least part of the time, his office told the New York Post in May.
He has been "spending time" in a Fifth Avenue apartment in a building owned by his father, real estate tycoon Bernard Spitzer, spokeswoman Lisa Linden told the newspaper.
Elliot Spitzer wants to "spend more time with his ailing parents, who live there," Linden said.
Bernard Spitzer is reported to have Parkinson's disease, a degenerative disorder of the central nervous system.
Silda Spitzer -- a former banking-industry lawyer who co-founded the GenerationOn non-profit, which seeks to foster social responsibility in young people -- has made no appearances on the campaign trail since her husband declared his candidacy early last month, the Post and New York Daily News reported.
She helped collect voter signatures so he could secure a spot on the Democratic primary ballot and has shown up at the campaign office, the News said, citing sources.
The Post reported a week ago Silda Spitzer told friends she planned to divorce her husband after the campaign.
"She has been telling friends: 'This is too hard. This is too rough,'" the Post cited a source as saying.
Linden told the News Thursday Spitzer's campaign would not comment on "speculation" about his private life.
"He's running on the issues that matter most to New Yorkers and to be an independent voice in the comptroller's office. That's what is at stake in this election," she said.
Spitzer canceled a Thursday evening forum with Democratic primary opponent Scott Stringer, 53, the Manhattan borough president.
The Stringer campaign criticized Spitzer's last-minute cancellation, saying Spitzer "cannot seem to find the time for the residents" of the neighborhood where the forum was to take place.
Spitzer resigned as governor in 2008 after admitting he routinely hired high-price prostitutes.
Another hopeful for the comptroller's post is Kristin M. Davis, a former madam involved in Spitzer's 2008 prostitution scandal. Davis is running on the Personal Freedom Party ticket.
Celebrity Families of 2014 [PHOTOS]