"If the Democrats sweep (the November elections), then we'd have no ability to fight back and get our message out," Hastert told Time magazine.
Several critics - both liberal and conservative -- have called for Hastert to resign in the wake of the Sept. 29 resignation of former Rep. Mark Foley, R-Fla., for sexually explicit electronic messages sent to male congressional pages. Questions were raised about what the speaker knew and when he knew it, Time said.
When a 16-year-old male House page sent an electronic message he received from Foley in 2005 to his sponsor, it appeared that Foley received a private warning and retained his co-chairmanship of the House Caucus on Missing and Exploited Children, the magazine said.
In the latest Time poll, nearly 80 percent of those who responded said they knew of the scandal, and two-thirds of those respondents said they thought Republican leaders had tried to conceal it.
Among registered voters polled, 54 percent said they were more likely to vote for a Democratic congressional candidate, compared to 39 percent who said they preferred Republican candidates.
Pregnant Mila Kunis wins 'Best Villain' at MTV Movie Awards
'Happy' fan videos make Pharrell cry [VIDEO]