Obama spoke to CBS News senior business correspondent Anthony Mason Wednesday before and after he held a town meeting in Atkinson, Ill., to wrap up a three-day bus trip across the Midwest.
"I don't think we're in danger of another recession but we are in danger of not having a recovery that's fast enough to deal with what is a genuine unemployment crisis for a whole lot of folks out there and that's why we need to be doing more," he said during an interview scheduled for broadcast on the network's "Sunday Morning" program.
Before beginning his family vacation on Martha's Vineyard, Mass., Thursday, the president signed an executive order setting up a government-wide initiative promoting diversity and inclusion in the federal workforce. The White House said the president will unveil a new jobs and economic growth plan after Labor Day.
Obama called his three-day swing through rural Minnesota, Iowa and Illinois a listening tour and said it wasn't a campaign trip. Despite overall employment topping 9 percent, more than 14 million people, agriculture has been doing well because of soaring commodities prices.
Asked if he should be going on vacation, Obama said there was no point in calling Congress back to Washington this summer "because I think that if all we're doing is the same posturing that we saw before the debt limit vote that's not going to encourage anybody."
Still, the economy is the major issue, Obama said, referring to Standard and Poor's recent downgrade of U.S. debt from AAA to AA+.
"I think what the markets were reacting to was the fact that the economy has not grown as quickly as it needs to," he said. "There have been a lot of headwinds: the European debt crisis, Japan, high gas prices from the Arab Spring and what a lot of folks are worried about is that the recovery that we have been on is stalling or not moving as quickly as it needs to."
Obama said public confidence matters and that he hopes House Speaker John Boehner, R-Ohio, will join him in "a grand bargain to deal with long term debt and deficit."
"Here we're talking about closing some loop holes in the tax code, making some modest adjustments in entitlements, paring back a little on programs that don't work," he said.