In an appearance on "Fox News Sunday," Bayh said the spending plan does not do enough to reduce the nation's debt.
"In the long run, I think the deficits and the debt are too high," Bayh said. "We need to get those under control. So that was the reason for my vote there."
Bayh and Sen. Bill Nelson, D-Neb., were the only Senate Democrats to oppose Obama's budget plan.
"Is there any reason to stop, you know, acting aggressively? No," Bayh said. "This is going to take longer than we would like. There's no magic answer. But there is some reason for hope. I agree with the president. We do need to stimulate the economy. The government needs to step in."
Also appearing on the program, Sen. Tom Coburn, R-Okla., said it was too early to tell whether Obama's economic policies are helping jumpstart the economy. As for healthcare reform, Coburn said he would oppose any proposal for a government-run healthcare system.
"We're very inefficient as a government in accomplishing anything," Coburn said. "And when we move more things to the control of the government, we're going to spend a whole lot of money."