Boehner, R-Ohio, had proposed raising taxes on incomes of more than $1 million and a companion bill that eliminated defense cuts mandated by the Budget Control Act as a backup in case negotiations with the White House failed to produce an agreement.
The Bush-era tax cuts and the across-the-board spending cuts kick in Jan. 1 unless Congress acts.
Boehner issued a statement saying a vote on his plan had been canceled "because it did not have sufficient support from our members." Earlier, the House had voted to advance the measure.
"Now it is up to the president to work with Senator Reid on legislation to avert the 'fiscal cliff,'" Boehner said in a statement, adding the House already had passed a bill that would extend the Bush-era tax cuts for everyone.
House Majority Leader Eric Cantor said with the cancellation of the vote, legislative business for the week had concluded.
"Members are advised that the House will return for legislative business after the Christmas holiday when needed," Cantor's office said in a statement.
Party leaders had expressed confidence throughout the day but announced shortly before 8 p.m. the vote had been canceled -- a major defeat for Boehner, who has had trouble controlling his caucus since he assumed the speaker's job, The Hill noted.
President Obama vowed to continue working on a solution.
"The president's main priority is to ensure that taxes don't go up on 98 percent of Americans and 97 percent of small businesses in just a few short days," a statement issued by the White House said. "The president will work with Congress to get this done and we are hopeful that we will be able to find a bipartisan solution quickly that protects the middle class and our economy."
The Office of Management and Budget also issued a statement calling Boehner's approach unbalanced and noted the "destructive" cuts that both the president and Congress want to avert were mandated so that all sides would work together to come up with a solution.
White House press secretary Jay Carney called Plan B "an exercise in futility."
"The Republicans in the House have decided to run down an alley that has no exit while we all watch," Carney told reporters.
Boehner earlier told reporters his plan was born out of frustration with the White House.
"For weeks the White House said if I moved on rates, that they would make substantial concessions on spending cuts and entitlement reform," Boehner said. "I did my part, they've done nothing."
Boehner has said Obama's latest proposal would raise more than $1.3 trillion in new tax revenue but cut spending only $930 billion.
The White House said Obama's most recent offer includes roughly $1.2 trillion in spending cuts.
Republicans said too many of Obama's proposed spending cuts came from items such as reduced interest payments on the federal debt, and said it was up to Obama to come up with more cuts.
Boehner counted interest savings as a spending cut in past deficit deals, The New York Times said.
Obama said in a news conference Wednesday he would reach out to congressional leaders in coming days.
He also told reporters he found it "puzzling" Boehner would offer a fallback plan when the two sides were only a few hundred billion dollars apart, and added he would be willing to offer more spending cuts.
"If you look at Speaker Boehner's proposal and you look at my proposal, they're actually pretty close," Obama told the news conference. "There are a few differences, but we're right there."