"I vowed that if my initiative did not succeed, I would resign and I have done so," Jebali said at a news conference after meeting with President Moncef Marzouki.
The BBC reported Jebali said his decision to step down was "a big disappointment," but he did it to "fulfill a promise made to the people."
"Our people are disillusioned by the political class. We must restore confidence," the British network quoted him as saying.
Jebali, who leads the Islamist Ennahda party, had sought to find common ground among the various political factions following the assassination this month of opposition leader Chokri Belaid. He had attempted to put together a caretaker Cabinet of technocrats to guide Tunisia until elections can be held.
"The failure of my initiative does not mean the failure of Tunisia or the failure of the revolution," he said.
Tunisia was where the so-called Arab Spring began two years ago with the ouster of Zine el-Abidine Ben Ali.
Jebali's proposed cabinet failed to win a majority in the National Constituent Assembly members who met Monday to discuss his idea of a non-partisan Cabinet, Tunisia Live reported.
Leaders of several political parties said they wanted a political Cabinet, not a non-partisan one as suggested by Jebali.
However, Mouldi Fehem, representing the opposition Al Joumhouri, told Tunisia Live his party wants a national coalition party that does not exclude any party.
Prior to Jebali's resignation, Fehem said he suspected negotiations would lead to the formation of a government that has both independent and affiliated politicians.
"It would be best if the main ministries remain neutral while all political parties are included in the rest of the government," he said.