The action came as Ford's brother urged him to take a leave of absence, CNN reported.
Ford protested the vote, saying, "Obviously, I can't support this." He added, "I perfectly understand where they're coming from."
City Solicitor Anna Kinastowski said during the session that "council delegated those powers and council has the ability to take those powers away.
Doug Ford Jr., a Toronto City Council member, urged his younger brother to take the leave, which would let him keep his position as mayor, the U.S. broadcaster said, citing a council member.
CNN did not say how Rob Ford, 44, responded to his older brother, 48.
Council members considered a number of actions against Ford, including cutting his budget. Some council members even refused to call him by his title, Canadian newspaper The Globe and Mail reported.
The council on Wednesday voted 37-5 in favor of Ford taking a leave.
The council does not have the power to remove Ford from office or force him to take a leave of absence.
A proposed resolution called for "all powers and duties which are not by statute assigned to the mayor" to be given to Deputy Mayor Norm Kelly, who is also on the City Council.
"Let's put a firewall around the mayor," council member Janet Davis said in remarks quoted by The Globe and Mail.
"He has no legitimacy here. Every time he speaks he embarrasses us even further," she said.
The measure will be formally debated in a special session Monday.
Most council members turned their backs on the mayor when he addressed them Thursday.
At the same time, a Canadian news and opinion network announced the two brothers would soon have their own weekly TV talk show.
The show, starting Monday night, will be titled "Ford Nation" and "pick up where they left off" after a Toronto news-talk radio station canceled their show last week, the Sun News Network said in a statement, adding the show could be "Canada's ultimate reality show."
Ford declined to comment on the council's latest moves but is consulting lawyers and plans to fight the latest council moves when they are debated Monday, the newspaper said.
But council member Giorgio Mammoliti, once a member of the mayor's executive committee, accused colleagues of a "left-wing coup" intended to undo the mayor's fiscally conservative agenda.
Ontario Premier Kathleen Wynne told reporters Thursday she considered Ford's actions "truly disturbing" and said her office could give the City Council new tools during the "extraordinary and unique" crisis so the city can function.
Along with those actions, Ford defiantly vowed to sue former employees who detailed a laundry list of erratic behavior to police, including alleged impaired driving, possible cocaine use, involvement with a suspected professional escort, misusing staff resources, uttering racial slurs and making sexually charged remarks.
None of the allegations has been proved in court.
"I've never had a prostitute here. I'm very happily married at home. That's very disturbing against my wife," Ford said Thursday.
Ford also denied allegations he wanted to perform oral sex on a former female staffer. In his denial, which was broadcast live on national television, he used a vulgar word to describe a woman's genitals.
"I might have had some drinks and driven," Ford also said before heading to the City Council meeting.
Davis expressed outrage at the meeting, calling Ford's language minutes earlier "distasteful, hateful [and] misogynist."
Ford at first refused to apologize but later appeared with his wife and lawyer to make an apology for his "graphic remarks."
"For the past six months, I have been under tremendous, tremendous stress," he said, reading from a statement.
"The stress is largely of my own making. I have apologized and I have tried to move forward. This has proven to be almost impossible," he said.
"The revelations yesterday of cocaine, escorts and prostitution has pushed me over the line and I used unforgivable language and again I apologize," he said.
"These allegations are 100 percent lies. When you attack my integrity as a father and as a husband, I see red," he said.
Celebrity Couples of 2014 [PHOTOS]