Hakimullah Mehsud, the group leader, removed Maulvi Faqir Muhammad as the No. 2 man during a council of the Taliban leaders, the BBC reported.
The official name of the group is Tehreek-e-Taliban Pakistan or TTP, which is different from the Taliban in Afghanistan.
Muhammad had been the Taliban leader in the northern Bajaur district tribal area, suspected of being home to al-Qaida leaders, the BBC said.
A TTP spokesman told the BBC Urdu language service the leadership was considering giving Muhammad another position but gave no details.
The New York Times reported tensions between Muhammad and Mehsud surfaced after Muhammad was accused of having peace talks with the Pakistani government.
"He was removed due to his involvement in talks with the government without the consent of our leadership," Taliban spokesman Ehsanullah Ehsan was quoted as saying.
Muhammad had sought the group's leadership when Mehsud's predecessor, Baitullah Mehsud, was killed in August 2009, but had to settle for being the deputy as it was decided to retain the top position within the Waziristan tribal area, the BBC reported.
The Times reported the removal of Muhammad was not well received among his supporters in Bajaur.
The report said the development might allow the Pakistani government to further weaken the group, responsible for the deaths of thousands in terror attacks.
In a telephone interview with journalists, some of the Bajaur commanders, expressing disappointment over the decision, even talked of setting up a rival group.
The TTP has lately been weakened both by the Pakistani military operations and by the U.S. drone strikes.
"The TTP's peak has passed, it's on the downslide," Khalid Aziz, a former provincial chief secretary, told the Times. "Its people are coming under pressure; they are starting to go back to their tribes."