The company said Ballmer would stay on to "lead Microsoft through the next steps of its transformation to a devices and services company," until a successor could be found.
"There is never a perfect time for this type of transition, but now is the right time," Ballmer said in a statement, noting shifting direction was the company's No. 1 priority.
"We have embarked on a new strategy with a new organization and we have an amazing senior leadership team. My original thoughts on timing would have had my retirement happen in the middle of our company's transformation to a devices and services company. We need a CEO who will be here longer term for this new direction," Ballmer said.
Microsoft said its board of directors had assembled a special search committee to find a replacement for Ballmer. It would be chaired by the board's lead independent director John Thompson.
The committee also includes Chairman of the Board Bill Gates, Chairman of the Audit committee Chuck Noski and Chairman of the Compensation committee Steve Luczo, the firm said.
"The board is committed to the effective transformation of Microsoft to a successful devices and services company," Thompson said in a statement. "As this work continues, we are focused on selecting a new CEO to work with the company's senior leadership team to chart the company's course and execute on it in a highly competitive industry."
"As a member of the succession planning committee, I'll work closely with the other members of the board to identify a great new CEO," Gates said. "We're fortunate to have Steve in his role until the new CEO assumes these duties."
2014: The Year in Fashion [PHOTOS]
Starbucks testing a latte that tastes like beer