"I feel it's the best thing for me," said Tomjanovich, who missed the final 17 games of the season to undergo treatment for bladder cancer. "I need to live a life with less stress."
Tomjanovich made the decision after meeting this week with owner Les Alexander, but will remain with the team in an unspecified front office role.
The Houston Chronicle reported Friday that the two sides negotiated a buyout of Tomjanovich's contract, which had two years and $12 million remaining.
Tomjanovich has been with the Rockets for more than three decades and coached the franchise to its only NBA championships in 1994 and 1995. But Houston has failed to reach the playoffs each of the last four seasons.
The 54-year-old Tomjanovich took a medical leave of absence in March and said he did not want to return if he was not 100 percent.
"The responsibility of people counting on you for happiness brings a lot of responsibility," Tomjanovich said. "At this stage, with the health situation, the best thing is to back out and be a regular guy for a while.
"I want to be soldier and not a general."
This season, the Rockets added rookie standout Yao Ming to a lineup that featured star guard Steve Francis. They went 43-39 and fell short of the postseason, but Tomjanovich insisted he was not abandoning a struggling team.
"Nothing is changed about how I feel about this team," he said. "There are so many positives. Whoever gets this job has a wonderful opportunity."
The Rockets did not name a successor. Assistant Larry Smith coached the team in Tomjanovich's absence.
Tomjanovich played 11 seasons with the Rockets and was a five-time All-Star. He is the third-leading scorer in franchise history with 13,383 points and his No. 45 jersey was retired in 1982.
"I am sure one day he will be in the Hall of Fame," General Manager Carroll Dawson said.
Tomjanovich is the winningest coach in team histoy with a 503-397 record. He also coached the United States to the gold medal at the 2000 Summer Olympics in Seoul.
Tomjanovich survived one of the scariest moments in NBA history in December 1997, when Los Angeles Lakers forward Kermit Washington blindsided him with a punch that resulted in serious facial injuries.