The decision came amid news reports his son, Tyrel, 22, had misused the governor's mansion for late-night parties without his parents present.
The former professional wrestler known as "The Body" was elected in 1998 as an independent. He has had a stormy relationship with both the press and the Legislature.
"I am not seeking re-election again," Ventura said in an appearance on Minnesota Public Radio. "I will not run again."
Ventura criticized media reports about his son's use of the official residence, saying, "My son is a terrific kid and he's being libeled. He's being slandered." The governor noted that he behaved "far worse at his age," adding the lives of members of family should not be subject to public sniping.
"I will always protect my family first," Ventura said. "You've got to have your heart and soul into these types of jobs. You've got to want to do it. I feel that it's time to go back to the private sector."
Ventura, who closed the governor's mansion this spring in a budget dispute, never really left the private sector. During his tenure as governor, he promoted two books on his life as well as an action figure, refereed a World Wrestling Association match and did a stint as a color commentator for the defunct Xtreme Football League. He justified the actions by saying he is not a career politician and did not come from wealth.
Ventura said his decision was not related to the strength of his opponents, Democrat Roger Moe and Republican Tim Pawlenty.
"I had a much tougher field in 1998. I had to defeat the son of the great Hubert Humphrey. I had to defeat a very popular St. Paul mayor," he said.
Ventura said among his regrets in leaving office after only one term is his failure to muster support for a referendum on reducing the two-house Legislature to a unicameral.