Appearing on CBS' "Face the Nation," Brown said U.S. elections can frequently turn on an unexpected gaffe that will both irritate voters and provide ammunition for well-funded opposition groups.
"I think it turns on if one of the candidates screws up first and makes a mistake," said Brown, 74, who himself ran for president in the 1970s. "Elections tend to move on the other person making the mistake."
Brown also indicated hard-nosed analysis of economic policy would not be as great a factor.
"I think it comes down to who do the American people have the most confidence in?" he said.
Brown said neither President Obama nor likely Republican nominee Mitt Romney could cultivate an image of being the better "manager" of the government and expect to appeal to voters.
"I don't think presidents function as managers," Brown. "We don't think of Churchill or de Gaulle or Roosevelt as managers of the government, but rather as strong individuals who stand clearly for a particular vision about their countries.
"And in those terms," Brown added. "We've seen over the last few years, drift, confusion, and a good deal of lack of confidence."