"I look at what's happening right now, I wish I were there," Romney told "Fox News Sunday" in a wide-ranging interview at his San Diego beach house, his first on television since he lost November's presidential election.
"It kills me not to be there, not to be in the White House doing what needs to be done," the former Massachusetts governor said.
"I have to tell you, the hardest thing about losing is watching this critical moment, this golden moment, just slip away with politics," he said.
He accused President Barack Obama of using automatic federal spending cuts known as the sequester to score political points.
"No one can think that that's been a success for the president," Romney said. "He didn't think the sequester would happen. It is happening. To date, what we've seen is the president out campaigning to the American people, doing rallies around the country, flying around the country and berating Republicans and blaming and pointing."
Ann Romney, who sat alongside her husband for part of the interview, said, "I totally believe, at this moment, if Mitt were there in the office, that we would not be facing sequestration right now."
Romney said his campaign's failure to connect with minority voters doomed his White House bid.
"We weren't effective in taking my message primarily to minority voters -- to Hispanic-Americans, African-Americans, other minorities," he said. "That was a real weakness.
"We did very well with the majority population but not with minority populations. And that was a failing. That was a real mistake," he said.
Romney collected 27 percent of the Latino vote, 6 percent of the African-American vote and 59 percent of the white vote, exit polls indicated. Obama garnered 71 percent of the Hispanic vote and 93 percent of the black vote.
Romney lost the overall popular vote to Obama 50.6 percent to 47.8 percent but lost the electoral vote 61.7 percent to 38.3 percent. Obama got 332 electoral votes to Romney's 206.
Romney said he underestimated the appeal of Obama's healthcare law to minority voters.
This is a statement he also made on a conference call the week after the election, when he blamed his overwhelming electoral loss on government "gifts," including healthcare reform, he said Obama gave to key groups, especially blacks, Latinos and young people.
Ann Romney blamed the media for not letting voters "really get to know Mitt for who he was."
When she was asked if it was true the campaign chose not to "let Mitt be Mitt" and show his more open, compassionate side, she said:
"Well, of course. It was partly -- it's true. But it was not just the campaign's fault. I believe it was the media's fault as well, in that he was not being given a fair shake, that people weren't allowed to really see him for who he was."
When asked, "What about the media?" she said, "I'm happy to blame the media."
The Washington Post pointed out the Romney campaign's strategy was originally to keep warmer aspects of his life, such as his numerous charitable deeds when he volunteered as a church bishop, out of public view. It was only in the campaign's final weeks that it widely shared this side of him, the Post said.
Ann Romney said she shed tears after the loss.
"Yes, I cried," she said. "When you pour that much of your life and energy and passion into something and you're disappointed by the outcome, it's very -- it's sad. It's very hard."
She said her sorrow didn't stem from the failure to attain a personal goal but rather from a lost opportunity to serve the country.
"The dream was to make a difference. The dream was to serve," she said.
Up until Election Day, she said, she and her husband thought they would win.
"We were a little blindsided," she said.
"You know the great 'Princess Bride' line, 'Mostly dead'?" Ann Romney said. "I'm mostly over it. But not completely. And you have moments where you, you know, go back and feel the sorrow of the loss. And so, yes, I think we're not mostly dead yet."
The "mostly dead" line comes from a magician named Miracle Max in William Goldman's 1973 fantasy romance novel. Max pronounces a farmhand named Westley "mostly dead" and returns him to life, though Westley remains partially paralyzed and weak.
Ann Romney said she knows her husband "would have been a fabulous president, and I mourn the fact that he's not there."
Mitt Romney -- who is scheduled to give his first speech since the election at the Conservative Political Action Conference in Washington March 15 -- said he planned to remain a political voice.
"I'm not going to disappear," he said. "I'm not running for office. I don't have a big organization that's out speaking in my behalf.
"But I care about America. I care about the people that can't find jobs," he said.
"I care about my 20 grandkids, the kind of America they're going to have. And sitting on the sidelines when so much is at stake is just not in my nature," he said.