"If I were president today, I wouldn't be looking to go spend 10 days on Martha's Vineyard," Romney told WLS-AM, Chicago's "Don Wade & Roma Show" as President Barack Obama prepared to leave Thursday for a 10-day vacation on the affluent summer colony island off Cape Cod. Obama planned to vacation at the 28-acre Blue Heron Farm in Chilmark, population 843, where he has spent part of the past two summers.
Chilmark is the island's least densely populated town, and the farm the Obamas are renting goes for $50,000 a week.
Chilmark has an average household median income of $41,917 but the highest average property value of any city or town in Massachusetts.
"Now, Martha's Vineyard is in my home state of Massachusetts, so I don't want to say anything negative about people vacationing there," said Romney, 65. "But if you're the president of the United States and the nation is in crisis -- and we're in a jobs crisis right now -- then you shouldn't be out vacationing. Instead, you should be focusing on getting the economy going again."
If he were president, "the first thing I'd do is go back to my office immediately," Romney said, adding he also would "pull back members of Congress [from their vacations] and focus on getting the job done."
"Over the course of his time in Martha's Vineyard, he will be getting updates from his economic team," White House deputy press secretary Josh Earnest told reporters at Otis Air Force Base on Cape Cod, adding Brain Deese, deputy director of the National Economic Council, will be in Martha's Vineyard next week "to provide regular updates and briefings for the president."
"In addition, we do anticipate that the president will be touching base with the economic team back home."
White House press secretary Jay Carney told reporters last week he didn't think Americans "would begrudge the notion that the president would spend some time with his family."
"There's no such thing as a presidential vacation -- the presidency travels with you," Carney said.
Obama "will be in constant communication and get regular briefings from his national security team as well as his economic team," Carney said. "And he will, of course, be fully capable if necessary of traveling back if that were required. It's not very far."
Romney is scheduled to be on the island for a fundraiser Aug. 27. But his visit isn't a vacation, Romney's office stressed. It's simply what Romney must do as a presidential hopeful, the office said.
Another Republican presidential hopeful, former House Speaker Newt Gingrich, said Obama "ought to cancel his vacation -- period."
Real estate magnate Donald Trump, who toyed with running for president, told Fox News Channel this Obama vacation couldn't have been timed worse.
"Here we have a country that really is going to hell in a hand basket," he said. "Let's not kid ourselves. What's happening to this country is horrible."
"Most Americans are very understanding of the need of a president to get away and to take a break and to have a vacation," Kenneth Walsh, who wrote "From Mount Vernon to Crawford: A History of the Presidents and Their Retreats," told National Public Radio's "All Things Considered."
"Where the problem comes for presidents is if they look like they're indulging themselves when the country is having hard times economically. And that's exactly where President Obama is now," he said.
CBS News reported Obama has taken 61 vacation days since taking office 31 months ago. At a similar point during his first term in office, former President George W. Bush had spent 180 days at his ranch in Texas, and during his first 31 months as president Ronald Reagan had taken 112 vacation days at his ranch in California.
The most dramatic example of a president escaping the White House was President John Adams, one of the nation's founding fathers and the second U.S. president, Walsh said.
Adams spent a record eight months at his home in Quincy, Mass., in 1799 while "his adversaries in Washington, in Congress, almost got us into a war with France," Walsh said.
Adams returned to Washington and resolved the overseas so-called Quasi-War, despite bitter opposition from Alexander Hamilton.
But news of the resolution did not arrive back to the United States in time to help Adams win a second term as president in 1800. He lost to Thomas Jefferson.
2014: The Year in Music [PHOTOS]