New Mexico and Michigan inaugurated new governors on New Year's Day, while 20 other states will do so in coming weeks.
"We must take immediate, hard action" to address a looming $2 billion budget deficit, Romney said in his inaugural address in the Massachusetts Statehouse.
"I'll not hesitate" in facing it, he said, promising "hard, fast action" to meet the crisis "head on."
He said it will be "a very busy New Year."
Romney, 55, who was elected in the Democratic-dominated state with the support of the state's growing number of independents, promised to reshape state government rather than raise taxes to deal with the deficit.
"Continuing to raise taxes to feed a bureaucratic beast can only result in devouring the means and motivation of our people," Romney said.
"We must invest in our people by building our economy, by repairing broken schools, by building more housing, and by vigorously safeguarding air and water," he said. "And we must make sure the fiscal crisis we face today with all its attendant cuts and pain is never visited upon us again.
"In this new century, perhaps as much as any other time in our history, the qualities of our free people emerge as our greatest strength, and because I believe that this is so, I believe that we best strengthen our state and our nation by investing in our people and in protecting their freedoms."
The millionaire Republican venture capitalist parlayed his success as head of the organizing committee for the 2002 Winter Olympic Games in Salt Lake City into his first election victory as governor of Massachusetts.
He is the son of the late former Michigan Gov. George Romney, a one-time presidential candidate and American Motors chairman.
His national stock is likely to rise if he is successful at solving the fiscal crisis in Massachusetts.
Close to President George W. Bush, Romney is expected to play a key fundraising role in the 2004 presidential campaign, but has promised to serve his full four-year term as governor.
Romney is the fourth Republican governor in a row in Massachusetts. He took over for acting Gov. Jane Swift, who inherited the job when Paul Cellucci was named ambassador to Canada. Cellucci had become governor when William Weld made a failed attempt to be named ambassador to Mexico.
Sen. John Kerry, D-Mass., a probable Democratic presidential candidate in 2004, said Romney will be facing some "tough choices" in the coming weeks.
"We're looking for sensitivity and common sense," Kerry said.
Romney was the only governor to be sworn in Thursday, but two were inaugurated Wednesday and many more are scheduled in the coming weeks.
Newly sworn-in New Mexico Gov. Bill Richardson, a former Clinton administration official and representative, is promising to raise teacher salaries, cut government waste and end political feuding in Santa Fe.
Richardson was officially sworn in at midnight Wednesday at the historic Palace of Governors and then delivered his inaugural address at a larger gathering on New Year's Day at the House of Representatives in the Roundhouse, the Capitol.
"Do not judge me on my promises," the Democrat said. "Judge me on my results. Let us now roll up our sleeves and get to work. Our partnership for the long haul begins today."
He made several references to the "long haul," appearing to silence some who have speculated that Richardson has his eyes on national office.
Richardson said he would support a 6 percent raise for New Mexico teachers financed by reducing administrative costs in education. In contrast to other states, New Mexico is in good financial shape as the new governor prepares to deliver his first budget to the Legislature.
Richardson also proposed a new secretary of education, an office that his predecessor, Republican Gary Johnson, also supported.
In Michigan, Gov. Jennifer Granholm won praise even from Republicans as she took office as Michigan's 45th governor on New Year's Day.
In a 12-minute speech long on history and patriotism, Granholm mentioned both the Bible and President John F. Kennedy. She is the state's first Democratic governor in 12 years and the first female to reach the state's top office.
Queen of Soul Aretha Franklin sang the national anthem after the inaugural festivities moved from the makeshift stage in sub-freezing cold outside the state Capitol Building to inside the warm Lansing Center.
Former Republican Gov. John Engler warmly applauded Granholm's remarks but skipped the indoor event though he was invited.
On Jan. 11, Illinois Gov.-elect Rod Blagojevich plans to revive a tradition for his inaugural.
The Democratic former representative plans a 200-mile whistle-stop train tour from Chicago's Union Station to Springfield, where he will host an old-fashioned free inaugural barbecue Jan. 12.
Blagojevich will be Illinois' first Democratic governor in 26 years. The formal ball will be held Jan. 13.
The idea of the railroad tour is to "underscore the key themes of openness, accessibility and new approaches that are central to the inaugural weekend as a whole," said a statement released by his transition team Thursday.
(With reporting by Phil Magers in Dallas and Al Swanson in Chicago).
Celebrity Families of 2014 [PHOTOS]