The National Hurricane Center in Miami said in its 11 p.m. advisory that what was left of Irene was centered about 50 miles north of Berlin, N.H., and about 105 miles south of Quebec City, Quebec. The system was still producing sustained winds of 50 mph as it moved to the north-northeast at 26 mph.
Coastal storm surges and heavy rains were to ease in New England through Sunday night, forecasters said.
The hurricane center said the tropical storm warning for the U.S. East Coast has been discontinued but was in effect for the south coast of New Brunswick from the U.S.-Canadian border to Fort Lawrence and for the south coast of Nova Scotia from Fort Lawrence to Porters Lake.
The warnings for those Canadian provinces will likely end early Monday.
At least 19 people died in hurricane-associated events along the U.S. East Coast, CNN said.
U.S. President Barack Obama warned Sunday that while the remnants of Irene moved into Canada, flooding was expected to worsen in the coming days.
CNN said more than 4 million people were without electricity and the federal government projected the wind damage from Irene was expected to top $1 billion.
"It's crazy. ... The water is moving between buildings, up, down, all sorts of different directions," Rich Graessle in Milburn, N.J., told CNN.
Philadelphia Mayor Michael Nutter said storm water was sending "couches, furniture, all kinds of stuff floating down the street." Two buildings collapsed in Philadelphia, Nutter said. No injuries were reported.
While the aftermath of Irene was expected to impact the Monday commute on the East Coast, Amtrak announced most of its Northeast Regional trains between Philadelphia and Washington would be running after being canceled during the weekend due to flooding, debris on tracks and power issues. Many other Amtrak routes also were expected to be back in operation.
The hurricane was downgraded to tropical storm status after making a second landfall early Sunday in New Jersey. It first came ashore in North Carolina Saturday.
Speaking with Homeland Security Secretary Janet Napolitano and Federal Emergency Management Agency Administrator Craig Fugate at his side in the White House Rose Garden, Obama said Irene "remains a dangerous storm that continues to produce heavy rains."
"This is a storm that has claimed lives," Obama said, warning flooding near rivers is expected to worsen and the recovery effort "will last for weeks."
"This is not over," he said.
Weather Channel meteorologist Jim Cantore said New England should brace for more damage from the "prolonged wind event" and rivers are sure to reach "record crests."
A storm surge of 4 to 8 feet was expected to threaten coastal areas of New York's Long Island as well as parts of Connecticut, Rhode Island and Massachusetts, the center said.
"Near the coast, the surge will be accompanied by large, destructive and life-threatening waves," the center said. "Higher than normal astronomical tides are occurring this weekend."
New Jersey Gov. Chris Christie, speaking on NBC's "Meet the Press," warned residents of dangers that persist even after the storm has moved on.
"We have downed power wires all over the place. We have flooding of streets," Christie said. "Everyone will be fine if we stay in our homes, we let the storm pass, and then we wait to hear an assessment for people when they can go back out."
"Let the local officials give you that all clear," Fugate said on the program. "And if we want the power back on quick, stay off the roads, let the power crews get out there and get to work. They don't need you out there sight-seeing. Stay home unless it's urgent, and don't go back until the local officials tell you it's OK."
Gov. Bob McDonnell said Irene caused the second-largest power outage in Virginia history, leaving 2 million people without power.
The National Weather Service was investigating numerous reports of tornadoes in North Carolina, where Irene made landfall Saturday morning before moving back over the Atlantic Ocean.
New York City's entire transit system was shut down as a precautionary measure Saturday for the first time ever, and streets were virtually deserted as the winds and rains began, The New York Times reported.
Nearly 400,000 people in New York followed Mayor Michael Bloomberg's evacuation order and warning of flooding and power outages, the Times said.
Regardless, the Times said emergency personnel were called to evacuate "several dozens" of people whose homes had flooded in Staten Island.
In Boston, officials announced Saturday night the entire transit system was shutting down as a precautionary measure, The Boston Globe said.
All three major airports in the New York metropolitan area were shut down Sunday, causing a cascade of canceled flights that stranded travelers all across the country, KLAS-TV, Las Vegas, reported.
Notable deaths of 2014 [PHOTOS]
Man wanted for abduction of missing UVA student