John Franklin Riggs and his family were fishing in Tangier Sound when the storm hit at about 7 p.m. Tuesday, capsizing their boat, The (Salisbury, Md.) Daily Times reported.
Riggs said he decided to leave his father, sister, niece and nephew, who clung to the capsized boat, while he swam to shore.
He reached the home of Angela Byrd at about 1 a.m. Wednesday.
"He came to the right house," Byrd said. "He said, 'I've been swimming since sundown; I need help.'"
The Deal Island Fire Department and the Maryland State Police were called in to rescue those still in the water, WBOC-TV, Salisbury, Md., reported.
"It's an amazing story," said Deal Island Fire Chief Donald Ford. "The man's a hero. He swam from Haines Point and ended up, when he came to shore, at Hall Point."
"There were a few storms in the area, and the boat turned upside down," said Sgt. Brian Albert at the Maryland Natural Resources Police. "Mr. Riggs swam to shore. These people are very lucky. No one was injured."
Avril Lavigne denies racism in 'Hello Kitty' music video
Joan Rivers refuses to apologize for kidnapping joke