Search and rescue officials in Nova Scotia said rough seas and the cold water made it unlikely the men, all in the 20s, had survived after their 44-foot boat, the Miss Ally, capsized Sunday night, the Canadian Broadcasting Corp. reported.
The incident will now be handled by the Royal Canadian Mounted Police as a missing persons case.
George Hopkins, the father of one of the missing Woods Harbour fishermen, said his son, Joel, and the others were "the heart of the community."
"All them boys, we love them," Hopkins said.
"We live on the water. We don't leave the wharf with a fear in our heart. There's no fear there. It's like a miner going down to the mine. It's our lifestyle."
The names of the other victims hadn't been released.