Marine Scotland said the clams and worm were discovered at a suspected cold seep, a spot where hydrocarbons are released from the seabed, near Rockall, a rocky, volcanic outcropping about 260 miles west of the Western Isles, the BBC reported.
The British network said some types of commercial fishing could be banned in the area to protect the newly discovered creatures' habitat.
The new species of sea snail has been named Volutopsius scotiae, two types of clams were named Thyasira scotiae and Isorropodon mackayi, and the new-found marine worm is called Antonbrunnia.
Environment Secretary Richard Lochhead said the discoveries were surprising.
"Our oceans are often called Earth's final frontier and these new species prove just how much we still have to learn about this rich marine habitat," he said. "The area where these species were found is not currently fished and the confirmation of a cold seep is likely to result in the region being closed to bottom contact fishing."
Jim Drewery of Marine Scotland Science called the discoveries "absolutely incredible."