The court said the killings of more than 20,000 Polish army officers, police officers and civilians by Soviet secret police had occurred before Russia ratified the European Convention on Human Rights, RIA Novosti reported.
The convention allows citizens to appeal to the court, based in Strasbourg, France.
The case had been brought by 15 Polish nationals who are relatives of 12 victims of the massacre in Russia's Katyn Forest.
The court said Russia had "taken most of the investigative steps" into the massacre before it ratified the convention in 1998.
But the court's judges said Russia didn't fully collaborate with the court and didn't notify the court of a 2004 decision to discontinue the investigation.
Russian authorities also failed to provide 10 relatives of massacre victims with "any official information about the circumstances surrounding the deaths, nor made any serious attempts to locate the burial sites of the relatives."
Russia must pay the citizens who complained jointly about $6,570 in compensation, the court said.
The citizens lawyer, Ireneusz Kaminski, said he would appeal.
In the 1990s, Russia turned over documents to Poland that put the blame for the massacre on the Soviet Union, and the lower house of Russia's Parliament, the State Duma, last year approved a declaration recognizing the Katyn massacre as a crime committed by dictator Joseph Stalin's regime.