There was no immediate indication of how many of the more than 3,000 inmates on U.S. death rows would be affected.
California insists that none of the more than 600 inmates on its death row is retarded. But no real figures exist for the country as a whole, and the number could be quite high.
Many are now expected to file suit in federal court to block their executions based on their mental capacities.
The court split along strict ideological lines, with moderate conservative Justices Sandra Day O'Connor and Anthony Kennedy joining the liberals and providing the crucial swing votes.
Writing for the majority, Justice John Paul Stevens said the mentally retarded "who meet the law's requirements for criminal responsibility should be tried and punished when they commit crimes. Because of their disabilities in areas of reasoning, judgment and control of their impulses, however, they do not act with the level of moral culpability that characterizes the most serious adult criminal conduct."
Stevens said those "impairments can jeopardize the reliability and fairness of capital proceedings against mentally retarded defendants."
He noted that the Supreme Court upheld the execution of mentally retarded inmates 13 years ago in Penry vs. Lynaugh.
But since then, "the American public, legislators, scholars and judges have deliberated over the question whether the death penalty should ever be imposed on a mentally retarded criminal."
The subsequent consensus guides "our answer to the question presented by this case: whether such executions are 'cruel and unusual punishments' prohibited by the Eighth Amendment of the federal Constitution."
Stevens said the majority believes, along with 18 state legislatures that have acted recently to ban such executions, that "death is not a suitable punishment for a mentally retarded criminal."
Chief Justice William Rehnquist led the dissenters.
"The court (majority) pronounces the punishment cruel and unusual primarily because 18 states recently have passed laws limiting the death eligibility of certain defendants based on mental retardation alone," Rehnquist said, "despite the fact that the laws of 19 other states besides Virginia continue to leave the question of proper punishment to the individuated consideration of sentencing judges or juries familiar with the particular defendant and his or her crime."
Scalia said the court majority's attempt to show a consensus on the issue was "embarrassingly feeble."
The case which brought Thursday's landmark decision comes from Virginia.
Daryl Renard Atkins was convicted of robbing and shooting to death Eric Nesbitt, who had driven up to a convenience store in York County in August 1996. Atkins and a friend had been "drinking and smoking weed" and were looking for more spending money, according to court records.
Nesbitt was kidnapped, forced to use his bank card and then killed in a wooded area.
Atkins was convicted and sentenced to death. After his death sentence was reversed by the Virginia Supreme Court on a technicality, he was again sentenced to death by a state judge who found him a continuing threat to society.
In his petition to the Supreme Court, before last February's argument in the case, Atkins's court-appointed lawyers said his case was a good one to test the constitutionality of the execution of the mentally retarded:
"Pretrial intelligence testing of Daryl Atkins showed that he had ... (an) IQ of 59 .... The state's mental health expert testified that the testing and resultant IQ scores were correct."
(No. 00-8452, Atkins vs. Virginia)
Pregnant Mila Kunis wins 'Best Villain' at MTV Movie Awards
'Happy' fan videos make Pharrell cry [VIDEO]