The findings, published in Journal of the National Cancer Institute, were based on data from the Selenium and Vitamin E Cancer Prevention Trial, a rigorously executed, randomized and placebo-controlled trial that involved more than 35,000 men.
First author Alan Kristal of the Fred Hutchinson Cancer Research Center said the study sought to determine whether taking high-dose vitamin E, 400 International Units/day and/or selenium 200 micrograms/day could protect men from prostate cancer.
The trial began in 2001 and was designed to last 12 years, but it stopped in 2008 because it found no protective effect from selenium and there was a suggestion that vitamin E increased risk, Kristal said.
While use of the study supplements stopped, men were still tracked and after an additional two years the men who took vitamin E had a statistically significant 17 percent increased risk of prostate cancer.
The researchers found taking selenium supplements increased the risk of high-grade cancer by 91 percent among men with high selenium status at baseline. When selenium supplements were taken by men who had high selenium status to begin with, the levels of selenium became toxic, the study said.
The study also found only a sub-group of men was at increased risk of prostate cancer from taking vitamin E. Among men with low selenium at baseline, vitamin E supplementation increased their total risk of prostate cancer by 63 percent and increased the risk of high-grade cancer by 111 percent, the study said.
This explained one of the original findings, which was that only men who received vitamin E plus a placebo pill, and not those who received both vitamin E and selenium, had an increased prostate cancer risk, Kristal said.
The study found selenium, whether from dietary sources or supplements, protected men from the harmful effects of vitamin E.
"Many people think dietary supplements are helpful or at the least innocuous. This is not true," Kristal said in a statement.
"We know from several other studies that some high-dose dietary supplements -- that is, supplements that provide far more than the daily recommended intakes of micronutrients -- increase cancer risk. We knew this based on randomized, controlled, double-blinded studies for folate and beta carotene, and now we know it for vitamin E and selenium. Men using these supplements should stop, period. Neither selenium nor vitamin E supplementation confers any known benefits -- only risks."