Pastore said in a statement that when he first committed to the show, "I didn't realize just how physically demanding it would be for me," People magazine said Wednesday.
"Unable to put forth my best effort, I felt it appropriate to step aside and give someone else the opportunity," he said.
Pastore wished his professional partner Edyta Sliwinska and the other competitors good luck.
Executive Producer Conrad Green, noting the physical demands of the show, said "it pays to know your limits. We respect his decision."
Notable deaths of 2014 [PHOTOS]
Reports: Mila Kunis gives birth to baby girl