Halladay, 36, signed a one-day contract with the Toronto Blue Jays to as to retire as a member of the team be broke into the majors with.
"It's so much fun to play the game and to go out and to compete," Halladay said. "I looked forward to that fifth day more than anything.
"To go out there and know that it's probably not going to feel very good and I'm not going to be able to do things the way I want to was very frustrating -- not only personally but I thought there was a certain responsibility to my teammates, to the organization.
"That part was very challenging for me to feel like you're giving it everything you can but there's something holding you back. That was a major factor in deciding this was the right time [to retire]."
Halladay missed a substantial part of last season due to shoulder surgery and was 4-5 with a 6.82 earned run average in 13 starts for Philadelphia.
He played 16 seasons -- 12 with the Blue Jays and four with the Phillies -- going 203-105 with a 3.38 ERA. He struck out 2,117 and walked 592 in his career. Halladay led the American League with 22 wins in 2003 and won the Cy Young Award and took the National League version of the honor in 2010 when he won a league-best 21 games.
He was selected to the All-Star Game eight times. His career highlights include a perfect game May 29, 2010, against the Florida Marlins and a playoff no-hitter on Oct. 6, 2010, against Cincinnati.
Celebrity Families of 2014 [PHOTOS]