The actress said she was wed to actor Parker Stevenson and Swayze to dancer Lisa Niemi when they began an intense emotional -- but non-physical -- relationship while making the 1985 Civil War TV miniseries "North and South."
Alley told "ET" she and her co-star attempted to avoid "going down that road," but fell in love anyway.
"Both of us were married," Alley said, adding they did not "have an affair."
"I think what we did was worse," she said. "Because I think when you fall in love with someone when you're married, you jeopardize your own marriage and their marriage. It's doubly bad."
Swayze was still married to Niemi when he died of cancer in 2009.
Alley divorced Stevenson in 1997.
2014: The Year in Music [PHOTOS]