The 36-year-old English actor explained why he'd "rather not talk about" the 26-year-old American singer after reportedly getting testy about Swift in an interview with the Telegraph.
"What should I regret, in your mind?" Hiddleston responded when asked if he regrets "the publicity and gossip the romance engendered."
"Everyone is entitled to a private life," he explained. "I love what I do and I dedicate myself with absolute commitment to making great art and great entertainment, and in my mind I don't conflict the two."
"My work is in the public sphere and I have private life. And those two things are separate," the star asserted.
Hiddleston and Swift were first linked in June following the singer's split from DJ Calvin Harris, but broke up just three months later. The actor later admitted in the March issue of GQ that he was "surprised" the relationship "got so much attention."
"Taylor is an amazing woman. She's generous and kind and lovely, and we had the best time," he told the magazine. "Of course it was real."
"I have to be so psychologically strong about not letting other people's interpretations about my life affect my life. A relationship exists between two people. We will always know what it was," he added.
Hiddleston will next star in Kong: Skull Island, which co-stars Brie Larson and opens in theaters Friday. He is also slated to reprise Loki in Thor: Ragnarok.