"A lot of people were quite upset by that," Stevens told Today hosts Willie Geist and Savannah Guthrie, who said his unexpected death left her reeling.
"I spend a lot of my time apologizing," he said. "Yeah, I get a lot of, 'My wife was very upset by this. You have to apologize to my wife!' Then they call them over. 'Katherine! Come over here. This is the young man that upset you.' 'Say sorry!' You have to apologize, in public."
Stevens, who said similar exchanges happen "almost every day," said it was "difficult" but "exciting" to make the decision to leave the show.
"It's a testament to how much people love the show, how much people love the character. In a way it was nice, but it has upset a lot of people, so I'm sorry," he added.
Meanwhile, Downton's executive producer Gareth Neame has shut down reports that the popular period drama will end after its fifth season.
"That was totally invented by the British press, totally made up. Completely untrue," Neame told Zap2It Thursday.
Stevens has moved onto other projects. His film Summer in February debuts in select theaters January 17.
Celebrity Breakups and divorces of 2014 [PHOTOS]