The 31-year-old actor spoke to Access Hollywood on Monday about his numerous legal and personal problems. Short was charged with spousal battery in February and aggravated assault in March, was let go from Scandal in April and was arrested for public intoxication over Fourth of July weekend.
The battery and assault charges have since been dropped, but the star admits the last three months have "been one of the most trying times" of his life.
"I think there's a time when your personal life gets magnified by who we are as celebrities," he says.
"I have not been perfect, but I have not been all the things that's been in the press. I will take ownership. Me and my wife are going through problems, serious problems; however, the domestic part of it? ... I've never put my hands on a woman, physically, ever."
Short also addressed his departure from Scandal, and suggested that ABC decided not to renew his contract, not creator Shonda Rhimes. The actor starred as senior associate Harrison Wright on two seasons of the political drama.
"They did not throw me to the curb like trash," he says of the cast and crew. "That's my family. They knew that I was going through a tough season and rallied around me and wrapped their arms around me, from Shonda to the craft service man. We're a family."
"I think at a certain point, when you're [hearing] a barrage of stories after stories, ABC justifiably would want to distance themselves," he continues. "They did the right thing."
Short wants fans to know he is "still here" and "focused" on his self-improvement.