In an interview with ABC News, Madoff said while it is tough being ostracized from his family, he has settled into institutional life and is relieved at no longer having to hold his fraudulent investment scheme together.
"I feel safer here than outside," Madoff told correspondent Barbara Walters. "I have people to talk to and no decisions to make."
"I lived the last 20 years of my life in fear," he said. "Now I have no fear -- nothing to think about because I'm no longer in control of my own life."
Among Madoff's regrets, however, is the fallout from his downfall that shattered his family. Madoff said he had not spoken to his wife, Ruth Madoff, since their eldest son, Mark, committed suicide Dec. 11, on the second anniversary of Madoff's arrest in New York.
He said he had gotten past the despair that led to an unsuccessful suicide attempt by him and Ruth in 2008. He has been seeing a prison psychiatrist, but says the guards at the federal lock-up ask him daily if he is having suicidal thoughts.
As for the victims of his scam, Madoff was less despondent, ABC said.
"I understand why clients hate me," he said. "The gravy train is over. I can live with that.
"The average person thinks I robbed widows and orphans," he said. "I made wealthy people wealthier."
Notable deaths of 2014 [PHOTOS]