Letterman exited CBS' Late Show in 2015 after a 33-year run. He was replaced by Stephen Colbert.
"I stayed on television way too long," the comedian said. "It turns out nobody had the guts to fire me."
"You want to make sure you have some energy to direct towards other things," he continued. "All I cared about was myself and then the show was gone. And so I had to realize, oh, I've been looking through the wrong end of the telescope. There is more to life than, so tell me about your pet beaver."
Letterman now works with Habitat for Humanity and hosts Netflix's My Next Guest Needs No Introduction. He said that DeGeneres would be a guest on the show's upcoming second season.
Letterman also told DeGeneres a comedic story of how he thought he was going to jail after playing catch inside his office located on the 14th floor. A baseball he missed broke a window and fell towards a crowded New York street alongside broken glass.