The 35th annual Kennedy Center honoree spoke about his personal life on Thursday's edition of "CBS This Morning."
Letterman, 65, said he began taking medication about 20 years ago to treat his depression. He said it was a long road to finding the right prescription drug and how much of it worked for him.
The "Late Show" host said at one point he gave up taking his meds because he hated the side effects.
"And part of that created in me this nervous anxiety. And then I was really screwed. So that's when I said to [my doctor] Louis [Aronne:] 'OK, OK, I'll try anything just to get rid of this depression.' Because it was, it's different than, 'Oh, I don't feel good today.' It's different than feeling sad. It's different than feeling blue. It's really like a friend of mine says, 'It's the world with 20/20 vision.'"
Letterman said he waited until he was in his mid-50s to become a first-time father to son Harry because he didn't think he could handle his career and being a good dad.
"I just thought, when the topic would come up, 'I can't do both,'" he said. "I can't try to have a successful television show and be a father. And I was wrong about that -- because as difficult as being a father is, it's entirely complementary with everything else in your life. ... It's like you get your prescription updated. You can see things that you never saw before."
He went on to say he wishes he had "like, five or six kids."
Really? CBS interviewer Charlie Rose, 70, wanted to know.
"Well, no. I just say that because I think people would like to hear it," Letterman said. "I wish I had a little girl. ... I have a little boy now. I wish I had a little girl."
Told it's never too late, Letterman replied: "Look at me. You got a better shot than I do."
Letterman said fatherhood is a joy he never thought possible but also a source of anxiety for him.
"Makes you see everything different," he said. "But what I wasn't prepared for was the infinite anxiety that it triggers. You worry about everything. I don't care. Just throw out a topic, go through the alphabet, identify a word that begins with any letter that could be trouble, you worry about it."