Monday marked Mullen's first appearance on the late-night chat show.
During the interview, Letterman held up the photo of the scene inside the White House situation room during the May 2 raid by U.S. Navy SEALS in Pakistan.
Among the people in the photo are Mullen and U.S. President Barack Obama.
"First of all, when they said, 'We want you to be in the so-and-so room at 9-whatever,' what did they tell you?," Letterman asked Mullen. "Did you know immediately what this meant? You probably knew before everybody, right?"
"Well, this is something we'd worked on for months," Mullen replied, "so what you're looking at is the culmination of an extraordinary effort on the part of our military, our intelligence agencies, and, quite frankly, is the output of what I consider to be a courageous decision on the part of the President."
"I was surprised to read, and maybe everybody knows this, that the likelihood of Osama bin Laden being at that location -- about 55 percent, is that about right?" Letterman asked.
"Well, actually, Dave, there were a lot of numbers," Mullen said, "and I think that math is something that probably is overplayed. There certainly was not certainty that he was there."
Notable deaths of 2014 [PHOTOS]