"(Les) and I ... we have had this conversation in the past, and we agreed that we would work together on this circumstance and the timing of this circumstance," Letterman said during a taping of Thursday night's show. "And I phoned him just before the program, and I said, 'Leslie, it's been great, you've been great, and the network has been great, but I'm retiring.'