He became the king of late night but the many highs of his career never seemed to offset the lows, and the failure to succeed Johnny Carson on the Tonight show

David Letterman, who takes his final bow on Wednesday as a legend of the American invention known as late-night comedy, has been a constant, virtually inescapable presence for a whole generation of television viewers.

Starting as the puckish, peckish, insistently amusing kid brother; he became the commanding comedy perfectionist, before evolving into a cranky curmudgeon over the past decade or so: an on-camera version of the irascible neighbor spraying the garden hose to keep the kids off his lawn. Letterman has lived out a career – and life – in an electronic display window, one that faced directly into American bedrooms.

A man of complex temperament, daringly forthright on camera, insecure and reclusive off, Letterman has been a show business original since his first day as talkshow host – on an NBC daytime program in 1980 – the leader of a new generation of American comics, intent on parodying the entertainment world even as they lived in it, and off it.

For David Letterman, success was sweeping and elusive at the same time; fame brought him riches beyond his wildest imaginings – more than $30m a year at his peak – but never full satisfaction. He spent much of his career disappointed in the performances that bathed him in adoration from fans (and most critics), and unable to stop picking at the scar left by the dream job that got away.

Through it all Letterman experienced a rolling tide of low and high points, all of them chronicled and commented on by a public fascinated by his talent, his often stormy personal life, and his compelling, enigmatic, but undeniably authentic personality.

Facebook Twitter Pinterest Drew Barrymore turns to the audience after flashing host David Letterman during an April 1995 taping. Photograph: CBS Photo Archive/Getty Images

Any selection of low points in David Letterman’s career begins with the crashing disappointment of losing out on the goal that pushed him into television in the first place: succeeding the legendary Johnny Carson as host of NBC’s Tonight show. Letterman, who idolized Carson from childhood and ignited his career in a standup performance on Tonight in 1978 (he killed Carson with a joke about avoiding a brand of dog food that might solve his pet’s constipation, because “why screw up a good thing?”), set himself up as the perfect successor (he thought) by creating the nightly show that followed Carson starting in 1982, Late Night with David Letterman.

So complete was the expectation that Letterman would be Carson’s heir, Johnny himself routinely joked about it. The only ones not signed on to the plan were NBC executives, who saw a potential alternative in another young comic, Jay Leno, once Dave’s favorite guest on Late Night. Leno, as ambitious as Letterman but more willing to manifest it, managed a coup of sorts (in both senses of the word) by winning the favor of NBC when Carson retired in 1992.

Letterman, who had feuded with his network masters from the beginning (he called them “pinheads” and “worms”), was seen as too uncontrollable to be NBC’s signature star. Losing out to Leno, Letterman was devastated, to the point of believing his career was over. He was quickly pursued by every other television company, a sign of his unchallengeable stardom. Letterman eventually selected CBS, but almost reneged and stayed at NBC for a last chance at Tonight. The snub had lasting fallout on Letterman’s psyche, stinging him unrelentingly, even as he racked up ratings and awards in the CBS version of his show.

A most unexpected low point came a year into his CBS run, when he was at the top of his fame. Letterman was invited to host the Oscars in 1995, a role that Carson had been closely identified with. He accepted, but in typical fashion, was unwilling to kowtow to the usual conventions of the assignment – mainly, paying homage to Hollywood even as you skewer it. Instead, Letterman performed a larger version of his own show, including one signature bit, Stupid Pet Tricks, which had no particular relevance to that year in the movies. Letterman earned some of the harshest reviews of his career for the performance, and he demonstrated how lasting the effects of the lashing were by referring for years how he had “bombed at the Oscars”.

Facebook Twitter Pinterest David Letterman and guest Bill Murray in 1982. Photograph: Nancy Kaye/AP

One of the great phobias of Letterman’s life was anything that resulted in his feeling humiliated. As much as he sought to control everything that took place during the hour when he performed his show every night, he could not escape occasionally stepping on landmines that exacerbated his ingrained tendency toward self-loathing.

After Sarah Palin became a public figure during the 2008 presidential race, Letterman and most other late-night comics found her an inviting target for derision. A year later Letterman inadvertently crossed a line by joking, crudely, about one of Palin’s daughters. He said that Palin had been in the stands for a New York Yankees game and there had been an awkward moment in the seventh inning when her daughter was “knocked up” by Alex Rodriguez, the controversial Yankee slugger. Letterman intended the joke to refer to Bristol Palin, the daughter who had had a child out of wedlock. Unfortunately, Palin was at the game that night with Willow Palin, only 14 at the time.

After a pasting from Palin, who called him “sexually perverted”, Letterman was compelled to apologize, conceding the jokes were “ugly” (he had also said Sarah Palin looked like a “slutty flight attendant”), designed to elicit “cheap laughs”.

That was a deeply mortifying moment for a performer who detested mortification above all things. But it was eclipsed – spectacularly – by one of the most arresting on-air confessions in American television history. In 2009, after being threatened with blackmail by a lovelorn news producer at his own network, Letterman came on the air, sat at his desk, and wove a tale that included an appearance before a grand jury in New York and the revelation of an extortion letter asking him for $2m, all based on his dalliance with a young woman who had been his personal assistant, and who had become romantically involved with the producer. The episode took place only months after Letterman had married his long-time companion, the mother of his then five-year-old son.

Letterman laid it all out, referring to multiple affairs he had had with staff members, and acknowledging that he had done “terrible things”. He was excoriated in the press for taking part in what looked like a textbook example of offensive sexual office politics – the powerful male boss taking advantage of a female underling. He later revealed that he had expected to be fired over the incident. But Letterman managed to ride the scandal out, winning some unexpected respect for his unflinching honesty and for standing up to the blackmailer, who was arrested and sent to prison.

If the lows frequently brought the comic down to despair, the highs of his career, while much more frequent, seldom offered compensating bouts of elation – or even solace. Sometimes they only illustrated why he had reason to be unhappy. That was certainly the case in one of the most memorable examples of what became a Letterman trademark: found humor. That was the term the show’s staff used for concepts that had no scripted jokes, calling only for Letterman to be funny in the context of some real situation.

In 1986, the General Electric company took over the NBC network and Letterman, who enjoyed poking corporate suits mercilessly, decided to visit the GE headquarters bearing a welcoming basket of fruit. There was not much more than that to the idea, but the icy reception he received (“to deliver a basket of fruit you need paperwork?”) and especially a hostile encounter with a put-upon security chief (who would forever be famous for offering the “GE handshake” which was no shake at all), provided Letterman with precisely the fodder he needed for an excoriating and hilarious bit of rebellion against authority.

Facebook Twitter Pinterest President Barack Obama is pictured with host David Letterman in 2009. Photograph: Charles Dharapak/AP

Of course, it also underscored Letterman’s bad-boy reputation among the very people he needed to advance his career.

In his early years as a talk host, Letterman was barely interested in the talk. He looked to find moments where he could score with comedy. Later he became far more proficient at the art of interviewing and even became an essential destination for political figures out to impress Dave’s mostly intelligent and well-informed audience. Letterman got jabs in, but he also sought real information, and demonstrated clearly when a candidate impressed him.

One who consistently impressed him, both by being a good sport in the comedy and by having a résumé of bravery that spoke to Letterman’s admiration for real heroes, not the phonies of show business, was Senator John McCain. McCain had been a frequent and favorite guest before he was nominated to run for president in 2008. That year he of course made time to schedule a visit to Letterman’s couch. But the day of his appearance news of the American economy facing imminent collapse broke just hours before the taping. McCain called Dave personally to beg off, saying he was suspending his campaign and rushing to Washington because “the economy is cratering”.

Letterman opened his show that night with a detailed explanation for McCain’s last-minute cancelation (including effusive praise of the candidate) though he added that “something doesn’t smell right” about the decision. He continued, hilariously, in that vein, for almost half the show, and was able to pay off the bit when the internal feed of that night’s CBS evening news became available in his studio – and there was John McCain, not rushing to the airport, but about to be interviewed by the CBS anchor, Katie Couric.

That was plenty enough for Letterman to let loose his fire-breathing dragon of wit. He pummelled McCain to devastating effect (“Hey John, I got a question: you need a ride to the airport?”). The show was one of Letterman’s most sustained assaults of comedy artillery, and so wounded McCain that he felt compelled to insert a late trip to New York (a city in a state he had no hope of carrying) to seek Dave’s forgiveness.

That might have seemed preposterous in an earlier period of Letterman’s career, when he was seen as the mostly goofy purveyor of comedy bits like elevator races in the NBC headquarters building and strapping a camera to the back of a monkey for live shots. But by then Letterman had won his bona fides of legitimacy, with the world of politics and the wider world of public opinion. Nothing he did at any time in his career had more impact than his commentary when he brought his show back on the air in the aftermath of 9/11.

It was eight days later. No one in the world of comedy had dared venture out on to the stage, because none had a clue what to say. And all of them knew they weren’t the right choice for that impossible task anyway. They were all waiting for Dave. It had to be Letterman, not only because he was the longest-running host, he was also the one with the reputation for truth-telling, come what may. And he represented New York.

Letterman came out and sat behind his desk, with the model of the New York skyline behind him, and he spoke, extemporaneously, for more than seven minutes, of the terrible sadness, the bravery of the police and fire department, and the need for courage. He also told the nation that if they hadn’t known it before, they surely did now. “New York is the greatest city in the world.”

Nothing Letterman did, no words he ever said, would diminish the power of that moment, or eclipse the impact he had on the national consciousness. It was not a moment for an extraordinary comedian, but an extraordinary communicator. Letterman had always been distinguished by his skill not as a comic as much as a broadcaster, and he sealed his reputation that night.

As Letterman wraps up his career and his 22-year run on CBS, he has sealed another part of his legacy, one that seemed at first to be the lowest point of his professional life. Yes, he lost out on the dream he held in his heart since his earliest days thinking of a comedy career in Indianapolis, getting to sit behind the desk on the show that created the idea of late-night television, the franchise, the Tonight show. Nothing he achieved ever seemed to fully erase the disappointment of that loss.

But in moving on, in taking his prodigious talent to CBS, Letterman accomplished something surely more significant that merely being Johnny Carson’s successor. He became the first entertainer to stand up to the juggernaut of the Tonight show and survive – not only survive but thrive. He created, in his CBS show, the beachhead, the second front, a late-night franchise that will survive his own storied performance. In an achievement that may only be fully appreciated when he is long off the scene, David Letterman is leaving; but the show, his show, will go on.