Like an increasing number of baby-boomers, Paul Shaffer will be 65 and out of a job next year. Actually, it’s much more than a mere job; it’s his life-long vocation (granted, really half his life), the beating heart of his self-identity, and his dependable sanctuary of fellowship and fun—the sudden absence of which might be compared to the death of a treasured friend.

“Well, how can you grieve when you’ve had such a long run?” Shaffer asks me over dinner at Remi, one of his favorite haunts a block from The Ed Sullivan Theater, where he has led the CBS Orchestra for the Late Show With David Letterman for the past two decades.

Before that, he spent a decade on Letterman’s 12:30 a.m. NBC show, conducting “The World’s Most Dangerous Band” from his rock’n’roll keyboard. “It’ll be 33 years by the time we’re done. It’s been fantastic. It’s been absolutely wonderful. Anybody who would be complaining about that should be put away.”

Perhaps Shaffer isn’t in denial—the first of Elisabeth Kubler-Ross’s “Five Stages of Grief”—yet members of Letterman’s staff were openly crying in April when Dave, now 66, announced on the air that next season will be his last. “You have some weepers back there,” that night’s guest, Johnny Depp, told the late-night comic, while Shaffer quipped: “Do I have a minute to call my accountant?”

Chewing on a breadstick at Remi, having just taped Monday’s installment featuring actor Michael Cera and frisky dogs broad-jumping into a 20,000-gallon pool of water out on West 53rd Street, Shaffer says: “Of course we’re—or at least I am—enjoying every show much more now, knowing that there are a finite number left.”

These days naked-pated and a teensy bit stouter than when he started out as Dave’s musical director and comic sidekick (a tonsorial and corporeal evolution that is unsparingly documented on YouTub e ), Shaffer talks in the smooth, reedy, dulcet tones that are known to millions of viewers—a less cartoonish version of the voice he uses to send up slick showbiz insincerity.

Who can forget Shaffer’s scene-stealing cameo in the classic 1984 mockumentary This Is Spinal Tap—as Polymer records executive Artie Fufkin? His voice is occasionally punctuated by that famous laugh—Shaffer’s trademark rasping honk—and that endearing, mole-like squint.

“We were young when we started,” Shaffer says, acknowledging that the last days—before Comedy Central’s Stephen Colbert takes over the time slot in September 2015—will be poignant and possibly emotionally draining. “Yes, absolutely. It’s been so long that I can’t remember a time when I didn’t do this show,” he says. “There are upsides to it [the final curtain], of course. There’s the freedom to do lots of other things—to travel somewhere. I want to do all kinds of things. I want to keep doing music primarily, of course. I’d like to act, too. But it has really has been our whole life for all of us on the show, day in and day out.”

Years ago, however, Shaffer had believed that he might not be long for the job after getting visibly angry at his boss on the air—a remarkable departure from the bandleader’s normally genial if ironic onstage persona.

“I really lost my temper,” he recalls, recounting how he vented at Letterman when frequent musical guest Todd Rundgren showed up at the studio too late to rehearse, and Dave (perhaps needling Paul) kept announcing that the band would play a string of completely unprepared Rundgren hits.

“Well, I couldn’t play all his songs because we hadn’t had a chance to rehearse, but Dave kept coming back to it,” Shaffer recalls. “He just kept firing it in, and I lost it and I started yelling at him on the air. I said, ‘Listen, anything you want, I give you! You want a song by the Gin Blossoms, you got it!’ Then I felt terrible afterwards. What have I done? The man pays my salary.”

After the show, Shaffer phoned Letterman in his office to apologize, “and he was laughing. He said, ‘That was hilarious. Feel free to do as much as you want whenever you want to do it. You can come over and sit on my head if you want.’ ” Shaffer adds: “I thought he was pretty damn nice, because I thought I was gonna get fired.”

Not a chance. The Canadian-born Shaffer—who grew up in Thunder Bay, Ontario, where he started watching Johnny Carson in the early 1960s via the NBC affiliate in Duluth, Minn.—sees his role with Dave as an updated composite of Carson’s bandleader Doc Severinson and sidekick Ed McMahon. “Doc and Ed—D’ed,” Shaffer says.

“It can take a certain amount of adjustment for a musician, whether jazz or rock, to get his head around the job of providing cues for comedic situations,” says Shaffer, who parsimoniously punctuates Letterman’s monologue jokes and celebrity interviews with the odd comic chord, or otherwise chimes in with a comment, question, or a simple “uh-huh” or “oh yeah,” as needed. “Sometimes, rather than speak to the camera, Dave will turn and speak to me,” Shaffer says, “and I got to realize he needed sort of a bed of sound.”

In one memorable moment, prodded by Letterman, Shaffer asked Julia Roberts, who had recently broken up with a boyfriend, “You getting laid these days?” Hilarity ensued.

“Not everybody wants to do it”—that is, cue up the laughs—“or understands that it could possibly be important to do. But it was always my favorite thing to do,” Shaffer says, adding that “less is more.” Regarding a rival talk show, no longer on the air, he says: “I remember the Leno band and what their punctuations would sound like—a certain disjointed timing sometimes.”

The fierce and often bitter competition with Jay Leno, who took NBC’s Tonight Show, the prize Dave was denied, and came from behind after Dave’s 1993 CBS debut to consistently beat him, had the all trappings of a late-night cold war.

“Yes, of course,” Shaffer says. “What can be said? Dave was the best that ever did it. He would say Johnny Carson was, but I’ve worked for Dave every night and he’s just the smartest and most on-top-of-it, the quickest. It’s the most spontaneous show on television. Maybe Jay was more commercial, or more universal. I don’t know.”

Shaffer acknowledges that with the march of time—through Letterman’s quintuple bypass surgery and a harrowing blackmail incident (during which the host admitted on the air that he’d “horribly hurt” his wife by sleeping with female members of the staff), the program’s anti-showbiz sensibility has softened considerably.

“I don’t think we can deny that we became the establishment,” Shaffer says. “We were making fun of the format—and then we became it. You have to kind of admit that. And I think Dave wants to be age-appropriate.”

But also real, unscripted, and sometimes painfully honest, as when Dave owned up to his extramarital misbehavior. “The show, to him, has always been something like a forum,” Shaffer says. “His attitude, I think, is if we can talk about it, how bad can it be? When he does that kind of thing, I think it becomes more than just a show. It is the real reality show. I think the Kardashians may set up a few of these situations.”

Shaffer, who was a something of a musical prodigy, studied classical piano and played in a high school rock band, and then a progressive jazz band at the University of Toronto, where he also DJ’ed on the campus radio station. (His attendance at John Lennon and Yoko Ono’s December 1969 press conference touting a peace festival is memorialized in a CBC documentary at around the 34-minute mark. Shaffer, who recalls that he was too intimated to ask a question, is the skinny kid with shoulder-length hair and a Fu Manchu mustache.)

He was named musical director of a full-dress production of Godspell at the tender age of 22, when he became fast friends with cast members (and future stars) Gilda Radner, Martin Short, Andrea Martin, Dave Thomas and Victor Garber. By the mid-1970s he was a member of the house band and an occasional performer on Saturday Night Live, where he was the piano player for Bill Murray’s oleaginous lounge-singer act.

He left the show in 1977 to star in a short-lived sitcom flop, in which one of his fellow cast members was Mickey Rooney, and returned to SNL after the cancellation.

Shaffer says he hit it off immediately with Letterman in 1982 when he was summoned to a meeting at 30 Rock, where Dave was looking for a bandleader for his soon-to-launch program. Their musical tastes were in synch, and Letterman was deeply knowledgeable, Shaffer says.

When he told Letterman that he wanted to play instrumentals of Motown hits and soul music, the former standup comic’s face lit up. “Well that sounds great,” he recalls Letterman saying. “I’ve always considered myself the Wayne Cochran of comedy anyway,” Letterman added, referring to the over-the-top soul singer-turned-Christian minister who favored extravagant outfits and a towering white pompadour.

Shaffer is among a happy crew of Letterman loyalists—including producers Maria Pope, Barbara Gaines and Jude Brennan, and bass guitarists Will Lee and Sid McGinnis, and drummer Anton Fig—who were present at the creation in the early 1980s and 1990s, when Dave launched and developed his groundbreaking, showbiz-satirizing Late Night program that aired after the more traditional Tonight Show Starring Johnny Carson.

When NBC declined to make Letterman Carson’s successor—as chronicled in a best-selling book and an HBO movie—he took much of his staff to CBS and there’s been astonishingly little turnover since the Late Show’s debut in 1993.

“He’s a very loyal employer—loyalty is big one for Dave,” Shaffer says. “I don’t think it’s any secret that he’s not comfortable around a lot of people. He’s not a social guy. So his core team are people among whom he does feel comfortable, and he will socialize with them for that reason.”

Indeed, Shaffer is not simply an employee, he’s a close-enough friend to have been invited repeatedly with his family to Letterman’s Montana ranch, where he and his two kids—a 21-year-old daughter and a 15-year-old son—spent quality time with Dave, horseback-riding and such, during the recent two-week August hiatus. “My wife didn’t come because she had a broken foot,” Shaffer says, referring to former Good Morning America booker Cathy Vasapoli.

And now, the show, the life, the camaraderie, is slowly but surely slipping away. “We’ll have to have a reunion,” Shaffer says of the Letterman-less future. “We have to, and I think we will”—even if Shaffer must organize it. “I’ll do whatever I have to do.”