Editor's note: CNN contributor Bob Greene is a best-selling author whose books include "Late Edition: A Love Story" and "Once Upon a Town: The Miracle of the North Platte Canteen."

(CNN) -- Some people are so big during their lives, even death doesn't seem to entirely take them away.

So it is with Frank Sinatra. He left this earth in May of 1998, yet there is seldom a day when you don't hear his voice drifting out of a radio, seldom a week when you don't catch a flash of his face on a television screen, or read a reference to him in a newspaper or a magazine. Sinatra: The word itself signals something. Those three quick syllables: sharp, snappy, staccato. The images the name brings to mind: the Rat Pack, ring-a-ding-ding, very good years, strangers in the night. Many adored him, some despised him; few were indifferent.

In New York, especially, his voice remains omnipresent. His "New York, New York" might as well be the city's official anthem. Many times when I've visited Manhattan I have walked past what was said to be Sinatra's favorite restaurant: an unprepossessing-enough-looking Italian place on West 56th Street called Patsy's. This, Sinatra legend has it, is the spot where he could relax, where he felt most at home.

I'd never gone inside. I had imagined it as a peak-of-the-mountain place, a restaurant where only the most savvy would congregate, men and women who were at the pinnacle of their games, who had long ago learned and mastered all the angles. After all, this was where Sinatra had his regular table, wasn't it? How could mere mortals have a shot at fitting in?

This trip, I came in for dinner. And learned a lesson.

Sinatra, in his chairman-of-the-board years, in his sell-out-every-seat-in-the-concert-hall decades, did, in fact, gravitate to this place. But it wasn't because he was the biggest name in entertainment. It was because at one point in his life, he feared that he might be finished.

"My grandfather was the first member of our family to know him," said Salvatore J. Scognamillo, the current chef and co-owner of Patsy's.

The grandfather -- Pasquale "Patsy" Scognamillo -- had co-owned a restaurant nearby called the Sorrento during the first years of the 1940s. The young Sinatra was brought in one day by his boss, bandleader Tommy Dorsey. "I've got this skinny kid from Hoboken," Dorsey reportedly told Patsy Scognamillo. "Fatten him up."

Sinatra swiftly became an international singing idol whose voice and face made women and girls scream and faint; riots broke out at his concerts. Patsy, meanwhile, left the Sorrento and opened Patsy's. Both men -- the crooner and the cook -- were doing well for themselves.

But in the early 1950s, Sinatra's career crashed. He was no longer a kid. His records stopped selling. His romance with Ava Gardner was on the rocks. His record company dropped him. The winner suddenly was being widely seen as a loser, washed up.

People who follow the Sinatra story know about the eventual comeback: how he landed a role in the movie "From Here to Eternity" and won an Academy Award, how his career zoomed again, how he became the living symbol of success and swagger.

Yet in those down years, no one could have anticipated the rebirth. Sinatra was a has-been, yesterday's news.

"He would come in to the restaurant alone for lunch," Sal Scognamillo said to me. I could tell that this was a thrice-told family tale -- or a thrice-times-thrice-told tale. That didn't make it any less compelling.

"My grandfather would sit with him," Sal said. "There would be people eating lunch who would avoid making eye contact with Sinatra -- people who used to know him when he was on top. Sinatra would nod toward them and say to my grandfather: 'My fair-weather friends.'"

One November, on the day before Thanksgiving, Sinatra asked Patsy if he would make him a solo reservation for the next day. "He said he would be coming in for Thanksgiving dinner by himself," Sal said. "He said, 'Give me anything but turkey.' He didn't want to think about the holiday, but he didn't want to be alone."

The restaurant was scheduled to be closed on Thanksgiving. But Patsy didn't tell Sinatra that; he told him that he'd make the reservation for 3 p.m. He didn't want Sinatra to know that he was opening especially for him, so he invited the families of the restaurant's staff to come in for dinner, too. He cooked for Sinatra, on that solitary holiday, and it wasn't until years later that Sinatra found out.

That's where the loyalty came from. That's why Sinatra never stopped coming to the restaurant. In later years, when Patsy's would be jammed with diners hoping to get a glimpse of him, few understood why the most famous singer in the world would single out one place as his constant favorite.

It was no big secret to the Scognamillo family. They all knew. A person recalls how he is treated not when he is on top of the world, undefeated, but when he is at his lowest, thinking he will never again see the sun.

"Up those stairs, that's where Sinatra used to have his table," I heard a man say to his date as they entered the restaurant. He's still packing them in, 13 years after his death.

Who remembers a kindness that comes when kindnesses are in short supply? Who most treasures being made to feel welcome when every door seems to be slamming shut?

In the wee small hours of the morning, only the lonely.

The opinions expressed in this commentary are solely those of Bob Greene.