The Sand Hill Grill is shut, and has been for a while. There is a sign in the window saying the building has been condemned, is unfit for inhabitation. But the mural is still there, on the rough brick wall. A black man in a white jumpsuit, bag slung over one shoulder, back turned. It is supposed to be Tommy Bennett, who they all called Burnt Biscuits because he once scalded his legs climbing over his grandmother’s stove, stealing away out the window with her cooking. Or so the story goes.

It was Bennett who carried Tiger Woods’s bag when he played his very first practice round at Augusta, in 1995. And it was Bennett who brought Woods down to the Sand Hill Grill that same year, because it was where he and all the other local caddies gathered when the working day was done.

The houses round the Grill are all beat-up clapboard, most coated in peeling paint, surrounded by chain link fence. It is only a short walk from Augusta National, around the Country Club and past Ty Cobb’s place, but it feels a long way distant.

“I remember like it was yesterday, the feeling as I drove up Magnolia Lane into Augusta National Golf Club for the first time,” Arnold Palmer wrote in A Golfer’s Life. “I’d never seen a place that looked so beautiful, so well manicured, and so purely devoted to golf, as beautiful as an antebellum estate, as quiet as a church.” Palmer felt a “powerful thrill and unexpected kinship with the place”. Almost every golfer here has a similar story. Except Woods. When he came in 1995 he did not feel that, or anything like it.

“I don’t know what I expected,” Woods wrote in his fascinating new book, The 1997 Masters: My Story, “but I was underwhelmed.” Maybe, he continued, it was because “I was surprised by how short the drive was”, or maybe it was because “the club had excluded black golfers for so long”.

In 1995 his father, Earl Woods, put it this way: “The average golfer that goes there is blown away by the history and tradition. That doesn’t impress the black golfer. Black golfers have nothing in common with Bobby Jones, no historical ties with Bobby Jones. They prevented blacks from being here for many, many years.” Forty-one, altogether. Till Lee Elder played in 1975, the year before Tiger Woods was born.

After Elder came Jim Thorpe, Calvin Peete and then Woods, whose mother is from Thailand, and who identifies as Cablinasian. Over the years, the Masters has had many more black caddies than it has black golfers. “The past is never dead,” wrote William Faulkner. “It’s not even past.” And at Augusta, it is all around. There are bridges named after Byron Nelson, Gene Sarazen and Ben Hogan, water fountains with brass plaques dedicated to Arnold Palmer and Jack Nicklaus. But if you are looking for a memorial to Augusta’s black golf history, that mural of Bennett at the Sand Hill Grill is one of the few you will find.

Black golfers have nothing in common with Bobby Jones. They prevented blacks from being here for many, many years Earl Woods

One day, Woods will surely have one of his own here, marking his four victories, and especially the one in 1997, when he became the youngest winner, shot the lowest score and won by the largest margin in the tournament’s history. Back then most journalists figured he was, as Rick Reilly wrote in Sports Illustrated, a 21-year-old who “ate burgers and fries, played Ping-Pong and P-I-G with his buddies, screamed at video games and drove his parents to the far end of their rented house”. But in Woods’s book about the tournament, 20 years later, he opened up about what was in his mind, as he rarely had done before. “I knew all about Augusta’s history before 1975 of not inviting players whose accomplishments should have gotten them in, and also the club’s history, until 1990, of not inviting a black person to join.”

Woods met one of those men, Charlie Sifford, soon after he turned pro in August 96. And read his autobiography, Just Let Me Play, before winning here the following April. They grew so close that Woods came to call him Grandpa and later named one of his children after him. Sifford taught Woods his history. It was Sifford, in Just Let Me Play, who quoted Clifford Roberts, Augusta National’s founding chairman, as saying “as long as I live, there will be nothing at the Masters but black caddies and white golfers”. There were certainly times when Roberts spoke warmly about black players, so that quote has been disputed. Sifford was happy to go on the record with his take: “All I wanted was a chance to play. Then motherfuckers kept me out.”

Sifford was not there to see Woods in 97. But Elder was. He was so keen to get to the course on Sunday morning that he was pulled over for speeding on the drive from Atlanta airport and given a ticket by an unsympathetic cop. He walked in Woods’s gallery, along with those two black Augusta members, Ron Townsend, who owned a TV network, and Bill Simms, who had a stake in the Carolina Panthers. Townsend and Simms had been invited to join Augusta National only after the controversy at Shoal Creek in 1990. That was when another Augusta member, Hall Thompson, proudly said that the Shoal Creek club “don’t discriminate in every other area except blacks”.

When Woods came up the 18th, Elder, Townsend and Simms were waiting for him, along, Elder said in a recent oral history on golf.com, with “every black who was working at Augusta”. It was not just caddies but “waiters, bus boys, bartenders, cleaning people and maids”.

When he was 14, Woods said in a TV interview that he wanted to win the Masters because “the way blacks have been treated there” meant “it would be really big for us”. All these years later, he wrote in his book that he “hoped my win, and how I won, might put a dent in the way others perceived black people” but that he “knew that none of this meant, necessarily, that things would change dramatically for minorities in golf”.

Corey Rogers, the historian at Augusta’s Lucy Craft Laney Museum of Black History, says that Woods’s win in 97 “broadened the interest of golf universally”, that “it made a sport that was somewhat popular within the African-American community really popular”. He adds: “Whereas many hardcore golf enthusiasts within the African-American community followed those that came before Tiger Woods, like Calvin Peete, Lee Elder, Tiger made golf appealing to the non-golf and non-sports individual.”

But Woods was right. There was no rush of black talent on to the professional tour. And these days, there are far fewer black caddies too. The Sand Hill Grill is shut, and has been for a while. But the mural’s still there, fading.