It’s a cool fall night, and I am at my alma mater’s law-review banquet. Our setting, the Audubon Tea Room, is every bit as posh as it sounds. Tropical centerpieces erupt from tables like floral fountains. Silk curtains tease hardwood floors. The swooping, vaulted ceiling, with its internal buttresses, reminds me somewhat of the beastly ribs that prop up the east wing of Notre Dame Cathedral. After a few speeches, our entrées arrive: succulent beef tournedos for some, trout for others, fresh root vegetables, and a dollop of buttery mashed potatoes. And as has become commonplace for me at celebratory events such as these, I feel a subtle, but substantial, lump in my stomach: indigestion brought on by my old companions, anger and shame.

Don’t misunderstand me. I’m aware that I’m sitting on my own shoulder again. That I should just slip myself into the moment like the others seated at my table. Call it navel-gazing. Call it whining. Call it what you will. But that’s the problem of double consciousness.

The annual event honors my former-school’s best. If you made it onto law review, then you’ve been granted admission into the upper reaches of the legal community. In this grand room, there are perhaps 280 of us, representatives from as far back as the Class of ’57 to the baby-faced graduates of ’10, ’11, and ’12. But that’s not what causes the flare-up.

I’m proud of the school, proud of my comrades in education, and proud of myself for having done well. Yet the voices are talking to me again. James Baldwin, Ralph Ellison, Toni Morrison, Nikki Giovanni, Amiri Baraka, my undergraduate professor Niyi Osundare, even Tupac Shakur and Kanye West. They’re a giddy, Greek chorus, or rather a ghostly, Pan-African one, trying to get my attention from the orchestra pit of my mind. My gut tells me to ignore them. But they’re insistent.

I find myself counting, against my will. There are four of us. Four African Americans out of 280. One from a class of the early eighties. Two from the nineties. And me representing the 2000s. In a city that has been majority black since at least the seventies, the chorus sings, How is this possible? But then I see the other half of the equation, the variable that explains all. I can’t tell how many servers are on duty, but they are legion. And almost every one of them looks like one of my aunts, uncles, cousins, me.

My table’s server stands over my shoulder, pouring Riesling. She is elderly, but not decrepit. Yet, in another context, this woman could be a deaconess at my church. No one else at the table tells her thank-you or even acknowledges her existence. Her face, her body, her life are erased by my tablemates’ failure to see. Although I often let the moment pass, tonight, apropos of nothing, I speak.

“Thank you,” I say. “How making out?” I’ve gone too far, I realize immediately. I’ve committed a faux pas by thanking her, inquiring about her feelings—and doing so in light Ebonic vernacular. But it’s only a faux pas in this place. Elsewhere, I would simply be conveying respect and acknowledging the dignity of this human in the way that I was taught by my elders.

When I say thanks, her face twitches with surprise. Not only do most people fail to see her, but she’s likely become accustomed to this erasure. And I’ve just destroyed her invisibility by bringing her into my reality. I’m a man in a several-hundred-dollar suit at a table with people in several-thousand-dollar suits. I’m supposed to ignore her. As smart as I supposedly am, I should have long ago developed a way to maintain distance without supporting the hierarchal system of my city. Again, I have failed.

This sense of failure follows me around New Orleans. It’s an acute ache that stings me almost anywhere in town, regardless of who I’m with or what I’m doing. But without a doubt the flare-ups, this racial indigestion, is worst when I’m dining in the French Quarter. When I was a student, I was fortunate enough to be pursued in much the same way that college football standouts are. This was in the waning days of largesse before the Great Recession killed the recruiting budgets of the big law firms. A common practice was to bring potential draft picks like me to flagship creole restaurants: Arnaud’s, Antoine’s, Galatoire’s, Commander’s Palace, Court of Two Sisters, Brennan’s.

I was dazzled, of course. I wasn’t like the other recruits. Many of my friends came from families of lawyers where such dinners were commonplace. I didn’t come from such a family. My family was working-class. My dad was a car salesman, my ma a home-health nursing assistant, my extended family truck drivers, cashiers, or unemployed. I was the first to complete college, the first to find a profession. Our idea of fine dining—indeed, the meals that I’m most nostalgic for on cold winter nights—took place at home. If we went out to eat at all, it was to the types of places that had smiling catfish in sailor hats on their napkins.

I was so inwardly focused, so unconscious during those salad days that I never noticed the things that I would years later, things that were there waiting to be discovered if I looked up or listened. For example, the “Old South” advertising campaigns the great restaurants of my hometown seemed to favor. One restaurant has a gallery of such items.

Once, I met some friends in one of the flagships. After finishing our bisques and baguettes in a room of golden chandeliers and Corinthian columns, we had time to kill, and this was a great group to kill time with. Although all of us lived in New Orleans, I was the most American of the group: one was Korean, another Canadian, another was raised in California, but by British parents. We found ourselves in the deepest penetralia of a 175-year-old establishment. I had visited before, but I’d never been in those secret rooms where pictures of doughy-faced, mid-twentieth-century men sat sternly around banquet tables deciding all our futures.

“Did you see this?” one of my friends, the Canadian, asked, as she glared at a display case. I was already discomfited by the black-and-white photos of the men in Mardi Gras masks that looked far too much like the Grand Wizard masks of the Klan. But this—this was different. There were several banquet programs from around 1910. Fish would be served. And shrimp étouffée. The star of this slip of paper was the drawing of the woman on the cover. She was stoop-shouldered, dark-skinned, and wearing a scarf over her hair. In other words, a mammy. “I don’t get it,” the Canadian said. “Why would someone do this?”

“Why” is a constant question. What is the fascination that people have with the Old South? And why does it attract millions of visitors to my city every year? There’s no doubt that New Orleans is beautiful. We don’t have mountains or beaches, but our architecture predates most anything west of the Mississippi River. And unlike the stoic churches and taciturn town halls of the Eastern Seaboard, there’s a lightness of touch—a mild whimsicality—to the mostly Spanish-era buildings in the so-called French Quarter.

That sleight of hand, calling something one thing when it’s really another, is a big part of the attraction. My Pan-African chorus roars with derision when I come upon one of the advertisement stands that populate the lobbies and anterooms of most local restaurants and hotels. Sure, there are plenty of ads that I expect, like the exhortations to go airboat riding. But then there are the ones nestled among dozens of others that, once I read the text, prick like a scorpion’s telson. An entire class of these pamphlets promote the Old South dining experience. Others, the Old South touring experience. One pamphlet advertises “the highly skilled workmanship of slave artisans” on display at a nearby plantation.

These Orwellian notions of freedom and commerce are not lost on me when I wander through Jackson Square, named for a planter whose wealth flowed like sweat from the backs of such artisans, human beings who literally slaved away at the Hermitage. Jack’s family owned more than 300 humans. He paid $1,000, in today’s dollars, for recaptured runaways and an extra $4,000 if the runaway was beaten before return. (This probably accounts for the schadenfreude I feel whenever I walk past the former location of a creole restaurant called Jackson, which didn’t last more than a year at its tony Magazine Street location.)

At the Audubon Tea Room, I dry my hands in the restroom and step back into the banquet area. A pair of servers hovers near the restroom door as if they don’t know what they should be doing with themselves. Their backs stiffen when they notice my approach. It occurs to me that these men serve at my pleasure or at least the pleasure of the people who hold my contract papers.

It seems this is simply the way things are done at Audubon (formerly known as the de Bore plantation). Audubon Park is named for the great naturalist John James (originally Jean-Jacques). He was born on his father’s plantation in Saint-Domingue. At least one of his half-siblings was the offspring of a chamber maid his father owned, as was Audubon himself.

I was raised in the primarily African-American suburb east of New Orleans known matter-of-factly as New Orleans East. Like many suburban kids, I didn’t get into the city that often and was surprised in my early twenties to encounter the unbridled splendor of Audubon Park, which for my money is the most beautiful urban park in America. I love Central Park in New York and Chicago’s Millennium Park, but Audubon has the most wonderful mixture of facilities—a zoo, a baseball diamond, a picnicking area on the Mississippi, countless ancient evergreens, and dual jogging paths that let right out on the Saint Charles Avenue streetcar line. Most important, Audubon Park has a lovely, verdant wildness to it.

Audubon Park was a new world to my young eyes. And I wouldn’t understand—wouldn’t begin to understand—the basis for the alien nature of the gorgeous park for many years. The patrons were the difference. At Joe Brown Park in New Orleans East, all of the patrons looked like me. They were postmen, beauticians, ministers in track suits and Adidas sneakers. In Audubon Park, the patrons were nearly all white. As someone who went to a high school that, like my elementary school, was more than 95 percent black, I simply wasn’t used to being around white people, let alone white people who drove Land Rovers and wore yoga pants, who chatted about wealth management, summer homes in Aix-en-Provence, the darling recipe for daube stew they discovered the other day, whose gaits and glares seemed somehow off compared to the welcoming amble and eyes of the folk I encountered back home.

But I wasn’t an idiot. I understood that this was Uptown New Orleans, that I was in a park surrounded by mansions and white-glove restaurants. This park was made for the pleasure of the people who lived in those mansions and ate in those restaurants. I occasionally saw other African Americans engaged in the same activities as the whites I saw. These were most often students like me who needed to get a sprint in before class. But most of the blacks I encountered were the workers who cultivated and preserved this pleasurable space. They were the cleaners who swept away detritus with their brooms and the guards who prowled the park perimeter in security pickups but never stopped or spoke to anyone, men who never made eye contact when I jogged to match their speed. It occurred to me that people who looked like me had one safe role in my divided city: We were made to serve.

My maternal grandfather, whom I called Grandpa, wasn’t the most talkative man, but if there was one thing I knew about him it was that he was a hard worker. He worked for the city sanitation department alternately driving dump trucks and street sweepers for about thirty years. In his downtime, he cut grass for extra money. In his downtime from his downtime, he cut my grandmother’s grass. I knew of all this from his mouth and through observation. But what I didn’t and wouldn’t know until more than twenty years after his death in the mid-1980s was that he worked in a place called the Plimsoll Club. New Orleans is not notorious for its exclusive clubs. This is by design. I wouldn’t understand until I was a grown man that places I had heard murmurings about throughout my life—the Louisiana Club, the Pickwick Club, the Boston Club—weren’t merely rumors. They once really existed. Some still existed. They had one thing in common: exclusion of the other. New Orleans’s wealthiest and most powerful leaders hung out in these spaces smoking cigars and eating finger sandwiches. Women were not allowed except as arm candy during Mardi Gras. And blacks were excluded, unless they were carrying the finger sandwiches.

To be fair, the Plimsoll Club was not one of those old-line establishments. It only dated back to the 1960s, not the 1860s. But I couldn’t help feeling out of place the first time I was there for a sit-down meal. This was a banquet thrown by the local chapter of the Federal Bar Association. The location was stunning. The club perched atop New Orleans’s World Trade Center, which was itself lodged in the crook of the Mississippi River’s elbow. From that vantage, one could see all the neighborhoods and restaurants of the city and imagine what the city was like back when the dividing lines were more obvious, back when all the land was neatly separated into plantations: Avart, LeBreton, Ducros.

Sitting down to nibble on my introductory Plimsoll salad, I had the feeling that I had made it. After all, wasn’t it true that my grandfather and grandmother’s generation had fought and often lost at great cost to both life and limb for my right to sit unquestioned in this room?

Yet, that feeling—the racial indigestion—disturbed me perhaps for the first time that bright afternoon. Grandpa had worked in this establishment most of his life. But he never mentioned the people he must have encountered, the lawyers, bankers, and politicians whose comfort he controlled and burnished. I can’t know whether this was out of some feeling of anger or shame. He never spoke of it, didn’t leave a journal, made no offhand comment to my preadolescent self that survived the decades. But I was aware enough of the Old South uniform that the Plimsoll waiters wore. It was a costume that I would see countless times and never question. Until I questioned it.

In 2013, Paula Deen stirred controversy when it was revealed that she was to throw an Old South wedding for her brother. And she would have gotten away with it, too, if not for a lawsuit filed by a former employee. What was one of the items that would have made the wedding perfect in Deen’s eyes? That all the servers be black men clad in the servile costume of plantation-era house negroes: baggy white coats, black pants, black bow ties. It wasn’t until the Deen incident that I realized how prevalent was this desire for a return to the good old days of the good Old South.

The Plimsoll waiters wore the costume. The countermen at Camellia Grill, too. So did the men (and women) who worked in so many of the Quarter restaurants. I noted such depictions on the wall next to the mammy-menu. It was what Uncle Ben wore on his boxes of instant rice. This discomfort is my grandfather’s legacy.

It reminds me of the Chris Rock bit where he describes his ultra-upper-class neighborhood in Alpine, New Jersey. Out of hundreds of families only three others are black: the families of Eddie Murphy, Mary J. Blige, and Jay Z and Beyoncé. His fellow blacks are legends of the entertainment world, absolute exemplars of success. In buying his mansion, Rock has vaulted his family into the realm of American exceptionalism. And he envisions living among other elites, perhaps Bill Gates or Richard Branson. Instead, he learns that his closest neighbor, a white man, is a dentist. Not a Nobel Prize-winning dentist, but an ordinary, yank-your-molar dentist. In other words, Rock accomplished incredible feats relative to the overall black community just to make it to above average in the mainstream community.

At my table in the Plimsoll Club, I’m astounded at the view, at the company, at the fact that I, a relatively new law graduate, was invited. In some sense I feel as though I’ve quietly, in my country-mouse way, climbed to the mountaintop of Dr. King’s dream. Later, I will recognize that my mountaintop is a mere plateau to most of the people in the room. I don’t finish the chicken entrée that afternoon.

An Irish scholar named John Elliott Cairnes turned his eye to America in 1861, just before the outbreak of the Civil War. This was a time when the Confederacy, quite shrewdly, was trying to convince Britain and France to float over and help smash Lincoln and his imperial desires to infringe on their states’ rights. Cairnes was an incredible investigator—think of Alexis de Tocqueville, but on a mission to analyze America’s racial hang-ups. In his research, he was most struck by slavery. Not that it existed—he knew that slavery had existed throughout history. But he was shocked at America’s particular innovation: racial branding. In ancient Greece, a man might have been a slave, but his child was automatically born free, and there was no correlation between skin color and free status. But the Americans had developed something new and vile; Cairnes posited that by linking color to slavery America had infected itself with something that—regardless of who would win the coming war—would last for generations.

I think of Cairnes when I’m preparing myself for one of these celebratory meals. I dress well. I get a haircut. But racial branding is the reason for my indignation, and the reason I can’t get my shoes shined. Most downtown skyscrapers have a shiner, often an elderly black man. I walk by and see a well-dressed white man seated on the throne, oblivious to the person beneath him. It’s the same discomfort I get watching football.

If you want to know which segment of America is doing poorly during a given decade, look at the players on the gridiron. Professional football players of the mid-twentieth century were largely the children of immigrants, men with hyphenated demographic backgrounds: Italian-Americans (Gino Marchetti, Alan Ameche, and Leo Nomellini), Polish-Americans (Mike Ditka, Ray Nitschke, and Alex Wojciechowicz) and even Jewish-Americans (Sid Luckman, Marshall Goldberg, Charles Goldenberg, and Benny Friedman). Today, the vast majority of non-quarterbacks are black.

It’s the same discomfort I feel when a fellow professional raves about improvements in New Orleans’s public schools, which were obliterated, like the housing developments, not by Hurricane Katrina, but by administrative decisions. New Orleans now has the most extensive charter-school system in the nation, a fact that I found comforting (the public schools of which I am a product were mostly awful) until I heard something strange while noshing on charcuterie outside of a friend’s party. I’d struck up a conversation with a young woman, white and new to the city, who worked at a particularly well-respected charter near the riverbend. She explained the rigid rules the students are required to follow: They can only travel from class to class in a counterclockwise direction. If their class is next door, then they must circumnavigate the entire school. Students are compelled to make eye contact with adults at all times during their interactions, but to avoid eye contact if not in active conversation. Failure to comply results in penalties. I try to imagine the partners of any of the firms where I’ve worked placing their children into such a penal environment.

I can’t help but wonder what is the cumulative effect of a sorting system that removes so many people from the path of opportunity our nation is known for. Some facts and figures: Over half of all black males in New Orleans have been reported to be unemployed or underemployed; about 35 percent of us lived in poverty in 2013; and approximately 100,000 of us lacked the economic and political power to return to the city after Hurricane Katrina and the federal floods (this figure includes thousands of citizens whose homes were undamaged but declared off-limits by HUD and HANO, the federal and local housing authorities, not to mention the decisions of the insurance companies, the state of Louisiana, FEMA, and others). Amy Schumer has caught flack for some of her crass racial comedy. But in her movie Trainwreck there’s a moment when her father, played by Colin Quinn, gets into an argument with another resident of the convalescent home where he lives. Whereas the resident argues that Babe Ruth was undeniably the greatest baseball player of all time, Quinn’s character retorts that it was easy for Ruth to be the best in a world that actively and totally excluded people with brown skin.

Whenever I’m walking along the streets of New Orleans and see a pair of young men clothed in the false dignity of their black bow ties and sloppy white coats, I wonder what each of them would have been—not in the best of all worlds, but in an average world, where equality is more than a national myth we believe to make ourselves feel better. I wonder about those same men when I pass Newcomb Boulevard, an Uptown street populated by people so wealthy and powerful that they simply blocked it off and claimed it as their own private thoroughfare. In the average world that I envision, would any of those young men have grown up on that street? Would they have inhabited those opportunities, leveraged themselves into doctorates and yachts off the coast of Tarifa, Spain? And how long could any of us, those young men or myself, walk down that street at night before having our bodies tackled and hauled away to Orleans Parish Prison, the nerve center of the most incarcerated city in America?

These fears of erasure are what power the electric fence both between so-called races and within them. If we embrace a human value system based on color, where light is right and dark is other, we create the New Orleans I’ve always known. Where we’ve had several black mayors, all of whom were light-skinned and had straight hair (except for the one with no hair).

It was at the Zulu Ball that I experienced a particularly fierce bout of indigestion. The Zulu Social Aid and Pleasure Club is the preeminent African-American carnival club. On Mardi Gras morning, the Zulu floats and marching bands act as the herald of Rex, which is the unquestioned “King of Carnival.” The Rex men wear those creepy pseudo-Klan masks to prove their primacy. An all-white organization, Rex successfully thwarted legal efforts in the 1990s to force integration of the membership. Rex’s motto is Pro Bono Publico or “for the public good.” Yet, many people who flock to see Zulu, bright and early, at 8:30 a.m., take a break during Rex to freshen up for the late parades. Hundreds of thousands pack Saint Charles Avenue and Canal Street to watch the Zulu participants who sport African-savage grass skirts, throw coconuts at parade-goers, and wear blackface.

The Zulu Ball is legendary. It starts late and goes into the wee hours. To reveal too much about the goings-on would be ungentlemanly. But I can say that I’d never seen so many beautiful women of all ages, shapes, and sizes in stunning ball gowns. And unlike the other organizations which hire expensive caterers to provide food, Zulu attendees bring their own dishes. It’s a potluck. It was around 2 a.m., as I feasted on juicy fried chicken, devoid of any discomfort whatsoever, that another attendee mentioned that a rival club, another African-American carnival outfit, used to meet nearby. I’d heard of the club in the same murmured way I heard about the Louisiana, Boston, and Pickwick Clubs. But this club was all-black and its claim to fame was utilizing the brown-paper-bag test. And so the feeling returned and my chorus cackled.

It’s when I’m out dining in my hometown that I feel most like a war correspondent covering a conflict that most people are unaware is being fought. I want to turn to the woman seated next to me at the Audubon Tea Room and explain what I feel. I want to connect and discard my alienation. But it would take more time than I have. I could find the words. But I don’t trust that she would hear them. I haven’t had a fine meal in New Orleans in years.