Out on a Harlem street corner in the summer of 1963, Clarence Funnyé, New York chapter president for the Congress of Racial Equality (CORE), set up a bank of TVs tuned to every major network. Over the course of several Saturday afternoons, Funnyé stood in the street for hours, offering a dollar to every passerby who could spot a black face on one of the channels. After six consecutive Saturdays, he’d given away $15.

Funnyé’s stunt was a part of CORE’s TV Image Campaign, an effort to pressure broadcasters into airing more, and more positive, images of black Americans on network television programs—and, just as importantly, in the advertisements that paid for them. Since its debut in 2007, Mad Men, AMC’s much-lauded show about the lives of Madison Avenue advertising executives in the 1960s, has become the target of a high-profile image campaign of its very own. Episode after episode, critics have assailed the show for failing to deal with race and for relegating black characters to the margins. With Mad Men’s fifth season about to debut on March 25, the chorus of critics will no doubt return as well.

Last season, Slate’s sister site The Root debuted a Mad Men Black People Counter, which, like Clarence Funnyé on the street corner, tracked the number of black faces on the show from week to week, typically noting nothing more than an appearance by Carla, Don and Betty Draper’s maid. “Although much has been made of the show’s treatment of race,” wrote Daniel Mendelsohn in the New York Review of Books last year, “the ‘treatment’ is usually little more than a lazy allusion—race never really makes anything happen in the show.” Critics have catalogued and dissected the show’s sporadic and fleeting tangents on race, like the subplot regarding copywriter Paul Kinsey’s self-regarding dalliance with a black activist girlfriend. During Mad Men Mondays at the New Republic, Matt Zoeller Seitz warned that treating race “as a looming presence and nothing more is dramatically risky. Whatever Mad Men is doing here, it had better pay off.” On Slate’s DoubleX blog, Latoya Peterson opined bluntly that if Mad Men continues to ignore race, “it is truly written by cowards.” But what much of the commentary misses is that Mad Men is not just a show that happens to be set in the racially charged 1960s. It’s a show about advertising. And it is advertising, not Mad Men, that is written by cowards.

The way we see race handled in the offices of Sterling Cooper is an accurate depiction of the perverse relationship that existed between blacks and Madison Avenue at the time, and indeed since the beginning of the modern advertising age. As the industry began to mature in the 1920s and 1930s, manufacturers of brand-name goods mostly ignored black consumers; the pages of black newspapers were dominated by advertisements for beauty care products specific to the black market, cigarettes, and liquor. This was in part because nobody thought black consumers had much money, but mostly because nobody thought much about them at all. Even those companies that did recognize the considerable purchasing power that blacks had didn’t advertise to them for one simple reason: They were scared for their brands to be associated with blacks. An industry handbook published in 1932 tried to encourage companies to market to blacks by reassuring them that black America was so invisible to white people that it was possible to reach those shoppers in relative anonymity.

This thesis would prove to be false. After World War II, seeking a competitive advantage over market leader Coca-Cola, Pepsi’s staunchly liberal CEO Walter Mack decided to radically expand his company’s marketing efforts in the underserved black community. He hired a team of black sales reps and marketers to blanket the rural Black Belt of the South and the urban enclaves of the North. This team was responsible for hiring some of the first black fashion models used by a major brand. They created the first in-store, point-of-purchase displays targeting black shoppers and hired Duke Ellington to endorse Pepsi from the stage during his shows. Some even went so far as to marry their brand with the moral uprightness of the emerging civil-rights movement, disseminating information about Coca-Cola CEO Robert W. Woodruff’s outspoken support for segregationist policies and politicians.

Black consumers, intensely loyal to institutions that showed them respect, turned out in droves. The campaign was a great success. It was so successful that it had to be killed, and quick. While Coca-Cola remained wholesome and All-American, Pepsi was becoming known as “nigger Coke.” Fearing a total collapse among white soda drinkers, at a conference for regional Pepsi bottling executives in 1949, the otherwise progressive Walter Mack took to the podium and said he would not let Pepsi become, in so many words, “a nigger drink.” His top black sales rep, who was in attendance, got up and walked out. Shortly thereafter, the company’s black marketing efforts were quietly scuttled. In 1953, singer/actress Polly Bergen was rolled out as “the Pepsi-Cola Girl,” a fresh-faced, lily-white makeover for the brand.

Pepsi’s dilemma—the dilemma of every advertiser at the time—was captured perfectly by Mad Men creator Matthew Weiner in the Season 3 episode “The Fog.” Squirrely account man Pete Campbell is charged with finding a new campaign strategy for the struggling Admiral Television brand, and it gradually dawns on him that the company’s otherwise dismal sales are actually quite robust in places like Oakland, Chicago, D.C, and Kansas City. “Great jazz cities,” observes Paul Kinsey, the pipe-smoking Negrophile, leading Pete to a moment of epiphany: The Admiral brand is taking off among black consumers. He pitches the client a black-targeted marketing campaign, ads in Ebony and black newspapers, outreach to black communities, the works. Pete is no socially conscious crusader. Indeed, he’s so white and such a child of privilege that he has no understanding of the politics of the color line, and he’s quickly shut down by his superiors, men who would have been all too familiar with the fate of Pepsi’s forward-thinking campaign. “Admiral Television has no interest in becoming a colored television company,” fumes agency partner Bert Cooper, calling the junior man on the carpet.

Cooper is smart enough to be a coward. He knows what was true then and, sadly, is still true today. Will white consumers abandon a product once its brand is too black? Yes, they will. Will black consumers abandon a product once its brand is too black? Yes, not wanting to be stereotyped, they will. Even as multicultural image campaigns rightly lobby for more and better black representation in commercials, and as much as America now embraces the endorsement of certain black celebrities, the politically incorrect truth is that there’s a tipping point. The moment a product is “ghetto,” white consumers are gone—and then black consumers are gone, too.

Weiner and Co. deftly addressed the evolution of the issue two years later in Season 4. It’s now the summer of 1965, and Sterling Cooper faces a new Pepsi-grade problem. Its client, Fillmore Auto Parts, is being picketed for refusing to hire blacks in its Southern stores. Black workers and consumers are no longer content to be invisible, and the answer to the race problem is no longer as simple as it was for Admiral Television. Peggy Olson suggests hiring Harry Belafonte to be a company spokesman, to show a racially progressive image. But once again the conservative attitude of senior management wins out. “Our job is to make men like Fillmore Auto,” Don Draper says, “not make Fillmore Auto like Negroes.” Coarse as that may sound to modern ears, it’s not wrong. I mean, it’s wrong ethically, but it is the correct business strategy. If an advertising agency has a moral conflict with a client, the only appropriate means of protest is to refuse to partner with that client. So long as an agency stays on, its role is to protect the client’s balance sheet. If Fillmore Auto’s brand came under fire in today’s racism-averse world, Sterling Cooper’s responsibility would be to hype the company’s “commitment to diversity,” regardless of the economic reality faced by its black employees. Either way, Don Draper’s maxim still applies. The job is to sell auto parts, not agitate for social justice.

This is not to say that ad agencies shouldn’t be held to account for fair treatment of minorities in the office or on the airwaves. They should. Society needs activists on street corners with banks of televisions. But any racial criticism of the business—the real one or the fictional one—has to proceed with an understanding of the universe in which that business operates. The most we can reasonably expect from Madison Avenue is that it isn’t flat-out lying to us, and even that’s a bit naive. For all their alleged hipness, most advertising agencies are fundamentally retrograde, risk-averse institutions. And the thing that they’re scared of is us, the fickle and racially biased marketplace. We’re the problem. The bank of televisions isn’t out there on the sidewalk to educate Don Draper. It’s there to educate the people on the sidewalk.

Mad Men isn’t cowardly for avoiding race. Quite the opposite. It’s brave for being honest about Madison Avenue’s cowardice. While Don Draper and Sterling Cooper may seem woefully behind the times, that just means Matthew Wiener is right on schedule, historically speaking. And if Mad Men’s schedule stays on the course it’s been following, it’s a safe bet that the season now beginning will finally bring us to the point when black consumers stand up and refuse to sit at the back of the advertising bus.

Next: Why Sterling Cooper was particularly afraid of the race issue—and why Season 5 may finally bring race to the center of Mad Men’s story.