Clarence Otis Jr was nine years old in 1965 when an uprising in his neighbourhood of Watts in Los Angeles helped fan the embers of unrest that swept through the US for the rest of the 60s. Nearly five decades later, Otis – who in 2004 became one of the nation’s seven black Fortune 500 CEOs when he was tapped to lead the Darden restaurant group – says the aftermath of that upheaval laid the ground for his future success.

In 1967, the president, Lyndon B Johnson, in response to the spiralling turmoil, established the National Advisory Commission of Civil Disorders, which, in a blistering report a year later, largely blamed white racism and indifference for the despair plaguing African Americans. A century after emancipation, it called on leaders of American institutions to address “pervasive discrimination in employment, education and housing which has resulted in the continuing exclusion of great numbers of negroes from the benefit of economic progress”. The report cast in stark relief white America’s systematic exclusion of black people from decent housing and education and positions in trades and professions, including news media, Hollywood and the corporate United States.

Johnson had already laid out his ambitious vision for a more racially just and inclusive nation in the early months of his presidency. In a 1964 speech at the University of Michigan, he said that a great society “rests on abundance and liberty for all. It demands an end to poverty and racial injustice.” In a nation flush with prosperity, some 36 million Americans, nearly three-quarters of whom were children or senior citizens, were living below the poverty line.

Johnson quickly ushered in sweeping legislation, including the Civil Rights Act of 1964, to outlaw discrimination, and the Voting Rights Act of 1965, which forbade literacy tests and other measures that had prevented black people in the south from voting. The 1965 Higher Education Act set up work study, federal grants, and low-interest loans for college, making it more accessible to the poor and working class. Medicare, Medicaid, food stamps and the 1966 Child Nutrition Act brought healthcare and greater food security to the poor and elderly. The Job Corps programme established skills training and temporary employment, and Head Start, as its name implied, offered early education along with medical and dental care to level the playing field for poor children.

Johnson moved quickly to enforce the new laws targeting injustice. He substantially increased federal funding for hospitals and schools, then used those funds as a lever to ensure compliance with the law forbidding discrimination based on race, colour or religion in public accommodation and facilities that received federal funding. He dispatched inspectors to schools and hospitals across the south to monitor compliance.

As Joshua Zeitz, the author of Building the Great Society, said, the results were “astonishing”.

Between 1965 and 1968, the number of black students in the south who attended better-resourced majority-white schools rose from roughly 2.3% to almost 23.4%, and peaked at 43.5% in 1988. Between 1964 and 1984, the gap in high-school graduation rates between black and white students closed from 24 points to 16 points. Hospitals, nursing homes and other facilities that received federal funding were desegregated under the watchful eye of inspectors.

Facebook Twitter Pinterest Clarence Otis Jr in 2010, then CEO of the Darden group. Photograph: Lawrence K Ho/LA Times via Getty

Following the passage of the Voting Rights Act, federal examiners swept into Alabama, Louisiana, Mississippi and Georgia, where, by the following January, about 90,000 voters were added to the rolls. By 1970, about 66% of African Americans in the deep south were registered, and most were able to vote.

The urban riots highlighted the persistence of discrimination that kept black people in overcrowded, under-resourced neighbourhoods and excluded them from most professions and unions. By the end of the 1960s, many of the doors that had long been closed to urban blacks were suddenly prised open.

Otis and his siblings – the children of a caretaker and a housewife – took advantage of programmes such as the Neighborhood Youth Corps, a federally funded initiative that created jobs for urban teens, and the Watts Towers Arts Center, which brought recreation and culture to his segregated and economically fragile neighbourhood. “We got a head start,” Otis said. “It definitely mattered during that era.”

Otis graduated among the top 100 of 700 students in Los Angeles’s predominantly black David Starr Jordan High School. Most of those 100 attended college – in Otis’s case, the prestigious Williams College, from which he graduated magna cum laude. He went on to Stanford Law School and, at 30, he was a vice-president at the New York-based investment bank First Boston Corporation. In 1995, he was recruited to Darden. He continued to move up the ladder, becoming chair and CEO nine years later.

Under his leadership, share prices almost tripled. During the last four years of his tenure, Fortune magazine listed Darden as one of the top companies to work for.

Otis assembled a senior leadership team that was 41% people of colour. “When you have these kind of senior leadership numbers, it becomes easier to make diversity a priority in everything you do – from hiring and talent management (development and promotion) to culture-building, to how you think about and treat customers,” he said.

But in recent years, diversity in many American industries has stalled and, in some instances, is in retreat. Otis stepped down in 2014, and with the retirement in 2018 of Kenneth Chenault as CEO of American Express, the number of black Fortune 500 CEOs has decreased to four. African Americans, who comprise roughly 13% of the US population, are fewer than 1% of Fortune 500 CEOs.

“The fact that we’re in this situation is, I think, a real problem and embarrassing for corporate America,” Chenault said before stepping down. “One of the biggest issues for our society is diversity and inclusion. We should have far more representation.”

Half a century after Lyndon B Johnson’s clarion call for African American inclusion in mainstream American life, black people and other racial minorities remain strikingly under-represented in most elite fields, particularly in management. Johnson may have underestimated the depth of white resistance to full racial equality. While northern whites had supported civil rights legislation aimed at the south, they resisted efforts to integrate schools and housing closer to home. In 1966, legislation barring discrimination in the sale and rental of housing sparked a vehement backlash. That year, Martin Luther King Jr, who had been felled by a rock during a demonstration in Chicago, remarked: “I think the people of Mississippi ought to come to Chicago to learn how to hate.”

By the end of the 70s, Johnson’s transformative vision became a target for conservative politicians from Ronald Reagan to Paul Ryan, who worked tirelessly to dismantle key programmes and policies that had begun to disrupt generational poverty, and which had created long-denied opportunity for African Americans and other disadvantaged groups for a quality education and gainful employment.

In her book White Rage, the historian Carol Anderson recounts how black unemployment had sharply declined during the 60s and 70s, nearly closing the racial gap. However, under the Reagan administration, federal jobs and programmes that had aided that progress were cut, causing black unemployment to rise to 15.5% – the highest it had been since the Great Depression. Black youth unemployment rose to 45.7%.

“At this point, Reagan chose to slash the training, employment and labour services budget by 70% – a cut of $3.8bn,” Anderson wrote. College enrolment among African Americans tumbled from 34% to 26%. “Thus, just at the moment when the post-industrial economy made an undergraduate degree more important than ever, 15,000 fewer African Americans were in college during the early 1980s than had been the case in the mid-1970s,” she wrote.

Many US school districts are now as segregated as they were in the 50s, when the supreme court deemed segregated schools unconstitutional. Millions of black and brown children are condemned to overcrowded, under-resourced schools in neighbourhoods that Johnson’s programmes were attempting, with demonstrable success, to improve.

The sluggish pace of change has in recent years sparked claims of a renewed commitment to diversity. In 2017, more than 500 business leaders publicly pledged to advance diversity and inclusion in the workplace. In 2018, the prestigious global law firm Shearman and Sterling hired its first-ever chief diversity officer and established a task force to increase diversity.

Facebook Twitter Pinterest President Lyndon B Johnson in 1968. Photograph: Bob Daugherty/AP

The Ford Foundation is betting the farm on its ability to help diversify American institutions, throwing the full weight of its more than $500m in annual grants into initiatives that promote equality. “Progress won’t come without us being uncomfortable,” Darren Walker, the foundation president, said.

“People want to believe we can have diversity and not really get uncomfortable … It requires incumbent leaders and managers to change their behaviour and practices. It means that institutions have to change incentive structures and to fundamentally interrogate their own behaviour, which is very uncomfortable.”

But therein lies the problem. There is little evidence that many are inclined to accept that challenge. Leaders in numerous fields have for decades stated their commitment to diversity, but recent employment data invite scrutiny of their efforts. Especially revealing is the significant under-representation of people of colour in some of the fields often touted as being among the most progressive, including the arts, journalism, academia, fashion and the film industry. As Walker said, the answer lies in large part in overcoming the sense that racial inequality and discrimination have been overcome. “The challenge in the progressive community is benign neglect; benign validation,” states Walker. “When you look behind the curtain, the emperor has no clothes.”

In 2015, an art museum demographic survey commissioned by the Andrew W Mellon Foundation found that 84% of museum curator, conservator, educator and leadership positions were held by white people. Racial minorities were most often employed as museum security guards. “The case is clear and urgent,” Mariët Westermann, vice-president of the Mellon Foundation, wrote in the report’s introduction. “Constructive responses to it will be critical to the continued vitality of art museums as public resources for a democratic society.” However, that same year, Samantha Niemann, a white student denied a Getty Foundation internship created to address the dearth of racial minorities in museums, filed a lawsuit charging discrimination. A 2018 Mellon survey indicated the numbers of people of colour in leadership remained small.

At every turn, purportedly liberal and elite sectors maintain racial custom and tradition in their hiring until they are publicly shamed or otherwise coerced into widening access to people of colour. In New York City, the mayor, Bill de Blasio, made waves when he announced that the city would use city funding as a hammer to pressure cultural organisations to diversify their overwhelmingly white boards. In the city, where the population is 32% white, its cultural workforce is nearly double that, with leadership decidedly whiter. The city also pledged $1m to subsidise diversity programmes at its cultural institutions.

From culture to fashion to entertainment and beyond, racial and ethnic minorities are still virtually excluded from top management and corporate boards in a number of exclusive sectors. A 2018 survey of the 15 largest public fashion and apparel companies found that only 11% of board seats were held by non-whites, and 73% of chief executives were white men.

The newspaper industry, which for decades has vowed to diversify its professional ranks, also came under scrutiny when, in 2016, Liz Spayd, the public editor for the New York Times, faulted the paper for “preaching” but not practising diversity.

Spayd noted that only two of the 20-or-so reporters assigned to the 2016 presidential campaign were African American and none were Latino or Asian. Moreover, all six White House reporters were white, while the Metro staff had only three Latinos among its 42 reporters in a city with the nation’s second-largest Hispanic population. The entire Style section writing staff was white, and none of the paper’s 21 sports reporters were African American, “yet blacks are plentiful among the teams they cover”.

The discomforting critique did not win the already embattled Spayd many friends at the Times, and in July 2017 the paper disbanded its public editor role, calling it outdated. Perhaps it is, but Spayd’s critique was not: while African Americans held 9% of New York Times newsroom jobs in 2015, in 2018 their percentage had slightly dipped to 8%. That said, the Times is far more diverse than most American newspapers. Four decades after the newspaper industry pledged to have newsrooms that reflect the proportion of minorities in the population by the year 2000, they remain disproportionately white.

Get the Guardian’s award-winning long reads sent direct to you every Saturday morning

While the numbers in television broadcasting have long exceeded those in radio and print, people of colour, despite gains, remain disproportionately under-represented. The 2018 survey of TV and radio newsrooms by the Radio Television Digital News Association found their numbers dwindle at the top. Just 6.4% of general managers were of colour, down 1% from 1995.

In 2019, CBS News caused a Twitter storm when it released its slate of 12 digital reporters and associate producers assigned to the 2020 presidential campaign. While the team included four people of colour, none were African American, the nation’s largest racial minority. The omission was especially conspicuous at a time when black people are disproportionately affected by a number of high-profile national issues, including over-policing and mass incarceration, gun control, voter suppression and widening income and education disparities. Among the critics was Alexandria Ocasio-Cortez, the newly installed member of the House of Representatives from New York, who tweeted: “Unacceptable in 2019. Try again.” Sarah Glover, president of the National Association of Black Journalists, said it is unfortunate that diversity remains a problem more than 50 years after the Civil Disorders commission report. “CBS’s political team takes previously heralded steps back half a century,” she said.

Hollywood also attracted unwanted attention over revelations that for two years in a row none of the Academy Award acting nominees were of colour, prompting #OscarsSoWhite – a campaign launched by the nascent activist April Reign – to go viral on Twitter. A Los Angeles Times study in 2012 found that academy voters were overwhelmingly white and male, and that fewer than 4% of the Oscars for acting had been given to non-whites. Four years later, Stacy Smith, co-author of a University of Southern California study, concluded that “there’s not just a diversity problem in Hollywood; there’s actually an inclusion crisis.”

If the tech industry portends the future, then the forecast at Google, LinkedIn, Facebook and the other Silicon Valley tech giants is uninspiring. At Facebook, African Americans comprised 1% of technical and 2% of leadership roles, according to Facebook’s 2018 Diversity Report. Eighty-three per cent of Silicon Valley executives are white, and mostly male.

In 2016, black and Latino candidates at Google each made up 4% of all new hires. Google’s overall workforce is 54% white, and 40% Asian. Meanwhile, roughly 26% of bachelor’s degrees in science, technology, engineering and mathematics fields were awarded to black and Hispanic students, according to the National Center for Education Statistics.

In 2018, in the wake of its fifth “disappointing” diversity report, Facebook appointed the former American Express CEO Kenneth Chenault as its first African American board member. Chenault was also added to the board of Airbnb, the tech hospitality company, in the wake of widespread reports of racial discrimination by hosts. Other tech companies, including Apple and Uber, have begun releasing their own diversity reports.

The severe under-representation of black, Hispanic and female workers in tech has fuelled tension and headlines. In 2017, a memo by a white male employee protesting against Google’s diversity efforts created an international firestorm, culminating in his firing. In 2015, Twitter – already under fire for its glaring under-representation of black and Latino staff (2% and 4% respectively) – triggered a backlash when it hired Jeffrey Siminoff, a white male, to fill a job as vice-president for diversity.

Renewed calls for diversity are playing out against the backdrop of resurgent white nationalism. In 2018, the FBI reported that hate crimes had spiked by 17% over the previous year, the third consecutive year of a reported increase. Meanwhile, a wave of protests at college campuses – including some of the nation’s most prestigious and progressive schools – has illustrated the extent to which racial tensions persist over many of the same issues that roiled campuses in the 60s. Among the colleges and universities under fire were Yale, Princeton, Oberlin and Wesleyan, where students complained about the racial climate, the paucity of teaching staff and students of colour, and the curriculum. Figures from the National Center for Education Statistics suggest why: during the fall of 2016, among full-time professors at degree-granting universities or further education colleges, 81% were white. These figures include faculties at historically black colleges and universities.

The fight for racial diversity has long been an uphill struggle, but it is now waged in a far more polarised climate, in which many white people now claim that they are being disenfranchised as others are afforded undue advantage. An NPR poll conducted in 2017 found that 55% of white Americans believe they are discriminated against, while, tellingly, a lower percentage said they had actually experienced discrimination. A Reuters survey in 2017 found that 39% of white people polled agreed with the statement that “white people are currently under attack in this country”.

Facebook Twitter Pinterest Ken Chenault, former CEO of American Express. Photograph: Shannon Finney/Getty

Many white people saw the election of Barack Obama as president as evidence that racism is passé and African Americans have achieved equality – if not unmerited advantages. The New York Times optimistically proclaimed “Obama Elected President as Racial Barriers Fall”, despite polling data that showed that in 2008 Obama received just 43% of white votes compared to 95% of black, 67% of Hispanic, and 62% of Asian votes. In 2012, Obama received 93% of black, 73% of Asian and 71% of Hispanic votes. The percentage of white voters dropped to 39%. Both times he lost the white vote, with white women divided and white men decidedly opposed.

Nonetheless, the term “post-race” became ubiquitous, while “political correctness” or “PC” – the valuing of sensibilities that fall outside of the white, Christian, heterosexual mainstream – became a slur. White men – who retain dominance over every influential field – increasingly claim that they, and not those who by law and custom had historically been shut out, are under siege.

Today, it is not uncommon to hear even those who are considered white progressives complain of white male discrimination, despite their overwhelming over-representation in most American institutions. Few take the time to consider that the centuries-long dominance of white males was only possible due to the subjugation of women and people of colour – a legalised oppression rationalised by baseless inferiority theories rooted in religion and the academy. Sadly, these beliefs continue to resonate, and some view their domination as a birthright and racial minorities who seek parity as unentitled trespassers.

Whether heralding a post-race era or lamenting that they were losing ground, many white people conveniently ignored the harsh reality of a widening economic chasm between black and white. Between 1983 and 2013, the wealth of the median black household declined 75%, and 50% for Latinos, while wealth of the median white household rose 14%. The median income for black people was 65% of that for whites. Black unemployment has consistently been at least twice as high as it is for white people for most of the past five decades.

Even as some bemoan the decline of white men in the workplace, the extent to which their numbers have dropped is primarily due to the progress of white women. Since 1968, when President Johnson expanded affirmative action policies to include women, white women have made far greater strides in professions and in college admissions than have people of colour.

Still, the idea that racial barriers have fallen has swayed court decisions. Among the most glaring examples is the 2013 US supreme court’s five-to-four vote striking down key provisions of the 1965 Voting Rights Act that for decades had protected African Americans from blatant acts of voter disfranchisement.

“Our country has changed,” wrote chief justice John G Roberts Jr in his unduly optimistic majority opinion. “While any racial discrimination in voting is too much, Congress must ensure that the legislation it passes to remedy that problem speaks to current conditions.”

In doing so, the court patently ignored current conditions and the continuing barriers to African Americans’ voting rights and those of other marginalised populations. Since that decision, more than 22 states have passed restrictive statutes making it harder to vote. In 2016, a US court of appeal found that North Carolina’s voting restrictions had specifically targeted African American voters “with almost surgical precision”. But earlier this year, the supreme court declined to rule on the issue of gerrymandering in the state. Critics warned that this decision could give a “green light” for abuse.

When viewed as a zero-sum game, it is perhaps understandable that those who historically had all the marbles fear that equal opportunity will mean they will have fewer. From that reductive vantage point, even those who see themselves as progressive adherents of American ideals might privately – and sometimes publicly – view diversity as a threat. Even while retaining a disproportionate share of the pie, many white people apparently fear that, in this land of plenty, gains for racial minorities will imperil them.

Anxiety over immigration and the nation’s shifting racial demographics overshadows critical economic problems afflicting Americans of all races. Between 1970 and 2016, the income gap between Americans at the top and the bottom increased by 27%.

Compounding growing inequality is the rise in automation and the fact that many Americans of all races are ill-prepared for jobs that would go unfilled if not for immigration. The same day that Trump scapegoated immigrants in a 2018 speech in Montana, a report from ADP and Moody’s Analytics warned that the nation’s labour shortage had reached a critical shortfall. It cited figures from the Bureau of Labor Statistics that showed that, for the first time, there were more openings than available labour.

In addition, climate change threatens to cause the loss of 1.2bn jobs in fields such as agriculture, fisheries and forestry, and a 30% drop in the GDP by the year 2100 unless the nations that most contribute to the rise in carbon gas intervene.

Why we should bulldoze the business school Read more

Into this void of imagination, leadership and progressive policies enter demagogues who prey on centuries-old fears to sow division, while growing income inequality across all races, and the abundant opportunities that are needlessly squandered, go ignored. Rather than post-race, the US is becoming post-white, and race appears to matter more than ever.

While the evidence is all around us, too few acknowledge the effect of racial bigotry or the resistance to racial inclusion in fields in which people of colour have historically been left out. Diversity – or, more pointedly, the paucity of it – is a vexing issue that cuts across political persuasions; even some of the paragons of progressivism often fail to notice how monochromatic their offices and boardrooms are. Like an endless loop, these segregated settings reproduce workplaces that reflect homogenous social spheres.

This is an edited extract from Diversity Inc by Pamela Newkirk, published by Bold Type Books

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.