In his keynote address at the Democratic National Convention in Boston on 27 July 2004, before 9 million viewers, Barack Obama presented himself as the embodiment of racial reconciliation and American exceptionalism. He had humble beginnings and a lofty ascent, and in him both native and immigrant ancestry and African and European ancestry came together. “I stand here knowing that my story is part of the larger American story … and that in no other country on Earth is my story even possible,” he declared. “America, tonight, if you feel the same energy that I do, if you feel the same urgency that I do, if you feel the same passion that I do, if you feel the same hopefulness that I do, if we do what we must do, then I have no doubt that all across the country … the people will rise up in November, and John Kerry will be sworn in as president.”

Kerry lost the election, of course, and Bush seemed poised to embody the future of the Republican party. But Barack Obama seemed poised to embody the future of the Democratic party.

Two weeks after his exhilarating keynote address, Barack Obama’s memoir, Dreams from My Father: A Story of Race and Inheritance, was republished. It rushed up the charts and got rave reviews in the final months of 2004. Toni Morrison, the queen of American letters, deemed Dreams from My Father “quite extraordinary”. Obama had written the memoir in 1995 as he prepared to begin his political career in the Illinois senate. In his most anti-racist passage, Obama reflected on assimilated biracial blacks like “poor Joyce,” his friend at Occidental College in Los Angeles. In Joyce and other black students, he “kept recognising pieces of myself”, Obama wrote. People like Joyce spoke about “the richness of their multicultural heritage, and it sounded real good, until you noticed that they avoided black people. It wasn’t a matter of conscious choice, necessarily, just a matter of gravitational pull, the way integration always worked, a one-way street. The minority assimilated into the dominant culture, not the other way around. Only white culture could be ‘neutral’ and ‘objective’. Only white culture could be ‘nonracial’ … Only white culture had ‘individuals’.”

Obama’s anti-racist litany continued in his critical revelation of the “extraordinary negro” complex. “We, the half-breeds and the college-degreed … [are] never so outraged as when a cabbie drives past us or the woman in the elevator clutches her purse, not so much because we’re bothered by the fact that such indignities are what less fortunate coloureds have to put up with every single day of their lives – although that’s what we tell ourselves – but because [we] … have somehow been mistaken for an ordinary nigger. Don’t you know who I am? I’m an individual!”

Ironically, racist Americans of all colours would in 2004 begin hailing Barack Obama, with all his public intelligence, morality, speaking ability and political success, as the extraordinary negro. The extraordinary-negro hallmark had come a mighty long way from the poet Phillis Wheatley to Barack Obama, who became the nation’s only African American in the US senate in 2005. Since Wheatley, segregationists had despised these extraordinary-negro exhibits of black capability and had done everything to take them down. But Obama – or rather Obama’s era – was different. Segregationists turned their backs on their predecessors and adored the Obama exhibit as a proclamation of the end of racism. They wanted to end the discourse on discrimination.

“He’s the first mainstream African American who is articulate and bright and clean and a nice-looking guy.” Presidential hopeful and Delaware senator Joe Biden might as well have labelled Barack Obama the extraordinary negro. Biden’s evaluations of his presidential rivals appeared in the New York Observer days before Obama stood in front of the Old State Capitol building in Springfield, Illinois, and formally announced his presidential candidacy on 10 February 2007. Obama brimmed with words of American unity, hope and change.

But Joe Biden’s comments – which he later “deeply” regretted – became a sign of things to come. What was to come over the course of the campaign was a reflection of the audacity of racist minds – from George W Bush to radio mega-personality Rush Limbaugh to Democratic stalwarts – who all viewed Obama as an extraordinary negro. In February 2007, Time magazine speculated that African Americans were expressing greater support for New York senator Hillary Clinton because of questions over whether Obama was “black enough”. It couldn’t be because they saw Obama as a long shot. It had to be that they did not see Obama as ordinarily black like them, meaning inarticulate and ugly and unclean and unintelligent.

Pundits were dubbing Hillary Clinton the “inevitable” nominee until Barack Obama upset her on 3 January 2008, in the Iowa primary. By Super Tuesday on 5 February 2008, Americans had been swept up in the Obama “Yes We Can” crusade of hope and change – themes he embodied and spoke about so eloquently in his stump speeches that people started to hunger. In mid-February, his perceptive and brilliant wife, Michelle Obama, told a Milwaukee rally: “For the first time in my adult life, I am really proud of my country, and not just because Barack has done well, but because I think people are hungry for change.”

Facebook Twitter Pinterest Barack Obama in 2005, then the only African American member of the US senate. Photograph: Tim Sloan/AFP/Getty

Racist commentators searched for dirt on the Obamas. When no dirt could be found, investigative reporters started checking their associates. In early March 2008, ABC News released snippets of sermons from one of black America’s most revered liberation theologians, the recently retired pastor of Chicago’s large Trinity United Church of Christ. Jeremiah Wright had married the Obamas and had baptised their two daughters. In an ABC News report, Wright was quoted proclaiming, in a sermon, that “the government gives them the drugs, builds bigger prisons, passes a three-strike law and then wants us to sing ‘God Bless America’. No, no, no … God damn America for treating our citizens as less than human.” Wright had discarded the very old racist lesson that had first been taught to slaves: that African Americans were supposed to love the US and consider it the world’s greatest country, no matter how they were treated. On top of his rejection of American exceptionalism, Wright had the audacity to preach that American “terrorism” abroad had helped bring on the tragic events of 9/11. To put it lightly, Americans everywhere were livid.

When Obama’s flippant characterisations of Wright as a fraught “old uncle” did not calm Americans down, Obama decided to address the controversy. On 18 March 2008, he stepped into the spotlight and gave a “race speech,” entitled A More Perfect Union, at Philadelphia’s National Constitutional Center. Having taught constitutional law, worked in civil rights law, and overseen successful political campaigns (including his current campaign, which analysts were already regarding as masterful), Obama could easily be regarded as an expert on many things: constitutional law, civil rights law, Chicago politics, Illinois politics, campaigning, and race and politics.

Obama dismissed Jeremiah Wright’s “profoundly distorted view”, but courageously refused to totally disown Wright. And then he opened his general lecture on race, explaining that socioeconomic racial inequities stemmed from the history of discrimination. From this firm anti-racist opening, he pivoted to the consensus racist theory of the “pervasive achievement gap”, to the disproven racist theory of “the erosion of black families” that “welfare policies … may have worsened”, and to the unproven racist theory that racial discrimination had bequeathed blacks a “legacy of defeat”. According to Obama, this “legacy of defeat” explained why “young men and, increasingly, young women” were “standing on street corners or languishing in our prisons”. He ignored the fact that this population was facing some of the nation’s highest unemployment and policing rates.

Those anti-racists like Jeremiah Wright, their “anger is not always productive”, Obama continued. “Indeed, all too often it distracts attention from solving real problems; it keeps us from squarely facing our own complicity within the African American community in our condition.”

It was a classic assimilationist retort: calling anti-racists “angry” for truly believing in racial equality, for not seeing anything wrong with black people, and for seeing everything wrong with discrimination. Like civil rights activists WEB Du Bois and Martin Luther King before him, Obama lumped these “angry” anti-racists in with angry, anti-white cynics to discredit them and distinguish himself from them. But Du Bois and King ultimately had to ward off the same “angry” and anti-white labels they had helped to produce. And now, Obama was doing the same thing, unaware that he was reproducing a label that his opponents would stamp on to him whenever and wherever he uttered another anti-racist word after this speech.

Facebook Twitter Pinterest Barack Obama’s 2008 speech at the National Constitution Center in Philadelphia

Obama encouraged African Americans to fight discrimination, take personal responsibility, be better parents and end the “legacy of defeat”. Obama did not offer any childrearing or psychological lessons for the presumably parentally and psychologically superior white Americans. He merely asked them to join him on the “long march” against racial discrimination – “not just with words but with deeds”. He left the Philadelphia platform on 18 March 2008, as he began, expressing the half-truthful analogy of continuous racial progression. “This union may never be perfect,” he said, “but generation after generation has shown that it can always be perfected.”

Segregationist and anti-racist critiques were drowned out by the eruption of praise across the ideological aisle. MSNBC political analyst Michelle Bernard framed it as “the best speech and most important speech on race that we have heard as a nation since Martin Luther King’s ‘I Have a Dream’ speech”. And it was not just Democrats who were fawning. Prominent Republicans – everyone from presidential candidates Mike Huckabee and John McCain to the Bush administration’s Condoleezza Rice and Colin Powell, and the Clintons’ old foe, Newt Gingrich – were also praising the speech.

Meanwhile, Republican producers of racist ideas got down to business, demanding to see Obama’s birth certificate, questioning whether Barack Hussein Obama was really an American, and suggesting that only real Americans, who were white like McCain, could live in the White House of the United States. The Obama campaign released a scanned copy of his US birth certificate, but the rumours of Obama being born in Kenya or some Islamic anti-American nation did not suddenly go away. They were not started out of ignorance, so why would they go away out of knowledge?

On 4 November 2008, a 64-year-old recently retired professor cast a vote for a major political party for the first time in her voting life. She had retired from academia, but not from her very public activism of four decades. She was still travelling the country trying to rouse an abolitionist movement against prisons. In casting her vote for Democrat Barack Obama, Angela Davis joined roughly 69.5 million Americans. But more than voting for the man, Davis voted for the grassroots efforts of the campaign organisers, those millions of people demanding change.

When the networks started announcing that Obama had been elected the 44th president of the United States, happiness exploded from coast to coast, and from the US around the anti-racist world. Davis was in the delirium of Oakland, California. People she did not know came up and hugged her as she walked along the streets. She saw people singing and dancing. And the people Angela Davis saw and all the others around the world who were celebrating were not enraptured from the election of an individual; they were enraptured by the pride of the victory for black people, by the success of millions of grassroots organisers, and because they had shown all those disbelievers, who had said that electing a black president was impossible, to be wrong. Most of all, they were enraptured by the anti-racist potential of a black president.

Behind the scenes of exploding happiness that November night and over the next few weeks, was the exploding fury of hate attacks on black people. The establishment was working overtime to take down some of the colour-blind rhetoric that had prevented consumers from seeing discrimination for a decade. It was working to put up something better: a portrait of America conveying that there was no longer any need for protective or affirmative civil rights laws and policies – and no longer any need to ever talk about race. “Are we now in a post-racial America? … Is America past racism against black people?” John McWhorter, professor at Columbia, asked in Forbes weeks after the election. “I say the answer is yes.”

When will the day arrive when black lives matter to Americans? It depends largely on what anti-racists do – and on the strategies they use to stamp out racist ideas. Racial reformers have customarily requested or demanded that Americans, particularly white Americans, sacrifice their own privileges for the betterment of black people. And yet, this strategy is based on one of the oldest myths of the modern era, a myth continuously produced and reproduced by racists and anti-racists alike: that racism materially benefits the majority of white people, that white people would lose and not gain in the reconstruction of an anti-racist America.

It has been true that racist policies have benefited white people in general at the expense of black people (and others) in general. That is the story of racism, of unequal opportunity in a nutshell. But it is also true that a society of equal opportunity, without a top 1% hoarding the wealth and power, would actually benefit the vast majority of white people much more than racism does.

Facebook Twitter Pinterest A Black Lives Matter protest in Minnesota last month. Photograph: Eric Miller/Reuters

Anti-racists should stop connecting selfishness to racism, and unselfishness to anti-racism. Altruism is wanted, not required. Anti-racists do not have to be altruistic. Anti-racists do not have to be selfless. Anti-racists merely have to have intelligent self-interest, and to stop consuming those racist ideas that have engendered so much unintelligent self-interest over the years. It is in the intelligent self-interest of middle- and upper-income blacks to challenge the racism affecting the black poor, knowing they will not be free of the racism that is slowing their socio-economic rise until poor blacks are free of racism.

It is in the intelligent self-interest of white Americans to challenge racism, knowing they will not be free of sexism, class bias, homophobia and ethnocentrism until black people are free of racism. The histories of anti-Asian, anti-Native and anti-Latina/o racist ideas; the histories of sexist, elitist, homophobic, and ethnocentric ideas: all sound eerily similar to this history of racist ideas, and feature some of the same defenders of bigotry in America. Supporting these prevailing bigotries is only in the intelligent self-interest of a tiny group of super-rich, Protestant, heterosexual, non-immigrant, white, Anglo-Saxon males. Those are the only people who need to be altruistic in order to be anti-racist. The rest of us merely need to do the intelligent thing for ourselves.

Historically, black people have by and large worked out that the smartest thing we could do for ourselves is to promote the notion of uplift. Beginning around the 1790s, abolitionists urged the growing number of free blacks to exhibit upstanding behaviour in front of white people, believing they would thereby undermine the racist beliefs behind slavery. Black people would acquire “the esteem, confidence and patronage of the whites, in proportion to your increase in knowledge and moral improvement,” as the abolitionist William Lloyd Garrison lectured free blacks in the 1830s.

History shows not only that uplift has failed, but that, generally speaking, the opposite of its intended effect has occurred. Racist Americans have routinely despised those black Americans the most who uplifted themselves, who defied those racist laws and theories that individuals employed to keep them down. So upwardly mobile black people have not persuaded away racist ideas or policies. Quite the contrary. Uplift has brought on the progression of new racist policies and ideas after blacks broke through the old ones.

Everyone who has witnessed the historic presidency of Barack Obama – and the historic opposition to him – should now know full well that the more black people uplift themselves, the more they will find themselves on the receiving end of a racist backlash. Uplift, as a strategy for racial progress, has failed. Black individuals must stop worrying about what other people may think about them. Individual blacks are not race representatives. They are not responsible for those Americans who hold racist ideas. Black people need to be their imperfect selves around white people, around each other, around all people. Black is beautiful and ugly, intelligent and unintelligent, law-abiding and law-breaking, industrious and lazy – and it is those imperfections that make black people human, make black people equal to all other imperfectly human groups.

The other major strategy that racial reformers have used is educational persuasion. As a strategy for racial progress, educational persuasion has failed, because it has been predicated on the false construction of the race problem: the idea that ignorance and hate lead to racist ideas, which lead to racist policies. In fact, self-interest leads to racist policies, which lead to racist ideas, leading to all the ignorance and hate.

Racist policies were created out of self-interest. And so, they have usually been voluntarily rolled back out of self-interest. The popular and glorious version of history saying that abolitionists and civil rights activists have steadily educated and persuaded away American racist ideas and policies sounds great. But it has never been the complete story, or even the main story. Politicians passed the civil and voting rights measures in the 1860s and the 1960s primarily out of political and economic self-interest – not an educational or moral awakening. And these laws did not spell the doom of racist policies. The racist policies simply evolved. There has been a not-so-glorious progression of racism, and educational persuasion has failed to stop it, and Americans have failed to recognise it.

History is clear. Sacrifice, uplift, persuasion and education have not eradicated, are not eradicating, and will not eradicate racist ideas, let alone racist policies. Power will never sacrifice self-interest, cannot be educated away from its self-interest. Those who have the power to abolish racial discrimination have not done so thus far, and they will never do so as long as racism benefits them in some way.

Facebook Twitter Pinterest Activist Angela Davis, who cast her first-ever vote for a major political party in supporting Obama in 2008. Photograph: Theo Wargo/Getty Images

I am certainly not stating that there are no Americans in positions of power who have ever tried to end racial disparities in their sphere of influence. But these courageous anti-racist powerbrokers are more the exception than the rule. As Americans have discarded old racist ideas, new racist ideas have constantly been produced for their consumption. That’s why the effort to educate and persuade away racist ideas has been a never-ending affair in America. That’s why education will never bring into being an anti-racist America.

To undermine racial discrimination, Americans must focus their efforts on those who have the power to undermine racial discrimination. History has shown that those Americans who have had the power to undermine racial discrimination have rarely done so. They have done so, however, when they realised that eliminating some form of racial discrimination was in their interest, much as Abraham Lincoln chose to end slavery to save the union. They have also conceded to anti-racist change as a better alternative than the disruptive, disordered, politically harmful and/or unprofitable conditions that anti-racist protesters created.

The most effective protests have been fiercely local; they are protests started by anti-racists focusing on their immediate surroundings: their blocks, neighbourhoods, schools, colleges, jobs and professions. These local protests have then become statewide protests, and statewide protests have then become national protests, and national protests have then become international protests. But it all starts with one person, or two people, or tiny groups, in their small surroundings, engaging in energetic mobilisation of anti-racists into organisations; and chess-like planning and adjustments during strikes, occupations, insurrections, campaigns and fiscal and bodily boycotts, among a series of other tactics to force power to eradicate racist policies. Anti-racist protesters have created positions of power for themselves by articulating clear demands and making it even clearer that they will not stop – and that policing forces cannot stop them – until their demands are met.

But protesting against racist policies can never be a long-term solution to eradicating racial discrimination in America. Just as one generation of powerful Americans could decide or be pressured by protest to end racial discrimination, when the conditions and interests change, another generation could once again encourage racial discrimination. That’s why protesting against racist power has been a never-ending affair in America.

Protesting against racist power and succeeding can never be mistaken for seizing power. Any effective solution to eradicating American racism must involve Americans committed to anti-racist policies seizing and maintaining power over institutions, neighbourhoods, counties, states, nations – the world. It makes no sense to sit back and put the future in the hands of people committed to racist policies, or people who sail with the wind of self-interest. An anti-racist America can only be guaranteed if principled anti-racists are in power, and then anti-racist policies become the law of the land, and then anti-racist ideas become the common sense of the people, and then the anti-racist common sense of the people holds those anti-racist leaders and policies accountable.

And that day is sure to come. No power lasts for ever. There will come a time when Americans will realise that the only thing wrong with black people is that they think something is wrong with black people. There will come a time when racist ideas will no longer obstruct us from seeing the complete and utter abnormality of racial disparities.

There will come a time when we will love humanity, when we will gain the courage to fight for an equitable society for our beloved humanity, knowing, intelligently, that when we fight for humanity, we are fighting for ourselves. There will come a time. Maybe, just maybe, that time is now.

Main illustration by Christophe Gowans

Adapted from Stamped from the Beginning: The Definitive History of Racist Ideas in America by Ibram X Kendi, published by Bodley Head on 6 July. Buy it for £16.14 at bookshop.theguardian.com

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.