I have often wondered how I missed the coming tragedy. It is not so much that I should have predicted that Americans would elect Donald Trump. It’s just that I shouldn’t have put it past us. It was tough to keep track of the currents of politics and pageantry swirling at once. All my life I had seen myself, and my people, backed into a corner. Had I been wrong? Watching the crowds at county fairs cheer for Michelle Obama in 2008, or flipping through the enchanting photo spreads of the glamorous incoming administration, it was easy to believe that I had been.

And it was more than symbolic. Barack Obama’s victory meant not just a black president but also that Democrats, the party supported by most black people, enjoyed majorities in Congress. Prominent intellectuals were predicting that modern conservatism – a movement steeped in white resentment – was at its end and that a demographic wave of Asians, Latinos and blacks would sink the Republican party.

Back in the summer of 2008, as Obama closed out the primary and closed in on history, vendors in Harlem hawked T-shirts emblazoned with his face and posters placing him in the black Valhalla where Martin, Malcolm and Harriet were throned. It is hard to remember the excitement of that time, because I now know that the sense we had that summer, the sense that we were approaching an end-of-history moment, proved to be wrong.

It is not so much that I logically reasoned out that Obama’s election would author a post-racist age. But it now seemed possible that white supremacy, the scourge of American history, might well be banished in my lifetime. In those days I imagined racism as a tumour that could be isolated and removed from the body of America, not as a pervasive system both native and essential to that body. From that perspective, it seemed possible that the success of one man really could alter history, or even end it.

I had never seen a black man like Barack Obama. He talked to white people in a new language – as though he actually trusted them and believed in them. It was not my language. It was not even a language I was much interested in, save to understand how he had come to speak it and its effect on those who heard it. More interesting to me was that he had somehow balanced that language with the language of the south side of Chicago. He referred to himself, unambiguously, as a black man. He had married a black woman. It is easy to forget how shocking this was, given the common belief at the time that there was a direct relationship between success and assimilation. The narrative held that successful black men took white wives and crossed over into that arid no-man’s-land that was not black, though it could never be white. Blackness for such men was not a thing to root yourself in but something to evade and escape. Barack Obama found a third way – a means of communicating his affection for white America without fawning over it. White people were enchanted by him – and those who worked in newsrooms seemed most enchanted of all.

But I could see that those charged with analysing the import of Obama’s blackness were, in the main, working off an old script. Obama was dubbed “the new Tiger Woods of American politics”, as a man who wasn’t “exactly black”. I understood the point – Obama was not “black” as these writers understood “black”. It wasn’t just that he wasn’t a drug dealer, like most black men on the news, but that he did not hail from an inner city, he was not raised on chitterlings, his mother had not washed white people’s floors. But this confusion was a reduction of racism’s true breadth, premised on the need to fix black people in one corner of the universe so that white people may be secure in all the rest of it. So to understand Obama, analysts needed to give him a superpower that explained how this self-described black man escaped his assigned corner. That power was his mixed ancestry.

The precise ancestry of a black drug dealer or cop killer is irrelevant. His blackness predicts and explains his crime. He reinforces the racist presumption. It is only when that presumption is questioned that a fine analysis of ancestry is invoked. Frederick Douglass was an ordinary nigger while working the fields. But when he was a famed abolitionist, it was often said that his genius must derive from his white half. Ancestry isn’t even really necessary. My wife, Kenyatta, was the only black girl in her Tennessee “gifted and talented” classes from age six. She could dance and double dutch with the best of them. Her white classmates did not care. “You’re not really black,” they would say. They meant it as a compliment. But what they really meant was to slander her neighbours and family, to reorder the world in such a way that confirmed their status among the master class. And if Obama, rooted in the world of slaves, could rise above the masters, all the while claiming the identity and traditions of slaves, was there any real meaning in being a master at all?

Denying Barack Obama his blackness served another purpose: it was a means of coping with having been wrong. Those of us who did not believe there could be a black president were challenged by the sudden prospect of one. It is easy to see how it all makes sense now – in every era there have been individual black people capable of defying the bonds of white supremacy, even as that same system held the great mass of us captive. I will speak for myself and say that before Obama’s campaign began, the American presidency seemed out of reach. It existed so high in the firmament, and seemed so synonymous with the country’s sense of itself, that I never gave the prospect of a black president much thought.

By the summer of 2008, it was clear that I’d made an error. Two responses were possible: (1) assess that error and reconsider the nature of the world in which I lived; or (2) refuse to accept the error and simply retrofit yesterday’s reasoning to this new reality. The notion that Obama was a “different kind of black” allowed for that latter option and the comfort of being right. But some of us had not wanted to be right. And when we asserted that “America ain’t never letting no nigger be president,” we were not bragging. Instinct warned me against hope. But instinct had also warned me against Obama winning Iowa, and instinct was wrong. And if we had misjudged America’s support for a black man running to occupy the White House, perhaps I had misjudged the nature of my country. Perhaps we were just now awakening from some awful nightmare, and if Barack Obama was not the catalyst of that awakening, he was at least the sign. And just like that, I was swept away, because I wanted desperately to be swept away, and taking the measure of my community, I saw that I was not alone.

There is a notion out there that black people enjoy the sisyphean struggle against racism. In fact, most of us live for the day when we can struggle against anything else. But having been, by that very racism, pinned into ghettos, both metaphorical and real, our options for struggle are chosen long before we are born. And so we struggle out of fear for our children. We struggle out of fear for ourselves. We struggle to avoid our feelings, because to actually consider all that was taken, to understand that it was taken systemically, that the taking is essential to America and echoes down through the ages, could make you crazy. But after Obama’s election it seemed that perhaps there was another way. Perhaps we, as Americans, could elide the terrible history, elide the national crime. Maybe it was possible to fix the problems afflicting black people without focusing on race. Perhaps it was possible to think of black people as a community in disproportionate need, worthy of aid simply because they were Americans in need. Better schools could be built, better healthcare administered, better jobs made available, not because of anything specific in the black experience but precisely because there isn’t. If you squinted for a moment, if you actually tried to believe, it made so much sense. All that was needed for this new theory was a champion – articulate, young, clean. And maybe this new champion had arrived.

That was one way of thinking about things. Here was another. “Son,” my father said of Obama, “you know the country got to be messed up for them folks to give him the job.” The economy was on the brink. The blood of untold numbers of Iraqis was on our hands. Hurricane Katrina had shamed the society. From this other angle, post-racialism and good feeling were taken up not so much out of elevation in consciousness but out of desperation.

It all makes so much sense now. The pageantry, the math, the magazines, the essays heralded an end to the old country with all its divisions. We forgot that there were those who loved that old country as it was, who did not lament the divisions but drew power from them.

Facebook Twitter Pinterest A Confederate flag with the name of US president Donald Trump, North Carolina, May 2017. Photograph: Shannon Stapleton/Reuters

And so we saw postcards with watermelons on the White House lawn. We saw simian caricatures of the first family, the invocation of a “food-stamp president” and his anticolonial, Islamist agenda. These were the fetishes that gathered the tribe of white supremacy, that rallied them to the age-old banner – and if there was one mistake, one reason why I did not see the coming tragedy, why I did not account for its possibilities, it was because, at that point, I had not yet truly considered that banner’s fearsome power.

The opportunity for that consideration came by coincidence. The eight years of Barack Obama bracketed the 150th anniversary of the civil war – America’s preeminent existential crisis. In 1861, believing themselves immersed in a short war, the forces of union thought white supremacy was still affordable. So even in the north the cause of abolition was denounced, and blacks were forbidden from fighting in the army. But the war dragged on, and wallowing in white supremacy amid the increase of dead was like wallowing in pearls amid a famine. Emancipation was embraced. Blacks were recruited and sent into battle. Later they were enfranchised and sent to serve in the halls of government, national and statewide. But in 1876, with the hot war now passed, and the need for black soldiers gone, the country returned to its supremacist roots. “A revolution has taken place by force of arms and a race are disenfranchised,” wrote Mississippi’s Reconstruction-era governor, Adelbert Ames.

They are to be returned to a condition of serfdom – an era of second slavery … The nation should have acted but it was “tired of the annual autumnal outbreaks in the South” … The political death of the negro will forever release the nation from the weariness from such “political outbreaks”. You may think I exaggerate. Time will show you how accurate my statements are.

So there was nothing new in the suddenly transracial spirit that saw the country, in 2008, reaching “for the best part of itself”. It had done so before – and then promptly retrenched in the worst part of itself. To see this connection, to see Obama’s election as part of a familiar cycle, you would have had to understand how central the brand of white supremacy was to the country. I did not. I could remember, as a child, the black nationalists claiming the country was built by slaves. But this claim was rarely evidenced and mostly struck me as an applause line or rhetorical point. I understood slavery as bad and I had a vague sense that it had once been integral to the country and that the dispute over it had, somehow, contributed to the civil war.

But even that partial sense ran contrary to the way the civil war was presented in the popular culture, as a violent misunderstanding, an honourable duel between wayward brothers, instead of what it was – a spectacular chapter in a long war that was declared when the first Africans were brought chained to American shores.

When it comes to the civil war, all of our popular understanding, our popular history and culture, our great films, the subtext of our arguments are in defiance of its painful truths. It is not a mistake that Gone With the Wind is one of the most read works of American literature or that The Birth of a Nation is the most revered touchstone of all American film. Both emerge from a need for palliatives and painkillers, an escape from the truth of those five short years in which 750,000 American soldiers were killed, more than all American soldiers killed in all other American wars combined, in a war declared for the cause of expanding “African slavery”. That war was inaugurated not reluctantly, but lustily, by men who believed property in humans to be the cornerstone of civilisation, to be an edict of God, and so delivered their own children to his maw. And when that war was done, the now-defeated God lived on, honoured through the human sacrifice of lynching and racist pogroms. The history breaks the myth. And so the history is ignored, and fictions are weaved into our art and politics that dress villainy in martyrdom and transform banditry into chivalry, and so strong are these fictions that their emblem, the stars and bars, darkens front porches and state capitol buildings across the land to this day.

The implications of the true story are existential and corrosive to our larger national myth. To understand that the most costly war in this country’s history was launched in direct opposition to everything the country claims to be, to understand that it was the product of centuries of enslavement, which is to see an even longer, more total war, is to alter the accepted conception of America as a beacon of freedom. How does one face this truth or forge a national identity out of it?

For now the country holds to the common theory that emancipation and civil rights were redemptive, a fraught and still-incomplete resolution of the accidental hypocrisy of a nation founded by slaveholders extolling a gospel of freedom. This common theory dominates much of American discourse, from left to right. Conveniently, it holds the possibility of ultimate resolution, for if right-thinking individuals can dedicate themselves to finishing the work of ensuring freedom for all, then perhaps the ghosts of history can be escaped. It was the common theory – through its promise of a progressive American history, where the country improves itself inexorably and necessarily – that allowed for Obama’s rise. And it was that rise that offered me that chance to see that theory for the illusion that it was.

Immersed in my reading, it became clear to me that the common theory of providential progress, of the inevitable reconciliation between the sin of slavery and the democratic ideal, was myth. Marking the moment of awakening is like marking the moment one fell in love. If forced I would say I took my tumble with the dark vision of historian Edmund Morgan’s book American Slavery, American Freedom. Certainly slavery was contrary to America’s stated democratic precepts, conceded Morgan, but in fact, it was slavery that allowed American democracy to exist in the first place. It was slavery that gifted much of the south with a working class that lived outside of all protections and could be driven, beaten and traded into generational perpetuity. Profits pulled from these workers, repression of the normal angst of labour, and the ability to employ this labour on abundant land stolen from Native Americans formed a foundation for democratic equality among a people who came to see skin colour and hair textures as defining features. Morgan showed the process in motion through the law – rights gradually awarded to the mass of European poor and oppressed, at precisely the same time they were being stripped from enslaved Africans and their descendants.

It was not just Edmund Morgan. It was James McPherson. It was Barbara Fields. It was David Blight. Together they guided me through the history of slavery and its cataclysmic resolution. I became obsessed and insufferable. Civil war podcasts were always booming through the house. I’d drag Kenyatta and our son, Samori, to the sites of battles – Gettysburg, Petersburg, the Wilderness – audiobooks playing the whole way. I went to Tennessee. I saw Shiloh. I saw Fort Donelson. I saw Island No 10. At every stop I was moved. The stories of suffering, limbs amputated, men burned alive, the bravery and gallantry, all of it seeped up out of the ground and enveloped me. But something else accompanied this hallowed feeling: a sense that the story, as it was told on these sites, as it was interpreted by visitors – most of them white – was incomplete, and this incompletion was not thoughtless but essential. The tactics of the war were always up for discussion, but the animating cause of those tactics, with but a few exceptions, went unsaid.

Facebook Twitter Pinterest Former slaves working as labourers for the Union war effort at White House Landing, Virginia, 1863. Photograph: Andrew J Russell/Medford Historical Society Col

By then, I knew. The history books spoke where tourism could not. The four million enslaved bodies, at the start of the civil war, represented an inconceivable financial interest – $75bn in today’s dollars – and the cotton that passed through their hands represented 60% of the country’s exports. In 1860, the largest concentration of multimillionaires in the country could be found in the Mississippi River valley, where the estates of large planters loomed.

Any fair consideration of the depth and width of enslavement tempts insanity. First conjure the crime – the generational destruction of human bodies – and all of its related offences – domestic terrorism, poll taxes, mass incarceration. But then try to imagine being an individual born among the remnants of that crime, among the wronged, among the plundered, and feeling the gravity of that crime all around and seeing it in the sideways glances of the perpetrators of that crime and overhearing it in their whispers and watching these people, at best, denying their power to address the crime and, at worst, denying that any crime had occurred at all, even as their entire lives revolve around the fact of a robbery so large that it is written in our very names. This is not a thought experiment. America is literally unimaginable without plundered labour shackled to plundered land, without the organising principle of whiteness as citizenship, without the culture crafted by the plundered, and without that culture itself being plundered.

White dependency on slavery extended from the economic to the social, and the rights of whites were largely seen as dependent on the degradation of blacks. “White men,” wrote Mississippi senator and eventual president of the Confederacy Jefferson Davis, “have an equality resulting from a presence of a lower caste, which cannot exist were white men to fill the position here occupied by the servile race.”

Antebellum Georgia governor Joseph E Brown made the same point: “Among us the poor white laborer is respected as an equal. His family is treated with kindness, consideration and respect. He does not belong to the menial class. The negro is in no sense of the term his equal. He feels and knows this. He belongs to the only true aristocracy, the race of white men. He blacks no master’s boots, and bows the knee to no one save God alone. He receives higher wages for his labor than does the laborer of any other portion of the world, and he raises up his children, with the knowledge that they belong to no inferior caste; but that the highest members of the society in which he lives, will, if their conduct is good, respect and treat them as equals.”

Enslavement provided not merely the foundation of white economic prosperity, but the foundation of white social equality, and thus the foundation of American democracy. But that was 150 years ago. And the slave south lost the war, after all. Was it not the America of Frederick Douglass that had prevailed and the Confederacy of Jefferson Davis that had been banished? Were we not a new country exalting in Martin Luther King Jr’s dream?

I was never quite that far gone. But I had been wrong about the possibility of Barack Obama. And it seemed fair to consider that I might be wrong about a good deal more.

But the same year I began my exploration of the civil war and the same summer I finished American Slavery, American Freedom, Harvard professor Henry Louis Gates was arrested. Gates was returning from a long trip. He was having trouble with the lock on his front door and so was attempting to force his way into his home. Someone saw this and called the police. They arrived and, after an exchange of words, Sgt James Crowley arrested, charged and jailed Gates for disorderly conduct. It caused a minor sensation.

Facebook Twitter Pinterest Henry Louis Gates, Sgt James Crowley and Barack Obama drink beer in the White House garden, July 2009. Photograph: The White House/Getty Images

Commenting on the arrest, Obama asserted that anyone in Gates’s situation would be “pretty angry” if they were arrested in their own home. Obama added that “the Cambridge police acted stupidly.” He then cited the “long history” of “African Americans and Latinos being stopped by law enforcement disproportionately”. I don’t know why I expected this would go over well. I don’t know why I thought this mild criticism from a new president in defence of one of the most respected academics at our country’s most lauded university in a case of obvious but still bloodless injustice might be heard by the broader country and if not agreed with, at least grappled with.

In fact, there would be no grappling. Obama was denounced for having attacked the police, and the furore grew so great that it momentarily threatened to waylay his agenda. The president beat a hasty retreat. He apologised to the police officer, then invited Crowley and Gates to the White House for a beer. It was absurd. It was spectacle. But it cohered to the common theory, it appealed to the redemptive spirit and reduced the horror of being detained by an armed officer of the state, and all of the history of that horror, to something that could be resolved over a beer.

And now the lies of the civil war and the lies of these post-racial years began to resonate with each other, and I could now see history, awful and undead, reaching out from the grave. America had a biography, and in that biography, the shackling of black people – slaves and free – featured prominently. I could not yet draw literal connections, though that would come. But what I sensed was a country trying to skip out on a bill, trying to stave off a terrible accounting.

Adapted from We Were Eight Years in Power: An American Tragedy by Ta-Nehisi Coates, which will be published by Hamish Hamilton on 5 October at £16.99. To buy it for £14.44, got to go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.