It is no surprise that the election of the first black president of the United States would occasion much thinking, writing and talking about the subject of race in America. An event that many did not think would happen in their lifetimes, happened: a man of African descent and – this may have been more culturally important – his black wife and children resided in the White House as the nation’s “first family”. President Barack Obama’s portrait would hang in government offices across the country, and in embassies around the world. He would be the commander-in-chief of the country’s armed services.

How proud this made Americans of all races. For black people, who had seen the rules of the game rigged against them in the most immoral ways – slavery and Jim Crow, and their aftermath – having a black man compete for and win the greatest prize in politics was beyond exhilarating. Yes We Can! That phrase, the Obama campaign’s insistent motto, also tapped into the desires of many of Obama’s white supporters who wished to produce evidence that there had indeed been racial progress in the country, including some who may have had a few doubts about the one-term senator with the “non-American” sounding name. Even his defeated opponent, Senator John McCain, took note of the historical significance of Obama’s victory as a praiseworthy thing. A majority of the electorate wanted America to “do it”; to overcome – in this particular way – all the racially-based limitations that had for centuries made the idea of a black president unthinkable. Countries across the globe, themselves not even close to doing anything like it, expressed surprise that Americans had done it, but joined the chorus of praise.

At the same time, how galling it was for the not insignificant number of white Americans who fervently believed that the US began as a country for white people, and should forever remain so. The president of the United States serves as a symbol of the nation; America’s face and voice to the world. All the reasons why many saw Obama’s election as evidence of the country’s endless capacity for adjustment and renewal, an occasion for pride, were for others evidence of America’s degradation, a source of intolerable shame and anger. Something had to be done. What was done, Ta-Nehisi Coates says in We Were Eight Years in Power, the book of essays that follows his bestselling and influential Between the World and Me, was to seek to erase with extreme prejudice the effects of the country having lived under a black president by electing the man Coates dubs in the book’s final essay “The First White President” (Trump’s “ideology is white supremacy, in all its truculent and sanctimonious power”).

Coates takes his title from the haunting words of Thomas Miller, a black South Carolinian who had been elected to state office during the years of Reconstruction after the civil war. Black people in South Carolina significantly outnumbered white people and, for a time, dominated the legislature. They had, in fact, as WEB Du Bois showed in his magisterial Black Reconstruction in America, instituted “good Negro government”; the very thing, Du Bois said, whites feared most. In the face of black success, they resorted to lies about the black men who served in office, creating a caricature of politics during Reconstruction that lived in history books and popular culture (for instance, DW Griffith’s film The Birth of a Nation) until well into the 20th century. By the time Miller talked of the achievements of “eight years in power”, in 1895, Reconstruction, the effort to make a new society in the south by bringing the four million African Americans freed after the civil war into full citizenship, was dead.

The south had been “redeemed” for white southerners, who used law and terrorism to recreate a regime of white supremacy in the absence of racially-based legalised slavery. Rather than share in the benefits of black advancement that would have lifted the south overall, white southerners chose to turn back the clock to the time when their superiority was unquestioned. Even poor white citizens who could have joined their black peers to shake the economic hierarchy that kept white elites in charge and non-elite whites near the bottom of the heap (just above blacks), in atavistic self-defence mode, opted for racial solidarity rather than economic advancement. The plaintive cry heard during the Obama years, “We want our country back”, was eerily familiar. In choosing this title, Coates makes plain his view that, post-Obama, the US is living under a nationalised form of a redemption government.

What are the characteristics of such a moment? Coates answers with essays first published in the Atlantic that range across politics (Malcolm X, Michelle Obama), culture (Bill Cosby) and history: “For most of American history, our political system was premised on two conflicting facts – one, an oft-stated love of democracy; the other, an undemocratic white supremacy inscribed at every level of government.” These essays are introduced by shorter “Notes” that contextualise the older pieces, and track the eight years of Obama’s presidency; Notes from the First Year, Notes from the Second Year and so on.

The most famous of the essays reproduced, the one that can be said to have put Coates on the map just before the phenomenon of Between the World and Me, is “The Case for Reparations” (recompense for historical crimes against African Americans). With this, Coates, who had been labouring for years building a following on his Atlantic blog, achieved what he calls in this book “writer fame”, not to be confused, he quickly adds, with “George Clooney fame”. But even this more modest level of fame disturbs him. He became a known quantity, interrupted on the streets and in “the café where I regularly wrote”.

While heartened by his great success, Coates indicates that it also perplexes him. Fame and influence are double-edged swords: they allow him to make a difference in the world, even as they intrude on his sense of himself and his writing, usually a solitary enterprise that stokes self-absorption. Passages in “Notes from the Sixth Year”, in particular, reveal what sets Coates apart and has made him so successful. He is remarkably open with his readers about his conflicts. This may come from his years of interacting with commenters on his blog, which makes him attuned to his readers and, to a degree, to trust them. Other commentators make the mistake of thinking that to accept the authorial and authoritative voices is to banish uncertainty: to be sure about some things requires appearing sure about everything. Coates ventures forth with certainty about many matters, while being open to questioning himself at points, and doing so in full view of his readers.

No non-academic writer today has a keener sense of the relevance of history to the problems about which he writes. Very significantly, Coates’s approach to history does not appear to be purely instrumental. One senses a genuine interest in, and curiosity about, the ways in which historical forces, always subject to contingencies, have moved us to the place where we presently stand. To say that this could only be so, given that his most consistent topic – race in present-day America – is ineluctably tied to history, is to miss why he has become such a powerful voice. Many write about race and history but, more often than not, with history as a garnish rather than an inextricable part of the main course. Details and facts, not generalities, matter. And to get details, one must do, and present, research, which Coates does in a manner accessible to the general public.

Ta-Nehisi Coates. Photograph: Startraks Photo/REX Shutterstock

Coates came to history relatively recently, and now has the zeal of the convert – in the best sense of that phrase. He confesses that he had not known the true history of slavery, the civil war and Reconstruction, did not know that the war was “a spectacular chapter in a long war that was declared when the first Africans were brought chained to American shores”. He has learned as his hero, and mine, James Baldwin, wrote: “The great force of history comes from the fact that we carry it within us, are unconsciously controlled by it in many ways, and history is literally present in all that we do.” Whatever one thinks about the issue of reparations, Coates’s treatment of the subject was so effective because of his deft use of scholarship, a feature shared in common with nearly all of the essays in this book. His is seldom, if ever, strictly an appeal to emotion or an invocation of morality; though both passion and morality are important to his presentations. Instead, the intellectual weight of the reparations piece, especially, is bolstered by Coates’s skilful deployment of the work of scholars writing over a number of fields who had delved more deeply into the relevant topics than any one journalist could.

We Were Eight Years in Power is not quite the opposite of an upbeat book. No one with a true historical sense could tie up the stories of America told in these essays with a tidy assurance that we shall overcome. Coates knows that we may not. The odds may even be against it. But he says something near the very last page that, again, shows his firm grasp of one of the most important lessons of history. Nothing was “inevitable” about the outcome of the last election in the United States, and we have “not yet” reached “the end of history”. The American story will continue, for the foreseeable future, at least. How that story unfolds will be a matter of the choices we make.

Annette Gordon-Reed’s The Hemingses of Monticello: An American Family won the Pulitzer prize for history. Her latest book is Most Blessed of the Patriarchs: Thomas Jefferson and the Empire of the Imagination, written with Peter S Onuf.

• We Were Eight Years in Power: An American Tragedy is published by Jonathan Cape. To order a copy for £14.44 (RRP £16.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.