Ta-Nehisi Coates is short on sleep. He did five interviews yesterday to promote his new book, We Were Eight Years in Power: An American Tragedy. Today there was another at 7am, then surgery “to get a little thing removed” from his neck. As his tall frame appears in the doorway of an office at his New York publisher, a bandage is visible above the collar of his blue suit jacket.

Coates is friendly but fatigued and yawns several times during the course of our conversation. Some questions animate him and he digs deep with evident passion; others elicit a brief “I don’t know”. The interview doesn’t always flow. But even on an off-day, Coates, 42, is more compelling than almost any other public voice about the state we’re in. The New York Times described him as “the pre-eminent black public intellectual of his generation”. The novelist Toni Morrison compared him to James Baldwin. He emerged as the equivalent of poet laureate during Barack Obama’s presidency, chronicling the spirit of the age. If anything, the advent of Trump has pushed his stock higher. Coates admits it is “tremendously irritating” to be in constant demand by the media, as if he is sole spokesman for African American affairs.

But he does have much to say about Trump and the divided states of America. His book is a collection of eight essays he published during Obama’s eight years in office plus new material, including an epilogue entitled “The First White President”, in which he contends that Trump’s ability to tap the ancient well of racism was not incidental but fundamental to his election win. Many people have called Trump a racist or white supremacist, but Coates has the rare ability to express it in clear prose that combines historical scholarship with personal experience of being black in today’s America.

Halifu Osumare, director of African American and African Studies at the University of California, says: “Ta-Nehisi Coates has done his homework, including much self-reflection. He clearly knows his literary forerunners – [Richard] Wright, Baldwin and Morrison, yet he speaks as a 21st-century writer. He eloquently conflates the personal, political and the existential, while telling it like it is.”

Certainly, in contrast with other commentators, Coates has no qualms about stating that the White House is occupied by a white supremacist (a term he does not apply to other Republicans, such as George HW Bush or George W Bush). He lays out evidence that Trump, despite his upbringing in liberal New York, has a long history of racial discrimination. There was the 1973 federal lawsuit against him and his father for alleged bias against black people seeking to rent at Trump housing developments in New York. Trump took out ads in four daily newspapers calling for the reintroduction of the death penalty in 1989 after five African American and Latino teenagers were accused of assaulting and raping a white woman in Central Park. Even after the five were cleared by DNA evidence, he continued to insist: “They admitted they were guilty.”

I’d felt like a failure all my life – ‘college dropout’ means something different when you’re black

He was once quoted as saying: “Black guys counting my money! I hate it. The only kind of people I want counting my money are short guys that wear yarmulkes every day.” More recently, Trump was a leading proponent of the “birther” movement, pushing the conspiracy theory that Obama was not born in the US and therefore an illegitimate president. While running for president, he said that a judge of Mexican heritage would be unfair to him in a court case because he was a “hater” and a “Mexican”. In one interview, Trump refused to condemn the Ku Klux Klan (he subsequently blamed a faulty earpiece).

In his epilogue, Coates writes: “To Trump, whiteness is neither notional nor symbolic, but the very core of his power. In this, Trump is not singular. But whereas his forebears carried whiteness like an ancestral talisman, Trump cracked the glowing amulet open, releasing its eldritch energies.”

Since then, there has been a white supremacist march in Charlottesville, Virginia, in which a civil rights protester was killed, prompting Trump’s comment that there were “very fine people on both sides”. Today, Coates adds the president’s visit to hurricane-hit Puerto Rico to Trump’s charge sheet: “Just yesterday, he goes to a part of the United States that’s been devastated by a natural disaster and throws toilet paper out to the crowd like they’re peasants or something. There are people in this country who will not be happy until Donald Trump is literally executing a lynching before they’ll use that term [white supremacist]. I’m not going to play around; let’s call things what they are.”

Last month Trump was at it again, condemning American football players who “take the knee” during the national anthem to make a statement against racial injustice. Throwing red meat to his base at a rally in Alabama, he called on team owners to fire them and to say: “Get that son of a bitch off the field right now.” The protest was started last year by Colin Kaepernick of San Francisco 49ers. Coates reflects: “Kaepernick’s protest has been very successful. I really appreciate the fact that he’s been giving away money to organisations; he pledged to give away a million dollars and he’s been doing it.”

But Trump used his familiar tactics to divert and distract, kicking up bitter divisions around the anthem, the military, how much sportsmen earn, the meaning of patriotism and, of course, himself. Amid the media storm, it was easy to forget what the original protest was about. “The police brutality element has been lost, but I think that is a danger that all protests face,” Coates says. “At some point, you’re always co-opted, successful protests especially. It happened in the civil rights movement. People forget that the 1963 march [on Washington] was for jobs: that somehow got lost, and it became this warm, fuzzy thing [now best known for Martin Luther King’s “I have a dream” speech].”

Donald and Melania Trump visit Puerto Rico two weeks after Hurricane Maria. Photograph: Reuters/Jonathan Ernst

The notion that all these issues would be resolved by Obama was always fanciful. Even so, Coates was swept up in the euphoria with millions of others in 2008 when the US elected its first black president. Had the nation – whose founding fathers were slave owners, and where today African Americans are incarcerated at more than five times the rate of whites – truly changed? Coates admits he took his eye off the ball. The racial backlash was coming.

“The symbolic power of Barack Obama’s presidency – that whiteness was no longer strong enough to prevent peons taking up residence in the castle – assaulted the most deeply rooted notions of white supremacy and instilled fear in its adherents and beneficiaries,” he writes. “And it was that fear that gave the symbols Donald Trump deployed – the symbols of racism – enough potency to make him president, and thus put him in position to injure the world.”

Trump did not come out of nowhere; he was the logical conclusion of years of racial dog whistles from the Republican party, which has sought to suppress the black vote through spurious claims of cracking down on fraud. Coates recounts: “Throughout his eight years in office, Barack Obama endured a campaign of illegitimacy waged either by pluralities or majorities of the Republican party. Donald Trump rooted his candidacy in that campaign. It’s fairly obvious.

“His first real foray out again as a political candidate was into birthism [Trump began questioning Obama’s birthplace in TV interviews in 2011], and a lot of people dismissed birthism as just something cranks do and we don’t have to deal with. That was a huge mistake: it underrated the long tradition of denying black people their citizenship and basic rights. That was what this was piggybacking off of, so it’s not a mistake that he started there and then became president at all.”

Coates does not make the claim that all 63 million people who voted for Trump are white supremacists; but they were, he points out, willing to hand the government over to one. It was an astonishingly reckless act. Coates’s book is a wake-up call to white America, a holding to account. “So this question, is everyone who voted for Trump a racist? This misses the point. Did everyone in Nazi Germany believe all the stereotypes about Jews? Of course not. It’s beside the point.

“When France deported its Jews, did everyone in France believe all this stuff? No, but that’s beside the point. Looking the other way has consequences and you might not be a racist or a white supremacist or a bigot, but if you voted for Trump, you looked the other way, you said it’s fine to have that in the White House, and a substantial number of Americans felt that way. That’s a statement.”

The San Francisco 49ers ‘take the knee’ during the national anthem in October 2016, as a statement against racial injustice.

Coates also takes issue with the media’s obsession with the white working class as a bloc that turned its back on Democrats and defected to Trump. His book challenges politicians and journalists who make earnest defences of Trump-voting communities as “good people” not motivated by bigotry. Countless articles and books such as Hillbilly Elegy by JD Vance, a memoir about growing up in the white underclass, have been studied as key to understanding the despair of small towns left behind by globalisation. Are they missing the point? Is class secondary to race?

“It’s not like most working-class people voted for Donald Trump; they did not,” Coates says. “Most white working-class people voted for Donald Trump and the through line that you find is whiteness, not class and not gender. It’s not like he only got men; he got a majority of white women too. So if you look at categories of white people you find Trump being dominant among them, in part because of the appeal he made, but also in part because the Republican party has effectively become in this country the party of white people.

“What’s happening is the white working class is being used as a kind of signpost tool… There is some effort not just to absolve white working people, but to absolve whiteness because here’s the deal: ‘Oh, it’s fine that white working-class people and white poor people voted for Donald Trump because over the past 30 years they’ve had unmet expectations. And it’s also fine that rich white people voted for Donald Trump because of tax cuts.’ Come on: everybody gets off the hook.”

And yet many senators, including Bernie Sanders, whom Coates supported in the Democratic primary, Al Franken and Elizabeth Warren have argued that a generation of economic stagnation is real, fuelling anger that led some voters to throw a grenade at the Washington establishment. Middle- and working-class parents are frustrated that their children will not have the same opportunities they did. Trump’s defeated opponent Hillary Clinton writes in her new memoir: “After studying the French Revolution, De Tocqueville wrote that revolts tend to start not in places where conditions are worst, but in places where expectations are most unmet.”

To that, Coates responds: “Those expectations are built on being white. People say that as though it’s indivisible from the idea of race. You want to talk about unmet expectations? Black folks have been dealing with that since we got here, so the notion that, ‘My child isn’t going to have it as good as me, so that therefore gives me the right to vote for someone who conducts diplomacy with a rogue nuclear state via Twitter’ – that don’t work. Bottom line is, a significant number of people in this country have tolerance for bigotry. No one, I don’t think, can act like they didn’t know. You know I think [Trump’s racist] comments were well reported and America just decided it was OK.”

When white voters make bad decisions, Coates argues, excuses are made; when black voters do it, they get the blame. Coates recalls how the election of Marion Barry as mayor of the District of Columbia [later to be caught on camera smoking crack cocaine] prompted articles suggesting people in the district should lose the right to vote. “So there’s all this kind of rope that’s given, all these excuses allowed when you’re white in this country. But if black people acted that civically irresponsibly, that rope would not be awarded.

“Like you take the opioid crisis and all of the compassion that’s doled out in the rhetoric? Where was that during the crack epidemic in the 1980s? I remember it well. I was in a city where that was going on. Where was all that compassion? Black people aren’t worthy of that. That’s a story that can be created for white people because they’re white, but we don’t get that sort of compassion.”

Democrats are said to be torn between an emphasis on economic justice that aims to win back Trump voters and an emphasis on racial justice that will energise its liberal base. Asked about the future direction of the party, Coates is hesitant: “I don’t know. I shouldn’t answer that.” But after a pause, he weighs in: “Here’s one thing. I don’t think they can get away from talking about race because of the way things are aligned. You’ve got to get to a state like South Carolina or Georgia: these states have large numbers of black and brown voters.”

If Obama was more radical, he wouldn't have become president. He's what the country wanted then

Coates grew up in Baltimore, where Francis Scott Key wrote The Star Spangled Banner and the first residential racial segregation law in any US city was enacted. More recently, it was famous for David Simon’s crime drama The Wire. “I had very little interaction with white people as a kid,” Coates recalls. “I think about what my world looked like as a child, a place that felt fearful, violent, then I’d put on the TV and I’d see that that was not the country at least as it advertised itself. That struck me and I always wanted to know why, what was the difference, why was my house not like Family Ties? That motivated a great deal of my work from the time I was young.”

His father, Paul Coates, was a Vietnam war veteran, Black Panther and voracious collector of books about the history of black struggle. Paul Coates had seven children by four women and was an intellectually inspiring father who also administered beatings. Coates has described him with affection as “a practising fascist, mandating books and banning religion”. The religion ban worked – Coates is an atheist – and so did the books, eventually. In February 2007, Coates, then 31, had just lost his third job in seven years and was trying to stay off welfare. He writes: “I’d felt like a failure all my life – stumbling out of middle school, kicked out of high school, dropping out of college... ‘College dropout’ means something different when you’re black. College is often thought of as the line between the power to secure yourself and your family, and the power of someone else securing you in a prison or grave.”

Married with a young child, he possessed intellectual curiosity and the gift of a wordsmith. He produced an essay about Bill Cosby that caught attention and led to a relationship with the Atlantic magazine, where he is now a national correspondent. His ascent coincided with Obama’s and a new world of possibilities. “It was as if I had spent my years jiggling a key into the wrong lock. The lock was changed. The doors swung open, and we did not know how to act.”

Coates made a splash with a 2014 article for the Atlantic arguing that the US should pay African Americans reparations for slavery. Then, a year later, came Between the World and Me, a rumination on black life and white supremacy, addressed to his teenage son in a letter form that evoked Baldwin’s The Fire Next Time. It argued that the “destruction of black bodies” is not simply a recurring theme of American history but its central premise. It won the National Book Award in nonfiction, sold 1.5m copies around the world and has been translated into 19 languages.

Barack Obama in 2011. ‘one of the smartest people I’ve ever talked to’. Photograph: AP Photo/Jacquelyn Martin)

As his star rose, Coates was invited to the White House. He got to spend time with Obama, whose fundamental optimism in America had convinced him that Trump could not win. He says: “He was tremendously intelligent, one of the smartest people I’d ever talked to, and he was smart in many ways. I met him a few times: one was with a bunch of journalists and he had the ability to address each journalist in their specific area in a very learned way. I thought he was brilliant.”

He reckons “in the main” Obama lived up to the impossibly high expectations of his presidency. “He had an incredible tightrope to walk and it’s difficult, man. You’re the first black president and you’ve got to represent a community, then speak to a larger country at the same time. If he was more radical he wouldn’t have been president. That’s what I’ve come around to: who he was was what the country wanted at that time. He can’t be me; not that he should want to be. But it’s a very different calling.”

Indeed, Coates sees himself as a writer – including of a comic-book series starring superhero the Black Panther – rather than an activist or potential politician. “That’s what I’m supposed to be doing because it’s what calls to me and it’s what I’m good at, what I excel at. I don’t really excel at this other stuff. I’m not a person who’s going to say whatever I have to say to get a coalition together, which is what you have to do in politics. I’m a writer.”

Towards the end of the interview, the questions become longer and Coates’s answers become shorter. He is probably relieved when it’s over, though he is too polite to say so.

Later he is busy tweeting links to articles about gun violence, nuclear war and earthquakes, jokingly chiding their authors for offering no hope. It is a charge with which he is all too familiar. “Our story is a tragedy,” he writes in We Were Eight Years in Power. “I know it sounds odd, but that belief does not depress me. It focuses me. After all, I am an atheist and thus do not believe anything, even a strongly held belief, is destiny... The worst really is possible. My aim is never to be caught, as the rappers say, acting like it can’t happen. And my ambition is to write both in defiance of tragedy and in blindness of its possibility, to keep screaming into the waves – just as my ancestors did.”

• We Were Eight Years in Power by Ta-Nehisi Coates is published by Hamish Hamilton (£16.99). To order a copy for £14.44 go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99