As Ta-Nehisi Coates began his book tour for We Were Eight Years in Power: An American Tragedy, he had a telling exchange with The Late Show’s Stephen Colbert.

“You’ve had a hard time in some interviews expressing a sense of hope in this country,” Colbert said. “Do you have any hope tonight for the people out there, about how we could be a better country, we could have better race relations, we could have better politics?”

“No,” Coates replied. "But I’m not the person you should go to for that. You should go to your pastor. Your pastor provides you hope. Your friends provide you hope.”

This wasn’t the answer Colbert was looking for. “I’m not asking you to make shit up,” he said. “I’m asking if you personally see any evidence for change in America.”

“But I would have to make shit up to actually answer that question in a satisfying way,” Coates shot back.

Colbert is not alone in his request, or in his frustration. It speaks to the quasi-prophetic space Coates occupies in American life that his pessimism grates so deeply on so many, that the central struggle over his work is not its rightness, or its righteousness, but whether it leaves too little space for sunbeams. Coates freely admits that he owes his career to Barack Obama's political rise, to perhaps the most optimistic politician in American life, and yet he refuses to pledge allegiance to hope or place any confidence in change. Indeed, his understanding of those terms puts him at odds with the American consensus.

We Were Eight Years in Power is an unusual book. It collects nine of Coates's Atlantic essays, one from each year of Obama's presidency, and one from its savage aftermath. Connecting the essays are lyrical, autobiographical reflections that situate the work in Coates's daily life, that track his steady evolution as a writer and a thinker.

The most interesting thread of these reflections traces Coates's slow loss of hope, the rising recognition, long before Donald Trump won power, that Obama's presidency would not have a happy ending. The question Coates poses, to himself and to us, is this: What does it mean to be hopeful about race in America?

There was a time when Coates believed in hope and change, or at least wanted to believe in it. "It was hard not to reassess yourself at, say, the sight of John Patterson, the man who'd 'out-niggered' George Wallace to become governor of Alabama in 1959, endorsing Obama," he writes. But then, in quick succession, came Shirley Sherrod, and the humiliation of the “beer summit," and the reaffirmation, for Coates, of "the great power of white innocence — the need to believe that whatever might befall the country, white America is ultimately blameless."

Coates is not a writer who grasps for easy answers. He does not condemn Obama for firing Sherrod, or for placating the police officer who had arrested Henry Louis Gates Jr. on his own porch by inviting him for a drink in the Rose Garden. "Obama was the first black president of a majority-white country,” he writes. “He should've feared white innocence!"

Everything had changed, and not enough had changed. It is in moments like these that you see Coates diverge from his critics. There are two ways of looking at the beer summit. It took place on the lawn of the White House, and the occupant of the White House was black. Hope. But even a black president of the United States still had to genuflect before white America's fear of black men, and its insistence that that fear is innocent and valid. Despair.

Americans venerate progress. Our national mythos is of a perfecting union, a country always striving to come closer to its ideals. To deny that there is hope is to deny that America is getting steadily better, and it is folly to deny that America is a better, fairer, more just country today than it was 50 years ago, or 100 years ago. This is the position of Coates's critics: There is progress, and therefore there is hope. This is what Colbert asked for when he asked for hope: evidence that we could be better, that we were becoming better. This is what New York magazine’s Jonathan Chait accused Coates of missing, saying he “defines out of existence the very possibility of steady progress.”

Reading Coates, I do not believe hope, for him, is synonymous with progress. Hope is prediction. It is about ultimate levels, not current trends. To be hopeful about race in America is not to say that slowly things will become less bad. It is to say that they will become good, equal, just. To be hopeful is to believe that America will one day embody its ideals, that it will atone for its past. Coates quotes Malcolm X, who said, "You don't stick a knife in a man's back nine inches and then pull it out six inches and say you're making progress."

There is a paragraph in Coates's book that I have read and reread. It is, to me, the clearest distillation of his worldview and its power. I do not think there is any doubt that this paragraph is true. I also do not think it is possible to live inside its truth and feel very hopeful:

Any fair consideration of the depth and width of enslavement tempts insanity. First conjure the crime — the generational destruction of human bodies — and all of its related offenses — domestic terrorism, poll taxes, mass incarceration. But then try to imagine being an individual born among the remnants of that crime, among the wronged, among the plundered, and feeling the gravity of that crime all around and seeing it in the sideways glances of the perpetrators of that crime and overhearing it in their whispers and watching these people, at best, denying their power to address this crime and, at worst, denying that any crime had occurred at all, even as their entire lives revolve around the fact of a robbery so large that it is written in our very names.

Though America may improve, its debts will never be repaid, its ideals will never be reached, the barest definition of justice will never be attained. It was, Coates says, his seminal article on reparations that crystallized this knowledge. "The reparations claim was so old, so transparently correct, so clearly the only solution, and yet it remained far outside the borders of American politics. To believe anything else was to believe that a robbery spanning generations could somehow be ameliorated while never acknowledging the scope of the crime and never making recompense. And yet that was the thinking that occupied mainstream American politics."

Here, again, you see Coates's insistence that mere progress cannot be the measure of hope. He had written an Atlantic cover story that set the entire country talking about reparations, that forced at least an intellectual reckoning with the idea and its unsparing logic. The article made him a celebrity, a “public intellectual." That’s progress, and for many, that would be hope. But no matter how sound his argument, reparations were no likelier to come to fruition the day after he published his article than the day before. Progress isn’t enough.

For Coates, progress can, and likely will, coexist with deep injustice and a society ordered around, and constantly rationalizing, its crimes. The villains will not be punished, and the victims, many of them dead, will never be made whole. This is not just American truth. It is a cosmic truth, seen across nations and across times. "Nothing in the record of human history argues for a divine morality, and a great deal argues against it," he writes. "What we know is that good people very often suffer terribly, while the perpetrators of horrific evil backstroke through all the pleasures of the world."

Last week, I interviewed Coates for my podcast. I asked him to describe the world in which justice had been done, in which equality had been achieved, in which hope was merited. "We have a 20-to-1 wealth gap," Coates replied. "Every nickel of wealth the average black family has, the average white family has a dollar. What is the world in which that wealth gap is closed? What happens? What makes that possible? What does that look like? What is the process?"

Even imagining that world, Coates makes ample space for tragedy. When he tries to describe the events that would erase America's wealth gap, that would see the end of white supremacy, his thoughts flicker to the French Revolution, to the executions and the terror. "It's very easy for me to see myself being contemporary with processes that might make for an equal world, more equality, and maybe the complete abolition of race as a construct, and being horrified by the process, maybe even attacking the process. I think these things don't tend to happen peacefully."

For Coates, even hope can be covered in blood.