Barack Obama began ignoring structural racism exactly six months into his first term, over a polite beer, in near silence. That long-forgotten White House “beer summit” – the first black American president, his white VP, a white cop and the black professor arrested for “breaking into” his own home – was the start of a pattern for Obama’s increasingly uncomfortable public position on racism: it’s anecdotal, occasionally practiced by misguided but well intentioned people, and best put politely behind us.

To be fair, Obama couldn’t even express empathy about one murdered black child, Trayvon Martin, without white folks going batshit crazy. But six years on, the candidate who said “race is an issue that I believe this nation cannot afford to ignore right now” looks, right now, like a president who thinks he can afford to ignore it.

Sure, he got up there at the UN on Wednesday, while engaged in the age-old tradition of American presidents bombing brown people overseas, and mentioned Ferguson, Missouri, and said that “yes, we have our racial and ethnic tensions”. This, however, is the same Barack Obama whose first public comments on Ferguson, from his Martha’s Vineyard vacation, made him not only seem bored by structural racism but actually made George W Bush look engaged in it for dipping his plane over Katrina-flooded New Orleans.

Where is the first black president’s justifiable anger about racism now that he never has to run for office again? Where is he when another black child, John Crawford, is murdered on camera by police, in five seconds, in the pet-food aisle of a Walmart? What is Obama doing about the prejudice and violence faced by brown people here at home?

As I sat in a New York auditorium Tuesday afternoon, disappointed that my black president had checked out on racism (if he had ever checked in), it became increasingly obvious that Obama has now turned over his public confrontation of racism entirely to another black man: Eric Holder.

The US attorney general may not exactly be the “Obama Anger Translator” dreamed up by Key and Peele, but Holder is Obama’s black id. Whatever fight this administration might have left on systemic racism would be channelled through him. And now that Holder is stepping down, whatever hope might’ve been left in this failed chapter is fading fast.

It had been refreshing to hear Holder, a month after Obama sent him to Ferguson in his place, talk bluntly in a speech here at NYU about how “the Bureau of Prisons currently commands about a third of the Justice Department’s overall budget”; how while “the United States comprises just 5% of the world’s population, we incarcerate almost a quarter of its prisoners”; and, most damning, how ...

... our system has perpetuated a destructive cycle of poverty, criminality and incarceration that has trapped countless people and weakened entire communities – particularly communities of color.

Holder deserves credit for his self-critique of government racism and for pushing reforms, notably in trying to restore felons’ voting rights, to work with Congress on voter ID sanity, to oppose data-driven sentencing, to reduce sentences for low-level drug offenders, to decriminalize HIV exposure, to ease up on legal pot (maybe), and to direct his Department of Justice to investigate the Ferguson police shooting – an example his civil rights department will pick up in the Walmart shooting and, perhaps, beyond.

Such injustices harm all Americans, but black and brown Americans excessively. Obama is fond of saying “a rising tide lifts all boats” and loathe to publicly address specifically racist inequities, like why black drivers are 31% more likely to be stopped by cops than white ones, or why black college students are as likely to get jobs as white high school dropouts.

Saying these things out loud falls on Holder, who, unlike his boss, talked about Ferguson this week as “an African-American man – who has been stopped and searched by police in situations where such action was not warranted”, and who has an “understanding of the mistrust that some citizens harbor for those who wear the badge”.

But how much can Eric Holder get done on racial justice, now that his resignation has arrived? He will stay in office until a suitable replacement can be found, and even if that replacement is another leader of color – like Deval Patrick – taking office in January, we’re talking work on borrowed time here.

Holder proudly said in his speech here that since since “Obama took office, both overall crime and overall incarceration have decreased by approximately 10%”. But he also admitted “the federal prison population has grown by almost 800%” since 1980. That a million black faces are incarcerated is a racist crisis of apartheid proportions, requiring an abolitionist level of activism. There is so much more for Eric Holder to do.

When even the first black president would rather not talk about the more than 2m legal slaves in the US – and a disproportionate number of those slaves are black – you start to realize the small scale of Holder’s proposed reforms. You start to understand Cornel West’s disappointment that Obama had you “looking for John Coltrane and you get Kenny G in brown skin”.

You start to look at Eric Holder, Obama’s point man on race, who will at least address aloud your rage at the very institutional racism the president himself seems afraid to name.

We are at a “moment of decision,” Holder told us in the silent auditorium on Tuesday. “Will we yet again turn a blind eye to the hard truths that Ferguson exposed? Or will we finally accept this mandate for open and honest dialogue?”

Then you start to see just how much Eric Holder’s got left to do, and how little time to do it, and your hope starts to fade. You start to think maybe the moment of decision should have started with Holder’s boss many beers ago, when the commander-in-chief tried to fight racism with geniality and his home brew instead of anger and his bully pulpit.