When Barack Obama was contemplating a run for the White House his wife, Michelle, asked him what he thought he could accomplish if he won. "The day I take the oath of office," he replied. "The world will look at us differently. And millions of kids across this country will look at themselves differently. That alone is something."

Four months after he was sworn in, at least one kid saw himself differently. It was May 2009 and 5-year-old Jacob Philadelphia had gone with his dad, a black ex-marine, to the Oval Office for a family photograph with the president.

With him were his mum, Roseanne, and his older brother, Isaac, 8. The boys were allowed to ask Obama one question each. The parents had no idea what they were going to say. Isaac asked why the president had got rid of the F-22 jet fighter. The president said because it cost too much. Jacob asked:

"I want to know if my hair is just like yours."

He was so quiet, Obama asked him to repeat the question. Jacob obliged.

Obama said: "Why don't you touch it and see for yourself?"

He bent down lowered his head so that it was within Jacob's reach.

Jacob paused. The president prompted. "Touch it, dude!" he said

Jacob reached out and rubbed the presidential pate.

"So, what do you think?" Mr Obama asked.

"Yes, it does feel the same," Jacob said.

The White House photographer snapped the moment. "Every couple of weeks the White House photographers change out the photos in the West Wing," Michelle Obama said at a fundraiser in September. "Except for that one. So if you ever wonder whether change is possible, I want you think of that little black boy in the Oval Office of the White House touching the head of the first black president."

The symbolic resonance of Obama's victory for black Americans has not diminished. At rallies the hawkers are still there with T-shirts setting him alongside Martin Luther King, setting his logo within Superman's crest or insisting: "I like my coffee black. Like my president". According to Gallup 90% of African Americans intend to back him and they plan to turn out at the same rate as white voters. No other block of voters is more loyal.

No other block of voters is more optimistic. Over the past few years polls have consistently shown that African Americans are more likely than any other group to be bullish about their own future, to think the country's best days are yet to come and that the economy is already recovering.

A Pew survey in January 2010 indicated that the percentage of black Americans who thought blacks were better off than they were five years before had almost doubled since 2007. There were also significant increases in the percentages who believed the standard-of-living gap between whites and blacks was decreasing. No wonder they love the president.

There was only one trouble with these assessments. They weren't true. African Americans, as a group, are far worse off now than they were when Obama came to power and the gap between whites and blacks in terms of wealth and income has increased under Obama's tenure. The overall rate of unemployment may be close to where it was when Obama took office, but black unemployment is up 11%. Meanwhile the wealth gap has doubled during this recession with the average white American now having 22 times more wealth than their black counterparts. So too has the educational achievement gap with the rate at which white Americans graduate from high school growing at a far faster clip than black students.

"We haven't seen much of the stimulus trickle down to our people here," Mark Allen, a Chicago-based community organiser who used to work alongside Obama, told the Washington Post. "I liked the community organiser Obama better than President Obama … Democrats say Barack has got 90% or whatever of the black vote wrapped up. What they don't tell you is it's 90% of those who actually come out and vote. What if it's 90% of just 30 or 40% who vote?"

In short, Jacob's odds of getting a decent job when he gets older actually got worse since he felt the president's hair, while the gap between his life chances and his white schoolmates widened and his odds of going to prison remained pretty much the same. In empirical terms "the change that [has been] possible" for Jacob and his family under Obama has been change for the worse. One can argue about the cause of those changes and the degree to which Obama bears any responsibility for either creating them or fixing them. But one cannot argue about the fact of them: the ascent of America's first black president has coincided with the one of the steepest descents of the economic fortunes of black Americans since the second world war both in real terms and relative to whites.

Then-senator Barack Obama with Reverend Jesse Jackson in 2007. Photograph: Jeff Haynes/AFP

Herein lies the dual paradox. The group that has fared worse under Obama is not only the group most likely to support him but also the most likely to feel optimistic about the deteriorating situation in which they find themselves. And why has that loyalty to the president yet to be fully tested? What do they know that the numbers don't show?

Discussing this dilemma within the black community can be tantamount to heresy. Wagons circle, messengers are shot, ranks close, critical faculties are suspended. "Too many black intellectuals have given up the hard work of thinking carefully in public about the crisis facing black America," Princeton professor Eddie Glaude told fellow academic Fred Harris recently. "We have either become cheerleaders for President Obama or self-serving pundits." Not only are criticisms shunned but even constructive critiques are unwelcome. At times it seems like questions as to how his tenure has affected black communities either should not be asked or, at the very least, should not be answered honestly.

"I have friends," says Virginia state delegate, Onzlee Ware from Roanoke, who is an ardent Obama supporter, "who, if I bring [his shortcomings] up as an intellectual conversation, they say I'm a traitor."

There are some sound reasons for this. The first is the overt racism that Obama has faced from a significant portion both of the political class and the public as a whole. There are plenty of reasons why one might oppose Obama that have nothing to do with race. When you look at how the things they accused the Clintons of – killing people, smuggling drugs from abroad, embezzling – the Obama's are not unique in being the targets of a right wing hyper-caffeinated lie machine.

Nonetheless, the nature of these particular lies and attacks have, as often as not, been rooted in race. Half of white Americans in one Pew survey shared the birthers' doubt that Obama was born in this country. The percentage of Americans who believed he is a Muslim has doubled since he took office. After the president produced his long-form birth certificate, Donald Trump demanded his college transcripts (claiming he was not smart enough to get into an Ivy League school). Newt Gingrich branded him the "food stamp president". Southern congressmen shout "liar" while he's speaking, Romney surrogates question his connection to "Anglo-Saxon values".

Far from his election signalling a post-racial era of equality it has exposed and unleashed a visceral level of intolerance that has produced the most racially polarised electorate for at least a generation. Having alienated blacks and Latinos, a recent poll revealed that Romney's support is 91% white – that's a higher proportion than any candidate since Bush's father stood in 1988 and may yet surpass it.

Meanwhile a recent AP poll revealed that, if anything, racist attitudes have hardened in the country since Obama's election. The poll showed that "51% of Americans now express explicit anti-black attitudes, compared with 48% in a similar 2008 survey". The proportion who express anti-Hispanic feelings is roughly the same. That's within the margin of error. But what's clear is that it's not going down. Anyone who seriously believes Obama's election ushered in a new period of racial harmony simply hasn't been paying attention.

In this atmosphere many African Americans become understandably defensive. Under such sustained racial onslaught the space for free-wheeling conversation and constructive criticism becomes limited because the issue has shifted from what Obama has done to who he is. Given the nature of the attacks the need to defend Obama's right to be in office at all eclipses any more nuanced conversation about his actual record.

Moreover much of the criticism Obama has faced from the black community has been either ridiculous or self-defeating. Last year former Princeton professor Cornel West led a well-publicised assault insisting that Obama has "a certain fear of free black men".

"It's understandable," he said. "As a young brother who grows up in a white context, brilliant African father, he's always had to fear being a white man with black skin. All he has known culturally is white. He is just as human as I am, but that is his cultural formation. When he meets an independent black brother, it is frightening ... He has a certain rootlessness, a deracination. It is understandable."

Essentialising Obama's racial and cultural makeup in such a way makes the very mistake that the right makes – assessing Obama not on what he does but by who he is.

Mona Renee Johnson weeps as Obama speaks at a campaign rally in Las Vegas, Nevada. Photograph: Kevin Lamarque/Reuters

It also harps back to an era of black political leadership, where black politicians emerged from the church or historically black colleges, and fought not to win office outside the black community (white people wouldn't vote for them) but to put the needs of that community on the agenda. There was, in a previous generation, a sense of ownership that black communities had over their politicians that no longer exists. This is partly progress. Ivy League universities will admit them, corporations will hire them, funds will come to them, white people will now vote for them. A whole range of opportunities are open to politicians of Obama's generation that were created by Cornel West's generation.

But that, in turn, has changed what it means to be a black politician and what, if anything, we mean when we talk about black politics. Unlike, say, Jesse Jackson or Martin Luther King, Obama was not politically produced by the black community, but presented to it after he had made his way through the mostly white elites. His political ties to the black community are not organic but symbolic. His arrival in the political class is hailed as the progress of a community when in fact it is the advancement of an individual.

"[Obama] is being consumed as the embodiment of color blindness," Angela Davis, professor of history of consciousness at the University of California, Santa Cruz, told me in late 2007. "It's the notion that we have moved beyond racism by not taking race into account. That's what makes him conceivable as a presidential candidate. He's become the model of diversity in this period … a model of diversity as the difference that makes no difference. The change that brings no change."

That is why criticisms of him for "not doing enough for his own people" both miss and devalue the point. The demand to close the racial gaps bequeathed by centuries of discrimination is not a sectional interest but a national one. Demands for equality and racial justice should be made to any president of whatever race or party.

Obama should do more for black people – not because he is black but because black people are the citizens suffering most. Black people have every right to make demands on Obama – not because they're black but because they gave him a greater percentage of their votes than any other group, and he owes his presidency to them. Like any president, he should be constantly pressured to put the issue of racial injustice front and centre and if black people aren't going to apply that pressure then nobody else will.

But in fact precisely the opposite has been happening. With Obama in the White House African Americans representatives have been backpedalling. Black politicians, too, have held their fire.

"With 14% unemployment, if we had a white president we'd be marching around the White House," said the chairman of the Congressional Black Caucus (CBC), Emmanuel Cleaver. "The president knows we are going to act in deference to him in a way we wouldn't to someone white." That's pathetic and counterproductive. These are the very people who are now showing up with empty hands and trying to galvanise the black community to go to the polls.

Their reticence is partly explained by the fear of a backlash. "If we go after the president too hard, you're going after us," Maxine Waters, a California Democrat in the House, told a largely black audience in Detroit last year. But then that's what leadership is about. Explaining to those audiences that there are large numbers of people lobbying for Obama's attention, including people with huge amounts of money and power. If the black community wants it they must demand it.

Some have spoken out. In August after a month-long round of job fairs organised by the CBC across the nation John Conyers, the longest serving black American in Congress said. "We want [Obama] to know from this day forward that we've had it. We want him to come out on our side and advocate, and not to watch and wait … We're suffering." Unfortunately it was followed by little in the way of action.

In the absence of that pressure Obama has felt little need to focus his attention on the problem, even rhetorically. In his first two years in office he talked about race less than any Democratic president since 1961. In all of his state of the union speeches he mentioned poverty just three times: last year's was the first since 1948 to not mention poverty or the poor at all. When he did talk about it it was to preach better parenting, healthy meals and greater discipline.

At a Congressional Black Caucus meeting in September he told his former colleagues: "Take off your bedroom slippers, put on your marching shoes. Shake it off. Stop complaining, stop grumbling, stop crying." Compare that to the meeting he had with bankers not long after he was elected when they thought he was going to impose serious regulation. "I'm the only thing standing between you and the pitchforks. I'm not out there to go after you," he told them. "I'm protecting you."

This would not be the first time that the black Americans have shown great loyalty to a Democratic president who did not return the favour. Bill Clinton is still revered even though when he ran in 1992 he made a special trip back to Arkansas to oversee the execution of Ricky Ray Rector – a black, lobotomised inmate so mentally incapacitated that when given his last meal, he opted to save the dessert for after the execution. When in power he signed off on a welfare reform that would prove devastating to large numbers of black families, especially women. He presided over an economic boom. Obama does not even have that.

It may be in this mixture of realism and low expectations that one can understand the logic of optimism in harder times. That black Americans are doing worse than everyone else, and that the man they elected to turn that around has not done so, does not fundamentally change their view of how American politics works; almost every other Democratic president has failed in a similar way while Republicans have not even tried to succeed.

Conversely the fact that a black man might be elected president, that enough white people might vote for him and that nobody has shot him, really has changed their assumptions about what is possible. Jacob's story from the Oval Office is new and inspiring; the story of his odds of success beyond that moment are wearily familiar.

The day Obama took office, the world may have looked at black America differently, but black America has yet to look at Obama differently. When he went from being an aspiration to a fact of political life, the posters that bore his likeness in socialist realist style over single-word commands like Hope, Believe and Change should have been replaced with posters bearing the single-word statement: power. As Frederick Douglass said: "Power concedes nothing without a demand. It never did and it never will."