Doug Wilder, 77, still meets people who wanted to vote for him when he stood for governor of Virginia back in 1989 but found they just could not do it. They said they would. They even thought they would. But when it came down to it, they just could not vote for a black man. "I've had people who tell me 'I didn't vote for you for lieutenant governor or governor. I wish I had that chance again'," he says.

On the eve of his election he led in the polls by 9%. On the day he won by less than 0.5%. They call it the Wilder effect - the shortfall between white voters' professed support for black candidates and their propensity to actually vote that way. They also call it the Bradley effect, after the Los Angeles mayor Tom Bradley who stood for California governor in 1982. Back then the deception continued even after some had cast their ballot. Bradley's exit poll lead was so significant that early editions of the San Francisco Chronicle projected his victory. He lost by just over 1%.

The question over the next two months is: will there be an Obama effect? And if so will it end like Wilder, in victory; or like Bradley, in defeat? At its heart lies the issue of how America understands the relationship between race and racism, and the degree to which claims of his post-racial candidacy have been misconstrued.

The issue here is not whether racism will cost him the election, but whether the race we think we are watching is in fact the race that is taking place. In essence, the Wilder effect exposes the gap between how comfortable a minority of white voters are in admitting their prejudice and how ready they are to act upon it.

Over the past 40 years racism has ceased to be socially acceptable. Nonetheless, it maintains broad appeal. The nation's most popular radio host, Rush Limbaugh, has referred to Obama as the "little black man-child"; and Fox News branded his wife, Michelle, as his "baby-mama". Yet while few are comfortable being called racist, racism persists. "Do I think there's been a seismic change?" asks Wilder, who is now the mayor of Richmond. "Yes. Do I think racism is still around? Yes. Do I think it's as pervasive as it has been? No. I don't subscribe to the idea that the majority of Americans are racist."

Quite. Yet black Americans are three times more likely to live in poverty and twice as likely to be unemployed or drop out of school. Such disparities do not happen by accident.

So we have racism but no racists. A system of discrimination - albeit much altered from 40 years ago - remains, yet no one will take responsibility for it. There are views that pervade, but apparently no one who actually holds them. Such is the contradiction that plays out between what some people tell pollsters and what they actually do at the polls.

Navigating these particular incongruities is the challenge of a new generation of black candidates. "In so much of the work I've done, I've found that you had to put people at ease on the question of race before you could even start to talk about what you were doing," explains the Massachusetts governor, Deval Patrick. "I don't fit a certain expectation that some people have about black men. And I don't mean that as anything other than an observation about my life."

This is simply a candid assessment of black middle-class life in America in or out of politics. Those who are keen to succeed must first negotiate racism in all its subtlety and plausible deniability. Wronged or not, they must avoid any hint of complaint lest it be taken for grievance; talented or not, they must avoid any hint of over-accomplishment lest it be taken for hubris.

Last week, one Georgia congressman referred to the Obamas as "uppity". After Obama delivered a throwaway line about Republicans trying to scare voters because he "doesn't look like all those other presidents on the dollar bills", the Republican attack machine went into overdrive.

"There's no doubt in my mind that what Senator Obama is trying to suggest - that he's a victim of something," said the South Carolina senator Lindsey Graham. An uppity black man playing the victim - meet Willie Horton's 21st-century cousin.

"Is it fair?" asks Wilder, referring to the demands placed on black candidates. "No. Life isn't fair. That's what you have to overcome ... If that's the bar, you meet it. What do I have to do to have that door opened for me? Knock on it. Or break it down. You tell me what I need to do. If it doesn't open and I have to break it down, I will develop the strength. If I knock on it and it doesn't open, I want to know why. Don't complain about it. Just do the job."

Obama has so far been knocking firmly and politely and doing the job with a sly cunning. His strategy has been to play up the historical resonance of his candidacy while downplaying any overt mention of either race or black people. This was particularly clear during his nomination acceptance speech in Colorado, which was deliberately timed to coincide with the anniversary of Martin Luther King's "I have a dream" speech. Yet he didn't once mention King's name, referring instead to a "young preacher" and quoting one of the more forgettable parts of the most memorable speech in American history. Race is central to his meaning, but absent from his message.

This is less a criticism than a description. Obama is navigating uncharted and decidedly choppy waters. It is difficult to see how else he could play it. And yet it is not without its problems. At any moment, while passing for the presidency, he can be outed by anything from a preacher to a fist bump or a magazine cover. Such is the lot of the incognegro.

For what people really mean when they refer to his candidacy being post-racial is that it is "not too obviously black". The trouble is that is as racial a category as any other, albeit a negative one. The paradox is that, with historically low levels of black and Hispanic support, John McCain's is the most "racial" candidacy we've seen for a long time.

The days of the Wilder effect may be waning. A report by the Pew Research Centre, following the 2006 midterms, found the polls on black candidates to be highly accurate. "Fewer people are making judgments about candidates based solely, or even mostly, on race itself," it concluded.

But then it only takes a few, and Wilder believes they are still out there. "It doesn't have the same salience. It doesn't have the same impact. But are there going to be people who swear they're Democrats; who swear they like him; who swear they believe in what he's saying? But yet when they go to the polls and grab that lever their hands will shake and tremble and they won't be able to pull it? Yes."

But the chance that Obama may follow in his footsteps does not worry Wilder. "I hope he does," he says. "Because if he repeats it, notwithstanding what it looked like, he would still win ... The only poll that counts is the one on election day."

g.younge@guardian.co.uk