A few years ago, when working for the Daily Telegraph, I travelled to Leeds on a story about a teenage computer geek who hacked into the Pentagon computer network. Knowing that our subject once worked as a paperboy, a photographer and I wandered into a newsagent's close to his home, hoping for some insights. None sprang forth, but we passed a few amiable minutes with the owner, a smiling, ruddy-faced man, and then left.

Ten minutes later I was called by the news editor in London, who was laughing so much he could barely speak. "We've just had a call from a bloke who said these two guys came into his shop claiming to be from the Daily Telegraph, but he said he knew they couldn't be because one of them was black and the other one was half-caste," he told me. "He was calling to warn us."

We should, perhaps, have done other things but instead we returned to the shop, part amused, part irritated. "By the way, I am black," I told the owner as we left the shop. "But you are wrong about him," I said, pointing to my colleague. "That's a suntan. He's just spent two weeks in Spain. "

These things rarely happen to me these days. Perhaps it's a result of living in London. Maybe I have been fortunate. But new research on black attitudes to racism suggests that for all the advances in race relations, many feel their lives are blighted by stereotyping and discrimination.

In what is billed as the most comprehensive survey of black opinion in 20 years, the Voice newspaper depicts communities of African-Caribbean and African origin as transparently aggrieved by the feeling that their pigmentation determines how they are viewed and ultimately how they will fare in life.

Of the 600 people questioned, 94% said there is continuing racism in the UK today, and the feeling was most acute among those of Caribbean backgrounds, 96% of whom felt advances have not gone far enough. More than a third felt that racism in the UK today is actually worse than three or four years ago - a galling statistic when one considers that most live in London, whose diversity helped win it the right to stage the 2012 Olympics - and 60% said black people fare worse than other racial minorities.

Those questioned bemoaned their failure to be promoted at work and the effects of institutional racism, with 80% citing inequalities in the criminal-justice system. It's pretty gloomy stuff. And for those of us who like to think that things took an upward turn in the late 1980s, prompted by the Macpherson inquiry into the death of Stephen Lawrence and all the changes it wrought, it seems a bit of a wake-up call.

It's always hard to predict the future but we can say two things with reasonable certainty. One is that things will get worse before they get better. The problems afflicting communities of African-Caribbean origin have long ceased to cause ministers any sleepless nights. There is too much petty crime within and involving our black communities. But we don't riot any more and we aren't intent on blowing things up. We have long ceased to be needy minority group number one as far as the authorities are concerned. Anyone expecting the government to change our situation - with cash or even with renewed attention - is going to have a long wait.

Even schemes born of good intentions, such as the government's Sure Start programme, don't have much impact on us. This week an analysis concluded that the system is so bureaucratic as to represent a "substantial wasted opportunity" for black and other ethnic-minority families. Some may have benefited, others not. But there has been no meaningful evaluation, so no one knows.

And so the second thing to say is that as racism isn't going anywhere for the foreseeable future, we had better develop better coping strategies for dealing with it, as our parents had to do in the 1950s. All the evidence is that those solutions are going to have to come from within: from the black-led organisations and voluntary groups; from the black supplementary schools; from the heavily supported and wealthy black churches; from parents keeping their children in school - with or without the help of the authorities - and keeping them off the streets.

The Rev Al Sharpton, the American civil rights leader, isn't to everyone's taste, but he has a saying that seems appropriate. "If a man knocks you down," he tells audiences, "that's on them. If you stay down, that's on you."

hugh.muir@theguardian.com