It is a paradox of the history of British migration that, while the first generation of postwar black immigrants came to this country to work, unemployment among their children and grandchildren is stubbornly high. Figures I've just received from the Labour Force Survey reveal that unemployment among young black people (aged 16-24) is a shocking 44% – over twice the rate of their white counterparts, of whom 20% are jobless.

Black and Asian migrants after the war helped rebuild the economy: working in factories; doing the night shifts; working on the railways and driving the buses. And a generation of West Indian women made a contribution to the health service that has never been properly celebrated. So whatever else the problem is, black Britons don't come from households that didn't value work.

One of the causes of high black unemployment is shared by working class males whatever their colour. Structural changes in the economy mean that the type of blue-collar jobs that the first generation of migrants did no longer exists. When I was a child, areas like Willesden and Park Royal in north-west London were full of manufacturing and light-engineering factories. The large black community there owes its existence partly to just those employment opportunities. But these jobs have largely vanished from London.

My father came to this country in the 1950s having left school in Jamaica at 14. But he was able to find factory work and rise to become a sheet metal worker with his own apprentices. Now there are no job opportunities for young people without formal qualifications. A generation ago it was still possible to leave school at 16 and become a bank clerk, a nurse, a local government officer. Now many employers recruiting for similar roles demand a degree. There is no question that a lack of qualifications holds some young black people back. But there is anecdotal evidence that black people emerging from university with the same qualifications as their white peers find it much more difficult to get employment. Lack of qualifications alone does not account for this level of unemployment.

What is clear is that this recession is hitting ethnic minorities disproportionately hard. And the figures can only get worse. Black people, particularly women, are more likely to work in the public sector. This is partly because in diverse inner-city areas the public sector is the biggest employer. But it is also because large public-sector organisations tend to have better, more transparent policies around equal opportunity. Yet the public sector is bearing the brunt of George Osborne's cuts.

In recent decades black people have made advances in all kinds of employment. But the Americans have a saying: "Last to be hired, first to be fired". This may be reflected in some of the racial disparities that are emerging.

Some people will be antagonised by any discussion of the fact that spiralling unemployment is hitting black people hardest. They may think it a price worth paying for cutting back on public spending. Or they may argue that it doesn't matter what colour you are. But the more unequal a society, the more unstable it is. And inequality with a racial dimension risks creating a time bomb. The immediate response to last summer's riots was (quite correctly) a call to restore order. But these figures are not irrelevant. Policymakers cannot afford to ignore black unemployment.

Hardworking immigrant grandparents would not want special treatment for this generation: after all, they themselves did not have any. But they would expect this society to care, and be prepared to examine carefully what the underlying reasons might be. That generation of migrants were God-fearing monarchists. So they would expect fairness and justice. And as their grandchildren might put it: "No Justice, No Peace".

• Follow Comment is free on Twitter @commentisfree

• This article was critised by Channel 4's FactCheck. See our reply here.