Black people have lived in Britain at least from Roman times, and some historians claim that north Africans were here as much as 3,000 years ago. We know that Indian people were here as far back as Shakespeare's time. The first Chinese visitor we know of was the Jesuit priest Shen Foutsong, who communicated in Latin when he worked at Oxford's Bodleian Library in the 17th century. His portrait still hangs in the Queen's collection. People of colour have been part of the fabric of British society for centuries, but you won't find many in official histories – either from the right (look at Michael Gove's draft national curriculum) or, more shockingly, from the left.

Ken Loach's feature-length documentary, The Spirit of '45, is one recent example. A documentary about the creation of the welfare state and its legacy, it presents us with Loach's vision of the British working class, united in the struggle for a better Britain. And though it covers the period from the 1930s up to the Thatcher era, everyone featured in the film is white – it's as if people like me have been bred out of the working-class gene pool.

In this Loach is swimming with the tide. Both Maurice Glasman (of Blue Labour fame) and David Goodhart, the former editor of Prospect magazine, are very influential in Labour's thinking – and both nostalgically emphasise the importance of continuity and community values in the British working class, as against immigrants, who threaten that continuity. While the working class is rarely discussed in mainstream left circles these days, the "white working class" is endlessly debated. "Working class" becomes indivisible from "white" in such debates.

This is not an accurate portrayal of the British working class, either now or in the past. My own father, an ex-seaman, was a British trade unionist in Liverpool from the 1920s onwards, and helped found the Chinese Seamen's Union. It was necessary: Chinese people ran much of the merchant navy in the second world war, and plenty died for us in conflicts up to and including the Falklands war, and yet they suffered horrible discrimination. Their pay and conditions were inferior to those of their white counterparts. Adding insult to injury, many were forcibly sent back to China after the war despite having settled here with families.

And many other people of colour contributed to the spirit of 1945 Loach celebrates: the black pilots who flew in the RAF in the second world war; the Chinese firefighters in Liverpool (including my dad) who fought the flames ignited by Luftwaffe bombs; the Nepalese Gurkhas and Indians who fought for Britain in two world wars, making up the largest volunteer army in world history. And of course, postwar, the thousands who arrived here in waves of immigration from the Caribbean in order to run the new NHS and public transport systems.

People of colour have performed important work in the labour movement throughout its history. I remember at least one south Asian shop steward at Ford Dagenham in the 1970s, and Asian women won a famous victory at the Grunwick film processing plant in a strike lasting from 1976 to 1978, in what journalist Paul Foot described as "a central battleground between the classes and between the parties".

Once you look beyond Britain, and the narrow definition of the British working class as only those who labour within its borders, the composition of those who have resisted capitalist predation looks different again. As George Orwell says in his bitter 1939 essay, Not Counting Niggers: "What we always forget is that the overwhelming bulk of the British proletariat does not live in Britain, but in Asia and Africa." Without cheap non-white labour from the empire, Britain couldn't have afforded the welfare state of which we're so proud. The blind spot that so irritated Orwell is alive and well on the left today.

While many on the left would sell us a mythologised, all-white Little England, it is worth remembering that the British working class itself has a proud tradition of rejecting racism; we shouldn't allow this to be airbrushed out of existence by those who purport to represent them. Even though the cotton famine of 1863 plunged Lancashire textile workers into life-threatening poverty (because the slave-grown cotton from America's southern states on which the industry relied was blockaded in the civil war), the textile workers actively supported the fight against slavery. In 1862 they wrote to Lincoln expressing their "hope that every stain on your freedom will shortly be removed, and that the erasure of that foul blot on civilisation and Christianity – chattel slavery – during your presidency, will cause the name of Abraham Lincoln to be honoured and revered by posterity".

As well as being a prominent anti-slaver, the Manchester radical Richard Cobden was a leading campaigner against the opium wars, when Britain attempted to impose cheap mass-produced opium on the Chinese population by force of arms. In 1936 East End Jews found themselves arm in arm with many diverse groups who turned out to challenge Mosley's Blackshirt fascists in Cable Street. In the second world war Brits welcomed black American GIs based here, to the dismay of the US army's Jim Crow segregationist officers.

Examples of cross-race class struggle are many: so why impose such a filter? Who gains? Constructing a narrative palatable to a constituency increasingly susceptible to the dishonest blandishments of the right is a divisive and dangerous game, especially in the current atmosphere of immigrant bashing and fear of the other. Harking back to a fictitious golden age when everyone was white represents surrender to an antagonism stoked up towards "outsiders", with polls showing that increasing numbers of us blame immigration for dwindling resources, though facts prove otherwise.

Loach's airy dismissal on Radio 4 of his whitening of our history was simple: "That's how it was. That's the record of the time. That's what people thought, that was the moment of the time." You can't include stuff "to suit our present sensitivities".

He's wrong; that's not how it was. And that matters: the left should educate people and inspire them with the truth. The ethnic cleansing of our history is an insult to the spirit of the working class it purports to represent.

• This article was amended on 17 July 2013 following a complaint from John Rees, an organiser of the People's Assembly and a former member of the SWP, to remove remarks which he denies making.