I was 14 in 1968 and one of the horrors of my teenage years was Enoch Powell. For a mixed-race kid, this stiff ex-colonial zealot – with his obscene, grand guignol talk of whips, blood, excreta, urination and wide-eyed piccaninnies – was a monstrous, scary bogeyman. I remember his name being whispered by my uncles for fear I would overhear.

I grew up near Biggin Hill airfield in Kent, in the shadow of the second world war. We walked past bomb sites everyday. My grandmother had been a “fire watcher” and talked about the terror of the nightly Luftwaffe raids. With his stern prophet’s nostalgia, bulging eyes and military moustache, Powell reminded us of Hitler, and the pathology of his increasing number of followers soon became as disquieting as his pronouncements. At school, Powell’s name soon become one terrifying word – Enoch. As well as being an insult, it began to be used with elation. “Enoch will deal with you lot,” and, “Enoch will soon be knocking on your door, pal.” “Knock, knock, it’s Enoch,” people would say as they passed. Neighbours in the London suburbs began to state with some defiance: “Our family is with Enoch.” More skinheads appeared.

It was said, after Powell mooted the idea for a Ministry of Repatriation, that we “offspring”, as he called the children of immgrants, would be sent away. “A policy of assisting repatriation by payment of fares and grants is part of the official policy of the Conservative party,” he stated in 1968. Sometimes, idly, I wondered how I might like it in India or Pakistan, where I’d never been, and whether I’d be welcomed. But others said that if we were born here, as I was, it would be only our parents who would be sent back. We would, then, have to fend for ourselves, and I imagined a parentless pack of us unwanted mongrels, hunting for food in the nearby woods.

Repatriation, Powell said, “would help to achieve with minimum friction what must surely be the object of everyone – to prevent, so far as that is still possible, a major racial problem in the Britain of AD2000.” It was clear: if Britain had lost an empire and not yet recovered from the war, our added presence would only cause more strife – homelessness, joblessness, prostitution and drug addiction. Soon the indigenous whites would be a “persecuted minority” or “strangers” in their own country. It would be our turn, presumably, to do the persecuting.

The influence of Powell, this ghost of the empire, was not negligible; he moved British politics to the right and set the agenda we address today. It’s impossible not to summon his ghost now that immigration is once again the subject of national debate. Politicians attack minorities when they want to impress the public with their toughness as “truth-tellers”. And Powell’s influence extended far. In 1976 – the year before the Clash’s “White Riot” – and eight years after Powell’s major speeches, one of my heroes, Eric Clapton, ordered an audience to vote for Powell to prevent Britain becoming a “black colony”. Clapton said that, “Britain should get the wogs out, get the coons out,” before repeatedly shouting the National Front slogan “Keep Britain White”.

A middle-class, only child from Birmingham, socially inept and repressed, Powell had taken refuge in books and “scholarship” for most of his life. He was perhaps happiest during the war, spending three years in military intelligence in India. Like a lot of Brits, he loved the empire and colonial India, where he could escape his parents and the constraints of Britain. Many Indians were intimidated by and subservient to British soldiers, as my family attested. Like most colonialists, Powell was a bigger, more powerful man in India than he’d have been in England. No wonder he was patriotic and believed giving up the empire would be a disaster. “I had always been an imperialist and a Tory,” he said.

On his return in 1945, Powell went into politics. Like the grandees he aspired to be, he took up churchgoing and fox-hunting. Before his speeches on race, he was an obedient, relatively undistinguished servant of the state. But he was also, in fact, a proto-Thatcherite: a supporter of the free market and lower taxes with a utopian vision of unregulated capitalism where, miraculously, everything people required would be provided by the simple need for profit. Soon, as Thatcher said, there would be no alternative.

But, in 1968, that great year of newness, experimentation and hope, when people were thinking in new ways about oppression, relationships and equality, there was a terrible return. This odd Edwardian figure popped up into public life, and decided to became a demagogue. Richard Crossman, in his diary of 1968, worried about Powell’s celebrity appeal to “mass opinion, right over our parliament and his party leadership”.

Appealing to the worst in people – their hate – is a guaranteed way to get attention, but it is also fatal. Powell talked in whole sentences and was forever translating Herodotus, so was known for his cleverness. But he wasn’t smart enough to resist the temptation of instant populism for which he traded in his reputation. Racism is the fool’s gold, or, rather, the crack cocaine of politics. The 1970s was a dangerous time for people of colour – the National Front was active and violent, particularly in south London, and it was an ignoble sacrifice for Powell to attack the most vulnerable and unprotected, those workers who had left their homes to come to Britain. He elevated his phobia to a political position, and there was no going back.

Like many racists, Powell was nostalgic in his fantasies: before all this mixing, there was a time of clarity and plenitude, when Britishness was fixed and people knew who they were. Powell refused to allow his certainties to come into contact with reality. He had wanted to know India, but barely troubled himself with Britain and, apart from some weekends in Wolverhampton, lived most of his life in Belgravia.

In contrast to the crude caricatures of people of colour perpetrated by Powell, the Guyanese-born, Cambridge-educated writer ER Braithwaite – who served in the RAF before becoming a teacher in the East End because he couldn’t get a job as a engineer – writes in detail about race between the late-40s and the mid-60s. Three important works in particular, To Sir, With Love, Reluctant Neighbours and Choice of Straws engage with this era. From this clear-eyed, brave novelist we learn about the everyday humiliations, abuse and remarks that people of colour had to face after being invited to help run the NHS and transport system. To make the future it wanted, Britain needed the best doctors, engineers, architects, artists and workers of all kinds, and it imported them, before insulting them.

ER Braithwaite in the early 1960s. Photograph: FPG/Getty Images

Powell liked to complain about every vile “imputation and innuendo” made about him; he was keen to be a martyr and victim. Braithwaite, for his part, really suffered. He catalogues the systemic and degrading exclusion from jobs and housing that so disillusioned immigrants about the British with their babble about fairness, liberty and the mother country. His books describe the rage and hate that relentless humiliation inevitably engenders – as colonialism did, in its time. Powell probably intuited the simple idea that tyranny creates resistance, and grasped that future conflicts would be caused by the tyranny he supported, hence his apocalypticism.

Powell developed his own schoolmasterish look. Always in black, sometimes in a long overcoat and occasionally in a little homburg, he was punky and subversive, and came to enjoy making everyone furious with his provocations. And he had the cheek to call us “a roomful of gunpowder”. He didn’t fit in; but he certainly liked to disorientate and traumatise us. After he spoke, we were in freefall; we didn’t know where or who we were. Powell wanted to confirm us as outsiders, as unintelligible and unwanted, but this helped us clarify things and created resistance. Out of Clapton’s statements, for instance, came Rock Against Racism, created by artists, musicians and activists to combat fascism. Then there was identity politics. We were not nothing; we had histories and, unlike him, we had futures.

Powell was creating the conflict he claimed to be the solution to. He soon found himself supported by the National Front. Powell had called himself a Nietszchean as a young man, but Nietzsche would have hated the wretched appeal to the mob or herd. Powell was merely addressing the bitter rabble, and, for so fastidious a man, this would have been distasteful, and he must have considered how incapable our intelligence can be when it comes to protecting us from the temptations of self-destruction.

He cheated his followers, because all he gave them was the brief thrill of superiority and hatred. Nothing substantial altered in the world, and the wild, amoral capitalism that developed from his Hayek-inspired economic vision created wealth for some, but otherwise had no respect for the homes or jobs of Powell’s followers, nor for the other things he cared about – tradition, national borders, patriotism or religion.

Although he was attacked and condemned by students wherever he went, he didn’t trouble himself to think about the profound social changes sweeping the country, as young people attempted to liberate themselves from the assumptions of the past. Britain wasn’t decaying, it was remaking itself, even as it didn’t know how the story would end.

In London now, if you stroll through the crowds on a bright Sunday afternoon near the museums and decorated shop fronts, even for those of us who have been here for years, this multiracial metropolis – less frantic than New York, and with more purpose than Paris, and with its scores of languages – seems like nothing that has ever been made before. And it grows ever more busy, bustling and compelling in its beauty, multiplicity and promise, particularly for those of us who remember how dull and eventless London could seem in the 70s, especially on Sundays.

Britain survived Powell and became something he couldn’t possibly have envisioned. He was a pessimist and lacked faith in the ability of people to cooperate with one another, to collaborate and make alliances. The cultural collisions he was afraid of are the affirmative side of globalisation. People do not love one another because they are “the same”, and they don’t always kill one another because they are different. Where, indeed, does difference begin? Why would it begin with race or colour?

Racism is the lowest form of snobbery. Its language mutates: not long ago the word “immigrant” became an insult, a stand-in for “paki” or “nigger”. We remain an obstruction to “unity”, and people like Powell, men of ressentiment, with their omens and desire to humiliate, will return repeatedly to divide and create difference. The neoliberal experiment that began in the 80s uses racism as a vicious entertainment, as a sideshow, while the wealthy continue to accumulate. But we are all migrants from somewhere, and if we remember that, we could all go somewhere – together.