Malcolm X wouldn’t recognise Smethwick these days. When he visited in 1965, the town north-west of Birmingham was certainly the most colour-conscious, perhaps the most racist, place in Britain. Its Conservative MP, Peter Griffiths, had been elected in the previous year’s general election on the slogan “If you want a nigger for a neighbour, vote Labour.”

The slogan helped buck national voting trends in 1964. Griffiths refused to disown it: “I would not condemn any man who said that,” he told the Times during his election campaign. “I regard it as a manifestation of popular feeling.”

In that election, Labour came to power in Westminster for the first time in 13 years with a national swing from the Tories of 3.5%. In Smethwick, though, there was a swing in the opposite direction: in what is surely the most racist election campaign ever fought in Britain, the Labour incumbent, shadow home secretary Patrick Gordon Walker lost on a 7.2% swing to the Tories that reduced his vote from 20,670 in the previous election to 14,916.

Griffiths, a local councillor born and bred in the Black Country, succeeded in mobilising racist working-class sentiment against a seemingly patrician Walker, whom local Conservatives derided for living in London’s genteel Hampstead Garden Suburb – far from industrial Smethwick, with its foundries, housing shortages and economic uncertainties. Two years earlier, Walker had opposed the introduction of the 1962 Commonwealth Immigration Act, which sought to restrict the entry into Britain of black migrants from Commonwealth countries. “How easy to support uncontrolled immigration when one lives in a garden suburb,” Griffiths had sneered at his Labour rival during the general election campaign.

As the defeated Walker left Smethwick town hall after the count 50 years ago today, Tory supporters yelled after him: “Where are your niggers now, Walker?” and “Take your niggers away!”

Local Conservative councillor Peter Griffiths canvassing on the streets of Smethwick in the 1964 general election campaign. Photograph: Express & Star

This racist campaign shocked Britain. In the Commons, the new Labour prime minister Harold Wilson called on then Tory leader Sir Alec Douglas-Home to disown Griffiths. “If Sir Alec does not take what I am sure is the right course, Smethwick Conservatives can have the satisfaction of having sent a member who, until another election returns him to oblivion, will serve his time as a parliamentary leper.” Twenty-five Tories walked out of the chamber in protest and proposed a motion deploring Wilson’s insulting language; Labour members countered by proposing a motion reproving the prime minister for insulting lepers.

One reason the Tories refused to condemn Griffiths was that he had found the Conservatives’ holy grail – undermining industrial working-class support for Labour. Margaret Thatcher would find a similarly winning formula in the 1980s by selling off council houses. Admittedly, Griffiths’ method had involved exploiting anxiety over a housing shortage in Smethwick and blaming it on immigrants but, still, it got him elected. When Griffiths died last year, John Spellar, the long-serving Labour MP whose Warley constituency covers Smethwick, reflected on what Griffiths’ racist electoral strategy amounted to: “Housing supply would be a concern to everyone, regardless of race, because people need to know that they can live close to their relatives. What the council of the day and Peter Griffiths did was to inflame the issue rather than try to resolve it and bring people together.”

Labour councillor Preet kaur Gill agrees. “The Conservatives played the race card in a way that is unimaginable now. Smethwick has become a place where people of all religions and ethnicities happily live together. It’s a success story rather than what it could have been – just ghettoes.”

That said, in 1964, Smethwick’s Labour party was part of the racist problem rather than clearly its solution. One of Gill’s predecessors as a Labour councillor was a man called Ken Burns who ran the Sandwell Youth Club and operated a colour bar there. Worse, there was a colour bar at the Labour club on Coopers Lane.

“Colour bars were common in the 60s,” recalls Harbhajan Dardi, 67, retired social worker, ex-director of Sandwell Citizens Advice Bureau and assistant general secretary of the Indian Workers’ Association. “Barbers would make the Asian customers wait three, four, five hours while they cut white people’s hair. In those days, the Ivy Bush down the road had a colour bar,” he tells me as we chat over tea in his home on West Park Road. “People like me couldn’t drink there.”

When Malcolm X came to Smethwick 49 years ago, he came to a town divided by race. The Indian Workers’ Association (GB) had invited him to show solidarity with Smethwick’s beleaguered black and Asian minorities. The American civil rights activist had flown from New York to Paris, where he was due to speak to a meeting of the Congress of African Students, but was refused entry to France as “an undesirable person”. He was then allowed into the UK to speak at the London School of Economics and the University of Birmingham Students Union, with a detour to Smethwick. After a whistlestop tour of the area, he had a pint in one of the few pubs that did not operate a colour bar. “It was full of Indians,” Avtar Singh Jouhl, one-time general secretary of the Indian Workers’ Association (GB), told the Independent a few years ago, “and they all wanted to shake his hand”.

“I have come,” Malcolm X told reporters as he posed for pictures in Marshall Street, “because I am disturbed by reports that coloured people in Smethwick are being treated badly. I have heard they are being treated as the Jews were under Hitler.” Reporters asked him what should be done. “I would not wait for the fascist elements in Smethwick to erect gas ovens.”

Malcolm X never did find out whether the fascists imposed a final solution on Smethwick’s immigrant population: nine days after his visit, he died in a hail of bullets at a New York ballroom.

Jack Beaula, Harbhajan Dardi and Bini Brown hold the heritage plaque marking Malcom X’s visit to Smethwick. Photograph: ThaboJaiyesimi/Demotix/Corbis

It was no accident that Malcolm X chose Marshall Street to walk down. This was the street that, if racists had prevailed, would have become all-white. In 1964, a group of white residents had successfully petitioned the Tory council to compulsorily purchase houses that came on the market on Marshall Street, and let them to white families only. Griffiths had his role in the policy: while serving as MP, he was also a council alderman who arranged for the authority to buy up the houses. Labour’s housing minister, Richard Crossman, frustrated the plan, refusing to allow the council to borrow the money.

Had Griffiths had his way, Britain would have become more like apartheid South Africa. In his 1966 book A Question of Colour? he wrote: “Apartheid, if it could be separated from racialism, could well be an alternative to integration.”

On a rain-soaked, windswept Monday lunchtime, I walk down the street they had wanted to keep white. There have been a few changes since Malcolm X visited. For one thing, it has become Griffiths’ nightmare – a mix of different religions and ethnicities. The electoral roll shows that Morrisseys, Mahmoods, Middletons, Singhs, Adams and Akhtars are all neighbours. I wonder what Malcolm X, who opposed integration of blacks with whites in the US (“You don’t integrate with a sinking ship,” he once quipped), would think about that. According to the 2011 census, minority ethnic groups (ie those who do not describe themselves as White British) make up 62.1% of the town’s population; in terms of religion, 39.6% describe themselves as Christian, 15.3% Sikh, 21.3% Muslim, 2.4% Hindu (14.6% expressing no religious affiliation). The most worrying statistic in the census, perhaps, is that 13.5% of Smethwick households have no residents who have English as a main language – the highest proportion of any town in Sandwell borough.

Gill says: “Smethwick has changed so much. In some areas there are more people of Bangladeshi origin than European. There are pockets where there is racism, but not much.” Gas ovens? Smethwick’s past half century hasn’t lived up to the more lurid predictions.

I knock on a few doors. No one I speak to is old enough to remember Marshall Street’s days of shame, and hardly anyone wants to talk to me about how race relations have improved since. When I ask a man with a Caribbean accent to show me where the plaque for Malcolm X is, he points down the street and raises a fist in solidarity with the late civil rights leader. I return the gesture, even though I look silly, or at least feel unworthy.

Enoch Powell, the Wolverhampton South West MP responsible for the notorious ‘rivers of blood’ speech. Photograph: Channel 4

I was raised in the Black Country when some people wanted to make it white. At the overwhelmingly white Alder Coppice primary school in Sedgley in the late 60s, I recall, children would march round the playground chanting: “Enoch Powell! Enoch Powell!” Powell was then the Conservative MP for nearby Wolverhampton South West, who in a notorious speech in Birmingham in 1968, played the race card just like Griffiths and was cheered to the rafters for pleading that immigrants be sent “home”.

A few years later, the Black Country-born comedian Lenny Henry would satirise such white delusions. In a gig at West Bromwich Plaza, Henry said the National Front wanted to give black people £1,000 to go home. Fine, Henry said: after all, that would more than cover his bus fare back to Dudley.

Was I one of the six-year-olds chanting Powell’s name in the playground and, albeit unwittingly (you’d think), supporting his call for immigrants and their families to be repatriated? I’m not sure, but I hope not.

In his speech, Powell spoke of a constituent – “an ordinary working man” – who was so concerned by the rising numbers of black immigrants that he was determined to see that his three children settled abroad. Immigration, he argued, could only result in civil unrest: “Like the Roman, I seem to see the Tiber foaming with much blood.” A Gallup poll suggested that 74% of the population supported his repatriation suggestion.

In the same year that Powell made his “rivers of blood” speech, 22-year-old Harbhajan Dardi arrived from India. “I came for an adventure more than anything else,” he says. “I didn’t intend to stay.” He had a degree from Punjab University but, like many postwar immigrants from the subcontinent to Smethwick, got his first job in a foundry. Ever since the 1760s when Matthew Boulton and James Watt opened the first foundries, Smethwick’s industrial success relied on supplying metal castings to factories in the West Midlands. In the 1950s, the boom in the Midlands car industry meant that there were labour shortages that were filled from Britain’s former colonies. “Colonial labour from the Commonwealth is greatly easing the labour shortage,” reported Smethwick’s labour exchange manager, Mr JE Stich, at the time. Yes, but often the “colonial labour” was paid less than white workers for doing more work, Dardi tells me.

Did Dardi face racism? “I did, but unlike the previous wave of immigrants to Smethwick, I was well enough educated and politically aware to fight against it. The first immigrants who came here after the war, well, many of them didn’t even know enough English to read the signs saying ‘No dogs, no Asians’.”

He got involved in the Indian Workers’ Association (GB), helping immigrants here to understand their rights. “Racism was overt. And then the 1976 Race Relations Act stamped out that kind of discrimination. We all know indirect racism still exists, but it is not so bad now.” The Ivy Bush, for instance, now serves curries six nights a week and is, Dardi tells me, a truly multicultural pub, a symbol of Smethwick getting along in ways undreamed of by either Griffiths or, one suspects, Malcolm X.

“I stayed and I’m glad I stayed,” Dardi says, as he breaks off from helping his five-year-old granddaughter learn how to do subtraction. He has led a rich life, always engaged in local community issues and politics. In 2001, he proudly tells me, he stood in the general election for Arthur Scargill’s Socialist Labour party in Warley and was the only one of its 114 candidates not to lose his deposit. In 2012, he was part of the community initiative to see Malcolm X’s visit to Smethwick honoured with a plaque. As I leave his house, Dardi points across the road to West Park opposite, where there is an outdoor gym, free for anyone to use. “I fought for that,” he says proudly.

“Second and third generations of immigrants are making lives for themselves here. Isn’t that great?” It is, but Dardi thinks there is a problem. “What they need to realise is that this is their home, that their home is not in India or Pakistan or wherever. They are here for good.”

For a while, though, it didn’t look as if such an outcome would be likely. In the community archive section of Smethwick Library, I flick through Peter Griffiths’ A Question of Colour?, which proves almost unreadable – a toxic tome musing on what Britain had to learn from apartheid South Africa. Another book, Paul Foot’s superb Immigration and Race in British Politics (1965), anatomises the Smethwick campaign in horrible detail. After Griffiths’ election, Foot reports, the residents of Marshall Street received race-hate material from groups such as the White Vigilante League, National Socialist and Keep Britain White. A group of Oxford undergraduates visiting Smethwick for the magazine Isis interviewed locals, finding 33% expressed bigoted views. “They want to do to the niggers what they did to the Jews,” said one woman. Perhaps Malcolm X’s fears weren’t completely unwarranted.

Daljit Singh Shergill, a leading member of Smethwick’s Sikh community and late father of local councillor Preet kaur Gill, talking with residents. Photograph: Preet kaur Gill

But Foot doesn’t just indict the racism of the time and Tory collusion with it. He also calls to account Labour corruption and cowardice. Foot castigated “the inability of the local Labour party, corrupted as it was by anti-immigrant sentiment, to hit back in a determined and principled way” against Griffiths and what he stood for.

Fifty years on, what are the lessons we can learn from what happened in Smethwick in 1964? For some, we have moved on to a better Britain, no longer hobbled by fear and loathing for immigrants or ethnic minorities. For instance, when Griffiths died last year, Paul Uppal, the Conservative MP for the Wolverhampton South West constituency that Enoch Powell represented in the 1960s, told the local paper: “I am glad to say that both Britain and the Conservative party have come an extremely long way since that infamous campaign in Smethwick in 1964. At the last general election, the number of Conservative MPs from an ethnic minority more than quadrupled going from two to 11, of which I was one.”

Smethwick-born film-maker Billy Dosanjh, whose father immigrated to Britain from the subcontinent in 1967 aged 14, isn’t quite so sanguine. “Characters like [Ukip leader] Nigel Farage and [ex-BNP leader] Nick Griffin are unbelievably similar to Peter Griffiths. In the 60s, immigration was associated by racists like Griffiths with bringing in disease. Now you have the same thing with Farage.” (Dosanjh means that Farage recently called on immigrants with HIV to be kept out of Britain.) “The difference is that while in 1964, Wilson dubbed Griffiths a parliamentary leper, today Cameron and Miliband are following Farage when they ought to be standing up to him.”

Smethwick’s Sikh temple, the Guru Nanak Gurwara, now dominates the town’s High Street. Photograph: David Sillitoe for the Guardian

Last Saturday, Dosanjh’s new film Year Zero Black Country was screened in Smethwick library. It’s a lovely, tender film that uses archive footage to create a portrait of the lives and struggles of those immigrants from the Indian subcontinent who came to the industrial heartland of England in the 60s and 70s (you can see if if you go to vimeo.com/90590680 and sign in with the password YearZero0000). Dosanjh tells me he is now developing the film with the BBC for a one-hour documentary.

Many of those who wanted to attend the screening in Smethwick on Saturday were delayed by the traffic jam that resulted when the English Defence League marched in Birmingham – a poignant reminder, says Dosanjh, that white racist Britain is hardly dead.

On the train home from Smethwick, I look at some old photos Preet kaur Gill emails me of her father, who died last week. Daljit Singh Shergill was president of Smethwick’s Guru Nanak Gurdwara, the first and biggest Sikh temple in Europe, which today towers over Smethwick’s High Street. There he is, in one snap, captain of the tug of war team, in another standing in his bus driver’s uniform in front of the No 6 to Bearwood, in another turbaned at a wedding, and in another in his lavishly bearded Sikh pomp sharing a joke with some white women. It’s impossible, for me at least, not to be moved by these images and the story they tell: the story of the people, like Shergill, who are too little hymned for making the Black Country and the rest of the West Midlands not just a little less white, but a lot better.

• This article was amended on 16 October 2014. An earlier version referred in error to “the late Avtar Singh Jouhl”. We have since been assured that Mr Jouhl is very much alive. Apologies to him and his family. It also said that in the 1964 general election Labour came to power in Westminster for the first time in 15 years; 13 years, that should have said. Finally, it erred in saying Patrick Gordon Walker “had seemed destined for a senior role in Harold Wilson’s Labour cabinet”. In fact Wilson appointed Gordon Walker as foreign secretary after the 1964 election despite his not having a parliamentary seat at the time; he stood in a byelection for the Leyton constituency in January 1965 but failed to win it, then resigned from the cabinet. Gordon Walker won Leyton in the 1966 general election and then served under Wilson as minister without portfolio and education secretary.