Few chance encounters have had a greater political impact than Gordon Brown’s fateful meeting with Gillian Duffy on an April morning in Rochdale in 2010. When the then prime minister was caught on a hot mic calling the Labour-voting pensioner a “bigoted woman” – after she cornered him with complaints about immigrants “flocking” into Britain – it did not just sink his floundering campaign. It set the tone for the way immigration would become the most toxic issue in British politics for the decade to come.

When New Labour came to power in 1997, just 3% of the public cited immigration as a key issue. By the time of the EU referendum in 2016, that figure was 48%. During those intervening years, the issue came to dominate and distort British politics – exactly according to the script established by Bigotgate. Brown’s gaffe both consolidated and gave credence to a political coding that would shape everything that came after: the “hostile environment”, the Windrush scandal, the EU referendum and the revival of Britain’s far right – deploying a narrative in which sneering, out-of-touch, big-city politicians who favour foreigners and open borders are hopelessly oblivious to the struggles and the so-called “legitimate concerns” of ordinary working people (who, in this scenario, are always white).

The Labour movement still bears the battlescars of Bigotgate, as divisions over immigration cut across party factions. With the Labour party about to release its election manifesto, an outburst against free movement of migrants from the leader of Unite the Union, the party’s biggest affiliate, threatens to thwart efforts to move to a more progressive position on the issue.

By the time Brown took his ill-fated walkabout in 2010, immigration was already polling as a top concern, overtaking the NHS and only topped by the economy for a few years following the 2008 crash. By then, politicians and the media had shaped a venomous narrative around the volume of new arrivals to the UK, the identity of those coming and why they did so.

For more than 30 years, the UK has been at the forefront of a neoliberal economic globalisation that deregulated markets and pared back the state – deepening the global economic inequalities that produce labour immigration. In the UK, local manufacturing-based economies that supported entire communities were shut down or shipped out. Across the global south, structural adjustment programmes imposed by the west as loan conditions were collapsing national economies. But these harsh economic realities did not appear in our media’s fearmongering accounts of a “tide” of migrants “flooding” into soft-touch Britain on supposedly spurious grounds and overwhelming the country.

Just as communities were exposed to the shocks of an unrestrained free market and a shrinking state, they were simultaneously bombarded with stories about “Slovak spongers” and cheating Czechs. Politicians of all stripes fell in line, producing hostile rhetoric and policies in response – and defining the issue as a reflection of supposed concerns over the exact number of arrivals, the “pace of change” in local communities and the need to exert control over migration. This was the catalyst for David Cameron’s foolish 2010 election pledge to introduce a target figure for “net migration” – which the Conservative party failed to meet, again and again, only enflaming public resentment and mistrust over the issue (this impossible target underpinned the Conservative’s hostile environment policy and produced the Windrush scandal).

Facebook Twitter Pinterest Bigotgate: Gordon Brown talking with Rochdale resident Gillian Duffy in 2010. Photograph: Jeff J Mitchell/Getty

In the run-up to the EU referendum, called by Cameron to block the popularity of Ukip and silence Tory Eurosceptics, the leave campaign repeated the idea that it was Labour’s policy to increase immigration that had made it a burning issue in British politics. More specifically, the blame was focused on a commitment made in 2004 by Tony Blair as prime minister. In a 2015 BBC documentary called The Truth About Immigration, the presenter Nick Robinson talked of immigration in the UK as an issue that “adds up to an awful lot of change and that’s led to an awful lot of anxiety”. He identified the source of all this worry: “A single decision taken without much thought in Downing Street, to allow immigration from new members of the EU from eastern Europe.” That decision by the Labour government would, Robinson said, likely be debated by historians in 50 years’ time as the “most significant taken since the second world war”.

According to this analysis – which was widespread across the media – what created the problem was the arrival of immigrants in larger numbers, not the way immigration was depicted, described, debated or demonised.

But what if this narrative is the wrong way around? Perhaps it wasn’t immigration itself that was such a defining issue of those 20 years – but rather, the way political parties and journalists discussed it and the policies implemented in response. The big assumption is that it was a foregone conclusion that there would be hostility to immigration, which in turn would become politically explosive in the UK. While Britain has always received migrants with initial suspicion, it was not inevitable that the issue would become so damaging or derail our politics so comprehensively.

By the 1990s, conflict across the world began to drive migration. Responding to the earliest of the 1980s arrivals, parts of the British media labelled Tamils fleeing Sri Lanka as “bogus” asylum seekers. This term was adopted by politicians to cast suspicion on asylum seekers and focus anxiety about immigration on fair play and the abuse of British generosity.

During this period, Conservative and Labour ministers presented their asylum policies in this fashion, premised on preserving the integrity of the system. The Conservatives in power during the early 90s ramped up rhetoric against “floods” of asylum seekers, even though Germany then had the highest numbers of asylum seekers in Europe. Tory legislation sought to hasten the “rapid rejection of a large number of unfounded claims”, which would “play a major part in deterring further abuse of the process”, in the words of Kenneth Baker, the Conservative home secretary between 1990 and 1992.

By the time Labour came to power in 1997, newspapers were routinely scaremongering about the new arrivals: that year the Daily Mail ran a story about a “flood of bogus asylum-seekers swamping Dover” while the Independent warned: “Gypsies invade Dover hoping for a handout.” The actual numbers were tiny. There were 32,500 overall asylum claims in 1997, with 81% refused asylum. In the decade to 2000, the UK accepted 1.9% of asylum claims from Sri Lanka; in France that figure was 73.6%.

The Labour government scrapped the rules barring spouses of British citizens from joining them in the UK and removed restrictions discriminating against homosexual and other long-term relationships. But things did not continue in this hopeful vein. The 1999 Asylum Act formalised the use of detention centres as a routine administrative measure rather than an exception, and also replaced cash benefits for asylum seekers with vouchers, since cash benefits were, in the words of Labour’s home secretary Jack Straw, “a major pull factor that encourage fraudulent claims”.

At the same time, the asylum dispersal system sent claimants, without a choice, into deprived parts of the country. This created the perception of a burden to be shared, while cutting newcomers off from legal and translation services. As resentment grew, asylum seekers faced harassment and physical attacks in areas where the media and politicians whipped up fury at their arrival.

Facebook Twitter Pinterest Jack Straw (right) in 2000, then the home secretary, watches as a stowaway is helped down from a lorry at Dover docks. Photograph: Sean Dempsey/PA

By 2002, asylum claims – most of which were coming from Iraq at that time – had peaked at just over 80,000 (it has not gone higher since), although Britain did not rank in the top 10 receiving countries in Europe in the number of claims per population. Still, Tony Blair pledged to halve the number of asylum seekers in the UK by September that year. Labour’s commitment to such reductions indicated they were prepared to believe that at least half of all claims were not valid. In 1998, at the height of media hysteria against Roma arrivals, Straw had claimed that not one of their asylum cases had been recognised as genuine. In 2001, the Home Office had sent immigration officers to Prague’s Ruzyne airport to discourage mostly Roma asylum claimants from boarding planes to Britain. Three years later, the law lords ruled that this practice amounted to “inherent and systematic” racism.

In 2001, then home secretary David Blunkett wanted to force asylum claimants to carry ID cards. A year later, Labour proposed the children of asylum seekers should be taught separately, saying they were “swamping” classrooms. Both plans faced massive opposition and were dropped. The 2002 Nationality, Immigration and Asylum Act limited access to legal aid and removed benefits from those claiming asylum later than three days after arrival. The act also removed the right to work while awaiting a claim – employment was considered another “pull” factor. And the 2004 Asylum and Immigration Act removed benefits from families whose asylum cases had been rejected: trying to “starve them out” as the Labour MP David Winnick protested at the time.

The Labour leadership held that, if the system could be shown to be fair, hostility to asylum seekers would diminish. Barbara Roche, Labour minister for immigration for two years until 2001, stands by this formulation: “If you aren’t rigorous about keeping asylum as what it’s for, you lose the concept,” she says. “I feel passionately about protecting it. So yes, there were some things that were made tougher.”

The trouble is that foregrounding “fairness” fuels public assumptions that immigration is somehow unfair or deceitful. Immigration lawyers describe a system premised on disbelief and a tendency to view migrants from wealthy white nations more favourably. As former barrister Frances Webber chronicles in her book, Borderline Justice, asylum seekers during that period were routinely assumed to be lying. In 2000, one asylum seeker was sent a letter from the Home Office stating: “The secretary of state believes your claim to be a pile of pants.”

Advisers of the time insist Labour had no need to make policy to placate tabloid readers. One top strategist for Blair says: “A government with a 150 seat majority is making the weather.” But a Home Office special adviser from that time says half of each day’s media cuttings were about asylum seekers. “There would be pictures of people trying to get on to lorries and trains and cutting fences … it was incredibly intense, you were under constant scrutiny, always in court, always assailed by forces from all sides.” None of that, he notes, made for an environment conducive to rational policy-making.

By this stage, little about the conversation on immigration was rational. In 2003, the Sun ran a front page fake news story under the headline Swan Bake, in which it claimed eastern Europeans were barbecuing swans pulled out of an east London park (following complaints, the paper did not apologise, even though there was no evidence to support this assertion). In a 31-day period in 2003, the Daily Express ran 22 front page stories on the subject.

In both policy focus and language used, Labour had added to this mounting impression of migrants and refugees as “sneaking into the UK”, “cheating” the system and “living off benefits”. Advisers of the time acknowledge that a negative view of asylum was seeping more widely into hostility to all migrants. The hope, according to one home office adviser, was that “if people saw we were dealing with abuse but also improving the channels for legal migration, we could get public trust back”.

But this is not how public sentiment works. “As we found out after 2004, that wasn’t the case – they simply moved on to Eastern Europeans [arrivals from new EU member states] as a focus of concern,” the adviser said. Charting the rising number of race-hate attacks around that time in cities from Stoke to Plymouth and Peterborough, IRR reports in 2012 show that migrants who were not asylum seekers, as well as black and ethnic minority British people, often faced hostilities exacerbated by asylum dispersal, tough political rhetoric and media scaremongering.

By 2004, 26% of the population saw the number of immigrants and asylum seekers coming into the country as the third most important voting issue, after public services and law and order. That same year, with cross-party support, Labour opened the borders to migrants from eastern Europe’s A8 countries: the Czech republic, Hungary, Latvia, Lithuania, Estonia, Poland, Slovakia and Slovenia. To this day, this is cast as a supposed mistake, with former and current Labour MPs lining up to apologise for it. In 2012, Labour leader Ed Miliband said immigration was “one of the areas we got wrong” and spoke of “local talent” being “locked out of opportunity”. And in 2013, Straw, the former home secretary, described it as a “spectacular mistake”.

Get the Guardian’s award-winning long reads sent direct to you every Saturday morning

Ahead of the A8 accession, a government report predicted that up to 13,000 additional migrants would arrive annually from those states. It was way off: in 2005, 129,000 migrants from the A8 entered the UK.

It was later painted as a crisis, but in truth, the British economy needed even those unexpected numbers. David Blunkett tried at the time to explain that migrants would in any case be coming to Britain in high numbers and it was better to have them in the labour market legally, paying tax. Study after study has showed that EU migration, in particular from A8 countries, produced a net economic gain for the UK. According to one authored by economists Helen Lawton and Danny Blanchflower in 2008: “The fact that the UK opened its borders to a flow of highly skilled, motivated, educated, low cost mobile workers upon EU-enlargement was a stroke of genius, for which the UK government should be given credit.”

Although the economy was booming through the 90s and early 00s, it was finance- and debt-fuelled and not equally enjoyed across the population. The social geographer Danny Dorling has dubbed Blair “the king of inequality”, observing that Labour from 1997 to 2010 presided over a period during which inequalities in income, health and wealth actually rose.

Don Flynn, the former director of the Migrants’ Rights Network, says: “It wasn’t that a group of politicians had taken their eye off the ball and immigrants had come through. It was that the immigration was the product of the type of economy created after the 1980s, with 80% of people working in a service industry of one sort or another, where the fastest areas of economic growth were London and the south-east, where a 24-hour, seven-day-a-week economy needed flexible workers doing antisocial hours in precarious, zero-hours, relatively low-paid jobs. And natives were going to do their level best to avoid those.”

While Labour in 2004 had underestimated the number of arrivals from new EU member states, the tabloids wildly overestimated those figures. On the eve of accession, columnists in the Daily Mail and the Daily Telegraph respectively predicted that 4 million and 4.4 million migrants would come to Britain. Channeling this alarmism, the Conservatives under Michael Howard (the son of a Jewish refugee from Romania) ran campaign billboards during the 2005 proclaiming: “It’s not racist to impose limits on immigration … Are you thinking what we’re thinking?”

As new arrivals settled outside of big cities, tensions over asylum dispersal and scaremongering over jobs and resources boiled over into racially motivated attacks. By 2006, some of the country’s most neglected areas were targeted by the British National party, which made gains in that year’s local elections.

At the same time, the idea that multiculturalism had failed – which had been percolating on the far-right – now spilled into the mainstream. The suggestion was that the left’s supposed promotion of diversity had failed to foster integration and disadvantaged the white working class. It set the tone for the BBC’s White Season, a documentary series that came out in 2008 and asked: “Is white working class Britain becoming invisible?” Trevor Phillips, then head of the equalities and human rights commission, echoed this sentiment when he said that the white working-class would need help in the economic recession, noting: “The colour of disadvantage isn’t black or brown. It is white.” By this reckoning “white working-class” was a minority, or protected group, that had hitherto been overlooked.

Describing loss and disadvantage in these cultural terms gave rightwing attackers a way to claim multiculturalism in itself licensed an indifference to rising immigration and a clampdown on discussion of the subject. It all dovetailed into the accusation that Labour had abandoned its working-class heartlands.

Labour politicians were also feeding this debate, concurring that they had not listened to “legitimate concerns” over migration. Earlier in 2009, MPs had claimed that the “pace of change” was too fast, or that, in the words of the home secretary at the time, Jacqui Smith, the “cultural and emotional impact” had not been properly understood. That same year, the immigration minister Phil Woolas said there was a need to find jobs for the “indigenous population”. Communities secretary Hazel Blears said: “White working-class people living on estates sometimes just don’t feel anyone is listening or speaking up for them.”

Facebook Twitter Pinterest Len McCluskey, general secretary of Unite the Union (centre) and delegates at the Labour party conference in September. Photograph: Gareth Fuller/PA

In January that year, a government report found that the white working-class felt ignored over immigration. Those in deprived areas, without much direct experience of immigration, felt “a real and perceived sense of unfairness” over the issue, coupled with a reluctance to speak out for fear of being branded as racist. Looking back now, it is striking that this report gives equal footing to “real” and “perceived” concerns – the idea that a sense of unfairness, not borne out by fact and misattributed to migrants, should be politically heeded. Four years after the Conservatives were criticised for using it on election campaign billboards, Gordon Brown said in a speech in November 2009 that it was “not racist” to talk about immigration.

Blue Labour – a faction positing that the party’s social and economic liberalism had led to a drop in working-class support and pushing social conservatism as the solution – emerged in 2009. A substantial number of the 4m votes Labour lost between 1997 and 2005 were from its working-class base. But some factions in the party explored other elements, such as economic alienation and New Labour’s disconnected, managerial style of politics. Diane Abbott says: “We moved post-Thatcher to this post-industrial society and [New Labour] really didn’t have a strategy to deal with it,” she says. “People were being affected economically, seeing their lives and communities shattered … and we didn’t help, we didn’t have an alternative.”

Labour lost the 2010 election and saw the Conservatives and Lib Dems form a coalition government. In opposition, Brown was not allowed to forget “Bigotgate”, and Labour was hounded by the accusation that it had shut down a conversation about immigration – even while the subject was self-evidently dominating politics. This charge was coming not just from the Conservatives, but also from inside the party. Standing as candidate for party leadership in 2010, Andy Burnham claimed: “There’s still an ambivalence among some in Labour about discussing immigration.” Both these themes dominated the 2015 election. One adviser to Ed Miliband, Brown’s successor as Labour leader, recalls the overwhelming media narrative of Labour as a party that “crashed the economy, opened the floodgates to immigration, was led by someone who looked nerdy and stabbed his brother in the back”.

“We spent five years being shit-scared,” says one of Miliband’s key advisers. “If we had the time again, I would force Ed to do a speech on the fallacy of blaming immigration for the reality of austerity cuts.” Instead, Labour produced nebulous pledges over a caring NHS and higher wages for working families. One of those – “Controls on immigration” – ended up on a party promotional mug, a dismal sign of a rightward shift in slogan form.

In March this year, an Ipsos Mori poll showed the public was more positive than negative over the impact of immigration on Britain. One in five said they had become less negative on the subject. When asked why, 51% said it was because discussions in the past few years had highlighted how much migrants contribute to the UK. This is an encouraging shift for those who have long argued that, had the left in British politics articulated a robust defence of immigration, this could have detoxified the public narrative.

Records of parliamentary debates show objections – roundly ignored – to New Labour’s hostile and punitive immigration policies dating back to when it took office in 1997, often raised by then backbenchers Diane Abbott and Jeremy Corbyn. Bill Morris, then secretary of the Transport and General Workers Union (the precursor to Unite), spearheaded a campaign against asylum vouchers, which led to them being scrapped in 2001, albeit to resurface in 2006. And in 2003, a Trade Union Congress report argued that the “blurring by media and governments of the distinction between refused asylum seekers, illegal working, illegal entry and criminal activity such as trafficking” was feeding a general suspicion around all migrants.

In the early years of Blair’s government, it was riding high on a historic victory, while the prime minister enjoyed extraordinary popularity, support from the Sun and the Daily Mail. Abbott, among others, views this period as a missed opportunity to reset the terms of the debate. “Tony Blair could have used his considerable persuasion in middle England to make a different case about the role of immigration and what causes asylum seekers,” she says. “He was in a very good position to change the conversation.”

But New Labour did not want to go against public sentiment. “There was a sense of real public hostility,” recalls a special adviser to the Home Office at the time. “And we couldn’t just turn around and say to people: ‘Your hostility is misplaced and wrong.’” This exact same instinct is raised by one of Miliband’s advisers from 2010 to 2015: “You can’t have your first sentence [to the public] being ‘You’re wrong about the thing that you’re thinking, the thing that you’re worried about.’”

Political scientists Anthony Geddes and Jonathan Tonge have written of a “ratcheting effect” whereby tough words and policy on immigration gives it more salience, creating, in turn, a public hunger for tougher words and policy. As Abbott puts it: “If the Labour party is banging on about immigration, our people feel legitimised in banging on about immigration. If we turn their attention to other things, they stop feeling as though it is the most terrible thing affecting their lives.”

Facebook Twitter Pinterest ‘Labour under Corbyn has changed the debate on austerity, wealth redistribution and renationalisation ... but has hardly rushed to steer a similar shift on immigration.’ Photograph: Chris J Ratcliffe/Getty

That politicians and tabloids are no longer “banging on” about immigration – certainly not at the fever pitch of the years prior to the EU referendum – may partly explain why immigration has dropped in prominence as a public concern: it now ranks just below the environment as the sixth biggest issue of the December general election. Brexit, which tops this same YouGov poll from last week, may for some be acting as a proxy for immigration. But perhaps the Windrush scandal has also had an impact, as well as staff shortages across health and social care, hospitality and farming. Labour’s own internal polling shows little public disagreement with the statement that: “Politicians blame immigrants or people on benefits to distract from their own failures.”

While Labour under Corbyn has changed the frame of debate on austerity, wealth redistribution and renationalisation, it has hardly rushed to steer a similar shift on immigration. A rare window to do so opened amid the public outrage over the Windrush scandal, where a generation of British citizens were caught in the government’s hostile environment dragnet, resulting in deportations and denied access to housing and healthcare. Amid a discussion that was unusually sympathetic, Labour MPs often ending up reinforcing stereotypes over “good” and “bad” or “illegal” immigrants – cementing the idea of a system that had failed to filter properly, rather than widening the frame to explore how an entire process was premised on suspicion and prejudice.

Meanwhile, the leadership’s strong credentials on immigration – earned over decades of showing up to support various cases and causes – has been dented by Labour’s pledge to end free movement. Grassroots dismay over this went beyond the need to defend an important principle, as the party was seen as closing off terrain that should have been used to make the pro-migration arguments that had been so lacking during the New Labour years. Triangulating over Brexit, the party has shown it is streaked with the same political reluctance over immigration that has long plagued Labour. Even elements that support the Corbyn project have misdirected a need to represent the working class into language that fuels populist-right frames of hostility to immigration. That much was clear in the recent intervention by the Unite leader, Len McCluskey, urging the party not to back free movement. This in turn prompted an exasperated backlash from campaigners and senior Labour figures alike, with one noting that McCluskey was “part of a backward-looking, small-c conservative nostalgia-tripping wing of the labour movement that wants ‘trade union rights for British workers’”.

'A dizzying maze': how the UK immigration system is geared to reject Read more

As the party prepares to launch its manifesto, campaigners are watching to see whether the immigration motion passed at Labour’s annual conference in Brighton in September, including a commitment to freedom of movement, will be honoured. Labour’s readiness to put the idea of EU citizens having full voting rights to parliament just weeks ago suggests this part of the motion could be adopted as policy. Last week, Corbyn said that free movement “enriches the lives of all of us”, crucially shifting into language that goes beyond economic contribution. “I want to make sure all those European Union nationals do remain here, can come here, will stay here and we will be happy to work with them as indeed many British people have made their homes in other parts of Europe,” he said. The latest YouGov polling shows 67% support for the retention of reciprocal free movement rights post-Brexit, something that Labour is now likely to commit to.

A source close to the leadership says the shift is down to “a lot of hard work internally and pressure from the membership”, adding that a softening of public attitudes to immigration and the Windrush scandal have both had an impact on MPs who might previously have hesitated over pro-migration policies. But the party’s own history on immigration suggests that it may take time to embed significant changes in rhetoric and policy. Even with a leadership that is theoretically supportive, there are likely to be wobbles along the way.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.