As holder of the ancient post of lord chancellor, responsible for courts and judges across the land, Derry Irvine had a knack for shutting down irritating interventions from his colleagues. During his time in office, from 1997 to 2003, Irvine’s fearsome reputation preceded him. Minor functionaries and even cabinet ministers would cower when the lord chancellor, who once compared himself to his 16th-century predecessor Cardinal Wolsey, felt they had not mastered their briefs.

Tony Blair might have thought that as the nation’s leader he would escape a lashing from Irvine. But “Young Blair” – Irvine’s name for the prime minister, dating back to 1976, when Irvine gave him his first professional opening as a pupil barrister – was briefly silenced during a fraught meeting early in Labour’s second term in which a worrying rise in the number of asylum cases was being discussed.

The gravest offence, in Lord Irvine’s eyes, was to call into question Britain’s solemn commitments on human rights, notably those made after the second world war in the European convention on human rights (ECHR). When ministers dared to broach the issue of drawing back from some aspects of the ECHR as a way of curbing asylum applications, Irvine’s response was sharp. “I don’t know why you guys don’t just adopt the Zimbabwean constitution and have done with it,” he told Blair and the home secretary, David Blunkett. The discussion was brought to a swift conclusion when Lord Goldsmith, the attorney general, pointed out that Britain would be in breach of its EU membership terms if it sought to wriggle out of its responsibilities under the separate ECHR. New Labour would have to find another solution to its immigration problem.

Immigration: let’s change the way we talk about it | Jonathan Freedland Read more

As today’s generation of political leaders prepares to fight an election that is in part a contest about the mistakes, judgments and assumptions Labour made in government on immigration, it is easy to forget just how much immigration and asylum haunted Downing Street throughout New Labour’s time in office. Between 1997 and 2010, net annual immigration quadrupled, and the UK population was boosted by more than 2.2 million immigrants, more than twice the population of Birmingham. In Labour’s last term in government, 2005-2010, net migration reached on average 247,000 a year.

The dramatic changes have left British politics ruptured. Immigration remains the No 1 issue on the doorstep, according to pollsters – a stream that feeds into the well of mistrust in politics. It has spawned the emergence of Ukip and helped create four- or five-party politics in the UK for the first time.

Despite the Conservative party’s recent travails over its broken pledge to bring net migration down to the low tens of thousands, the issue has been a special bind for Labour. In the wake of the party’s defeat in the 2010 election, there was a brief mass mea culpa about immigration, but even now Labour struggles to explain to a core part of its electorate the decisions that were taken on its watch.

Between 1997 and 2010, net annual immigration quadrupled, and the UK population was boosted by more than 2.2 million

Nigel Farage, the Ukip leader, has made capital out of his claim that the Labour government embarked on a deliberate policy to encourage immigration by stealth. Ukip often cites an article by Andrew Neather, a former No 10 and Home Office adviser, who wrote that the Labour government embarked on a deliberate policy from late 2000 to “open up the UK to mass migration”. Yet where Farage sees a political conspiracy behind the numbers, others veer towards the theory of history identified by the great 20th-century historian AJP Taylor, who always stressed the significance of chance events.

Even the most ardent defenders of the New Labour government acknowledge that such a wave of immigration was not purely down to chance. But the key players of the time show in candid conversations that they were struggling to cope with a new world of rapid population movement across porous borders. At times they felt they were stumbling from one move to another, unsure of the present, let alone the future.

Leap in immigration threatens Labour lead

As the head of Tony Blair’s policy delivery unit during his second term in office, from 2001 to 2005, Michael Barber did not get many chances to lie in on the weekends. One Saturday morning in February 2003, he took the liberty of sleeping until 8.50am. Ten minutes later, his phone rang. On the other end of the line was an anxious prime minister. Blair was once again fixated on the issue that had plagued his first term in office. “He was worrying away about illegal asylum applications,” Barber wrote in his diary.

In his new book, How to Run a Government, Barber recounts how he delivered the bad news: there had been a big jump in asylum applications as refugees from Afghanistan, Saddam Hussein’s Iraq and Kosovo headed to Britain. By Monday morning, having read the 1951 UN convention on refugees, Blair cleared his diary for the entire morning to allow him to spend five hours “forensically” going through the asylum caseload. The following day Blair summoned the relevant ministers to a meeting of the government’s emergency Cabinet Office Briefing Room (Cobra) committee, where he ran rings round most of the people in the room. Barber was deeply impressed with the detailed way in which Blair handled the issue.

Labour feared that the failure to grip the asylum challenge risked making the government look incompetent and – more damagingly – out of touch. “Immigration per se, but non‑European immigration [in particular], was a huge, huge issue for Tony Blair,” recalls Sir Stephen Wall, who was head of the Cabinet Office’s European secretariat between 2000-04. “I remember him saying, very soon after the 2001 election, ‘The one thing that could lose me the next election is immigration.’”

From the first year in office, the issue had hit the Labour government like a whirlwind. In 1997 net migration had been 48,000, but it rose extremely rapidly over the next 12 months, almost trebling to 140,000 in 1998. It was never to fall below 100,000 again.

Ministers and officials from that era recall in painful detail the apparently impossible task of dealing with the surge in asylum applications, as refugees fled to Britain. Almost every day, newspaper headlines would sneer at chaos in Whitehall as the Immigration and Nationality Directorate, later described as dysfunctional by ministers, struggled to cope.

Charles Clarke, who took over as home secretary in 2004, says that from Labour’s first days in office, the system for assessing applications for citizenship, residency and asylum claims was a mess. “We developed a massive backlog, particularly on asylum where we had cases waiting literally five years to be solved,” Clarke says. “That was the core problem that had built up behind an unmanageable set of issues. It was a complete nightmare and led to a sense of complete ungovernability of the whole system and that I think has undermined confidence in it.”

Facebook Twitter Pinterest Refugees near Calais in 2002 Photograph: Reuters

As the Blair government was consumed by the Iraq war in 2003, wiping domestic issues from the headlines, ministers at last started to turn around the numbers on asylum. The number of cases fell in the wake of the closure of the Sangatte refugee camp in Calais in 2002 and the introduction of greater immigration controls on the Eurostar in the UK and France. “The asylum claims … dropped like a stone by the time I left at the end of 2004,” says David Blunkett who served as home secretary between 2001-04.

But as Labour finally grappled with asylum, a separate and largely accidental process was under way, which was to have profound consequences a decade later.

EU enlargement and the big miscalculation

In 1999, an unlikely group of revolutionaries came together in the warren of nondescript offices and meeting rooms that make up the Cabinet Office. Appointed to the Performance and Innovation Unit, a small team of civil servants were given a vague and innocuous-sounding task: to assess some of the long-term strategic challenges for the UK. “It was your typical sort of civil service thing – four bright civil servants being asked to think big thoughts about the future of the UK,” remembers the unit’s then deputy director, Jonathan Portes, who as an undergraduate had studied maths at Balliol College, Oxford.

More on the best approach to migrant labour and to asylum seekers | Letters from Denis Macshane and Helen Bantock Read more

“We came out with a lot of obvious stuff,” Portes says. But there was one element in their findings that they had not expected at all. According to their analysis and many of the experts they spoke to, as globalisation gathered pace “immigration had suddenly become a big deal for the UK from an economic point of view – despite having been off the policy agenda in economic terms for the last 20 years or so at that point”.

Portes, who now speaks regularly about the benefits of immigration as director of the National Institute of Economic and Social Research, took his findings to his boss in the civil service. Portes managed to persuade him that they were so interesting that he should use the final six months of his time in the unit to research the economic benefits of immigration. The result was a 73-page report, published in November 2000, which claimed that the foreign-born population in the UK contributes around 10% more to government revenues than it receives from the state.

The civil servants in the Cabinet Office were enthusiastic about the report. But David Miliband, then the young head of the No 10 policy unit and a close friend of Portes, was less convinced. “We pushed this. It was not pushed by the politicians, and by politicians I mean David Miliband specifically,” Portes recalls. “It wasn’t that David didn’t think it was interesting, he just thought that the political costs and benefits of doing it were not positive.”

Miliband’s cautious approach challenges the claim that Labour presided over a secret conspiracy to increase immigration. But Blair picked up the report, which Portes says he regarded as “fantastic” – as did Barbara Roche, then the asylum and immigration minister, who had made a landmark speech in favour of immigration two months earlier, in September 2000.

“We were entering an age where a number of things were happening at the same time – you’ve got cheaper air fares, you’ve got mass telecommunications, it was a different world in which people travelled much more easily,” Roche remembers as she outlined one of the other key messages in her speech. This was to distinguish between asylum – a “precious concept” that should, in line with the Geneva convention, be granted to people in danger of persecution – and immigration, where Britain could impose controls.

Tale behind the UK’s open-borders policy on 2004 EU enlargement | Letters Read more

At the time Roche gave the speech, the fact that Britain could not control immigration from the EU was a relatively uncontentious issue. In the early years of the Blair government, income levels in most of the 15 member states were on a par with UK levels. Migration from the three poorer EU members at the time – Greece, Spain and Portugal, which joined in the “southern enlargement” of the 1980s – was relatively low, thanks in part to the generous EU funding of infrastructure projects in those countries.

In 2004, 10 more countries – eight of which had been part of the eastern bloc during the cold war – became members of the EU. (The former communist states were known, in the political jargon, as the A8.) In anticipation of the enlargement of the EU, Blair’s government took the precaution of asking academics to assess the likely levels of immigration from countries in central and eastern Europe that were noticeably less well off. Per capita GDP, as measured in purchasing power parity, of the eight new member states was less than half the EU average.

The report that was produced by the Home Office, published in 2003, did not predict a dramatic increase in immigration from Europe. The challenge for the team that wrote the report was a lack of data. The iron curtain meant that there was little recent history of free migration from eastern European countries. The authors therefore had to use Commonwealth countries, ranging from Australia to Swaziland, to make their forecasts. Based on their calculations, the report predicted that Britain would receive between 5,000 to 13,000 net immigrants per year averaged over a ten year period from the new member states.

The reality turned out to be quite different. The Office for National Statistic (ONS) estimates that between 2004 and 2012, the net inflow of migrants from the new members was 423,000. And this figure is expected to be eventually much higher when the 2011 census is fully analysed. Politicians mocked the authors when it became clear a few years later that the estimates had been wrong. (At one point in the paper, the authors had even characterised some of their calculations as “back-of-the-envelope”.) Blunkett, who was home secretary at the time the report was published, simply ignored it. “It was too low,” he recalls. “When they start talking about 13,000 you just start blanking out.”

In a memorable putdown, Chris Huhne told the Commons in 2008, when he was the Liberal Democrat home affairs spokesman: “As we know, Christopher Columbus thought that he had discovered India, when in fact he was in America. By comparison with the Home Office, he was a practitioner of pinpoint navigation.”

As one of the lead authors of the report, a weary Christian Dustmann has become used to the jibes and jokes. But he believes that his report was not as inaccurate as the critics suggest. “I think the problem is nobody really has read it,” says Dustmann, who is now a professor of economics at University College London. The projection of 13,000 net migrants per year over a decade, he explained, was based on the assumption that all 15 EU countries would open their labour markets to the newcomers, ensuring that the migrants would be reasonably evenly distributed across the EU. In the end, just Britain, Ireland and Sweden opened up. The other 12 member states, most notably Germany, exercised their right to impose “transitional controls”. “The German labour market was basically closed for Polish workers and that kind of changed everything,” Dustmann says.

Facebook Twitter Pinterest Eastern European migrant workers in Canary Wharf, east London. Photograph: Alamy

Indeed, tucked away on page 57 of the report is a careful qualification that a third of the 100,000 migrants predicted to head for Germany might travel to the UK instead if Berlin imposed restrictions on workers. Taking Berlin’s decision to impose controls into account, Dustmann says, allows him to add 33,000 to his prediction of 13,000 migrants, giving him an actual prediction of 46,000 net migrants per year. This is not far from the ONS figure of 50,000 per year, he says, though he acknowledges that the actual figure may be much higher.

Dustmann has had difficulty getting a fair hearing for his explanation. “I always said, over and over again, that these are net numbers, that it’s an average over 10 years and, most importantly, it was done under the assumption that Germany and other large European countries would also open up their borders. Nobody wanted to hear that and listen to that so it was very frustrating.”

Virtually all politicians now agree that the failure to impose transitional controls was a mistake. However, there are still commentators who are willing to defend the government’s record. Sir Stephen Wall, a former diplomat and EU adviser to Tony Blair, who is writing the official history of Britain’s relationship with the EU, believes there was a strong moral case for providing free access to workers from eastern Europe. “The primary argument was the political one – this was the right thing to do, we attached a lot of importance to them as democratic countries and keeping our position as the number one friend of eastern and central Europeans.”

In the immediate aftermath of the enlargement in 2004, Britain’s new EU allies helped the UK find its feet in the EU after the bitter split with France and Germany over the Iraq war the year before. Britain had been isolated in its support for the Iraq invasion among the 15 members of the EU at the time of the war. But the arrival in 2004 of the Atlanticist new member states – referred to as “new Europe” by the US defence secretary Donald Rumsfeld – evened the balance. The diplomatic benefits were clear. But all was not well on the home front.

The warning that went unheeded

As an unassuming figure at Westminster, John Denham tried to make as little fuss as possible when he resigned from his post as policing minister in 2003 in protest over the Iraq war. In the months that followed, the former charity campaigner focused on his new position as chairman of the House of Commons home affairs select committee and devoted even more time to his constituency, Southampton Itchen.

With the help of information from local council officials and union shop stewards, Denham spotted earlier than others the impact of immigration from the eastern and central European states. In February 2006, he wrote a prophetic memo to Tony Blair, chancellor Gordon Brown and home secretary Charles Clarke, warning that the number of immigrants was far higher than the government’s figures suggested. According to the memo, roughly 14,000 eastern European immigrants had arrived in Southampton within the previous 18 months. This was placing immense pressure on maternity services and leading employers to lower wages.

The response from the Whitehall machine to Denham’s memo was largely indifferent. “I sent a warning message to government about the impact of immigration in Southampton, saying Whitehall was not picking up quickly enough what was happening on the ground, or what the wider electorate were saying in response,” he recalls. “To be fair to government, it was probably true the impact at the time varied enormously from area to area, and there was real uncertainty about how long the impact would last. The reaction to migration was seen very differently in London, for instance, to other places.”

Outlining the impact on the everyday lives of his constituents, Denham argued at the time that resentment of immigration would grow. “One of the problems was that people were supposed to register if they were employed but many came as self-employed,” Denham says. “The biggest impacts were in self-employed trades like construction, where you didn’t have to register.” In the memo, Denham stated that the daily rate for a builder in the city had fallen by 50% since 2004. He also noted that hospital accident and emergency services were under strain because migrants tended not to use GPs as a first port of call. It also turned out that the local further education college had to close its doors after 1,000 migrants attempted to sign up for an English-as-a-second-language course on one day. Whitehall, Denham argued, was wholly out of touch with the concerns of his constituents. The government needed a comprehensive assessment to work out how it should deal with the surge in immigration.

Looking back on the memo, as he prepares to stand down from parliament at the election in May, Denham says that Blair’s intense focus on asylum was not matched a few years later as Britain dealt with migration from the new EU member states. “The research looked well founded and evidence based,” he says of the Home Office predictions. “It is what government is supposed to do. The whole irony of this is that in some respects Tony Blair was obsessed by immigration, particularly about illegal immigration and abuse of the asylum system, but on EU migration there was a catastrophic failure of the civil service machine.”

Jacqui Smith, who was home secretary between 2007 and 2009, when the financial crisis began, agrees with Denham that Whitehall appeared out of touch. “I can remember seeing Treasury papers that said if we limit migration we will reduce our growth. There is a justification for that argument at a macro level, but if you say to people, ‘You may have seen some changes on your high street but it’s OK for macro-economic growth and there will be cultural benefits in London, you sound like you don’t get it,” says Smith. “The person that benefits from that growth is often not the one worried that their kid is going to get less attention from the teacher in the classroom because there are people speaking Polish. At the very least you have to reassure people that you get that there are impacts from migration.”

David Blunkett, home secretary from 2001 to 2004, recalls that No 10 issued uncertain signals. “It changed almost monthly, so there would be, ‘Yes, we’re a global player, we believe in a labour market – if there’s going to be free movement, well, we might as well have the right to work rather than the right to be here.’ Then the next minute, ‘What does this mean? Have we got worries about the BNP?’ – bearing in mind that from 2000 onwards rightwing parties across Europe were actually becoming quite predominant.”

Facebook Twitter Pinterest Barbara Roche, a Home Office minister from 1999 to 2001, was one of the first Labour ministers to talk about immigration. Photograph: Frank Baron

Barbara Roche, who was Home Office minister from 1999 to 2001 – and one of the first Labour ministers to open a discussion about immigration – remembers how divisive the issue could prove on the progressive left. “It really was difficult to get the space to do anything politically and create some middle ground,” she recalls. “If we think the debate now is polarised, and so it is, it was even more polarised then.”

Ministers from the time believe that mistakes were made but they insist that the fundamental decisions were right. “I’m not one of those who thinks that the A8 countries being able to come into EU was economic or social disaster for us,” says Charles Clarke. “It’s nonsense to suggest it has terrible impact on public services – most of the people coming here were working, didn’t have children, and were not of age to be a big burden on NHS.” Blunkett largely concurs: “Of course we should have predicted the figures better, but I’m unrepentant about the decision to allow people from the A8.” The alternative, Blunkett says, would have been more illegal immigration, which would have had a far more negative impact on pay rates and standards of work. “I’m confident that it was the right decision, even though very few people now will stand up for it,” he says. “I just think that allowing people to move [to the UK] but not to work would have been a complete own goal.”

Sir Stephen Wall, who was Blair’s most senior EU adviser between 2000-2004, frankly admits an error. “We simply didn’t take account properly of the pull factor of England for people with skills who could probably find a bigger market [in the UK] for their skills – you know, the Polish plumber.”

Either way, it took the government a long time – the final two to three years – to realise the extent to which EU immigration had reshaped British politics. Jack Straw is frank about the failures, particularly the official projection of a mere 13,000 net migrants a year from the new EU member states. “It’s a case study of how good intentions and apparently good research can lead government in the wrong direction. But it was a very significant policy failure, done with the best of intentions and in a serious way because we’d got the research.”

The aftermath

By the early summer of 2007, the Blair era was coming to an end, leaving the Labour party to look nervously towards a fresh start under Gordon Brown. The need to draw a line under the legacy of the Iraq war – and to chart a new course on public service reform – dominated the thinking of the Brown circle as they prepared for the handover. But, alarmed by the success of what was seen as a dogwhistle campaign on immigration by the Tories at the 2005 general election, Brown also set in train plans to make a decisive break with the Blair era on immigration.

As Brown presided over his first Labour party conference in the autumn of 2007, a surprisingly long political honeymoon, dating back to his assumption of the premiership in late June, was juddering to a halt. Brown helped set in motion the decline in his fortunes, which ultimately led to Labour’s election defeat in 2010, when, in his address to the Labour conference, he pledged to create “British jobs for British workers”. He was widely mocked for this response to the concerns raised a year earlier by John Denham in his memo.

Brown helped set in motion the decline in his fortunes when he pledged to create ‘British jobs for British workers’

Brown’s remarks had one major flaw: there is no basis in law, under the terms of Britain’s EU membership, to discriminate in favour of British workers. But the speech showed that, right from the start of his leadership, Brown was grappling with issues that would come to haunt Labour in the years ahead.

Blunkett blames the global meltdown from the end of 2007 for making the issue of immigration so toxic for Labour. “Back in 2004 and all the way through to 2008 the economy was able to absorb young, able skilled workers who wanted to work and wanted to stay in the country for a short period of time while they made enough money to invest back home,” he says. Once the recession set in, that was no longer the case.

“The big mistake we made was not to put more money into integration and into preparation for people being dispersed either under the asylum programme and where there was a large diaspora coming in from countries like Afghanistan or Iraq. We invented the Migration Impacts Fund [in 2009] which the coalition abolished in June 2010, as well as the UN Gateway for refugees, but it was not enough.”

The 2009 European parliament elections, when Labour came third behind Ukip, showed how British politics had changed. By then, senior figures were reflecting on where they had gone wrong.

Straw acknowledges that during Labour’s time in office the political landscape shifted underneath their feet. “It’s much wider than just immigration. Somebody said to me recently, about the challenge Labour faces in the west coast of Scotland, ‘It’s not about politics, it’s about feelings. And they’re much more difficult to deal with.’ I think the same is true here. It’s about feelings, particularly for sections of society who feel they haven’t had a fair deal in life and so look for outlets for this frustration; such as the system and the parts of the community they perceive to have been dealt a better hand than they have.” Suddenly, around this time in 2009, the cabinet concluded – on a split decision – that it had to change tone on immigration, as ministers recognised that the pace of change was an issue in some localities and for some people the numbers mattered. Jacqui Smith, who attended this cabinet meeting, emphasises the point: “I used to tell the cabinet, ‘There’s an enormous distinction in the way people feel about immigration between those who live in London or other big cities and those who live in places like my old constituency of Redditch. I don’t think you understand the cultural and emotional impact of sudden change. This isn’t people being racist, but they see their country changing.’”

Facebook Twitter Pinterest Gordon Brown talks to Gillian Duffy during the general election campaign in 2010. Photograph: Jeff J Mitchell/Getty Images

By late 2009, Gordon Brown was making speeches saying: “People ask me, ‘Do you get it?’ Yes, I get it.” A year later, however, Brown was caught on microphone during the general election referring to a Rochdale pensioner – and lifelong Labour supporter – named Gillian Duffy as a “bigoted woman” after she raised concerns about immigration. Brown’s painful experience dismayed many in the Labour party who understood that concerns over immigration could only be tackled by addressing people’s fears at an intuitive level. This reflects the views, if not the policy prescriptions, of the Blue Labour thinkers, such as the academic and peer Maurice Glasman, who believe that the party can only ever hope to win back traditional working-class supporters if it addresses deep-seated fears on sensitive matters such as immigration.

Ed Miliband knows immigration can be a potent issue. But he tends to lean towards dry policy solutions, such as ensuring that employment agencies cannot discriminate in favour of migrant workers. If Labour is to reconnect with its traditional supporters, it may have to think how to respond to those who, in the words of Jack Straw, feel they have not had a fair deal in life as the country changed around them after a series of largely accidental steps.

Writing in 1883 on the growth of the British empire, John Robert Seeley, professor of modern history at Cambridge University, famously argued that “We seem … to have conquered and peopled half the world in a fit of absence of mind.” More than a century later, one of the largest mass population movements since the second world war brought more than two million immigrants into the United Kingdom – and in turn radically transformed the terrain of British politics. It would seem, in hindsight, to have been another fit of absence of mind.

Follow the Long Read on Twitter: @gdnlongread