Alastair Campbell has admitted that his newly published diaries show clearly how New Labour sowed the seeds of the UK’s vote to leave the European Union by failing to respond to growing concerns over immigration.

The former Downing Street director of communications said he feared the party became complacent on the issue due to Tony Blair’s electoral success, allowing the debate to shift in favour of a pro-Brexit vote in the EU referendum years later.

The diaries, covering between 2003-2005, also reveal the then chancellor Gordon Brown had growing concerns that Labour had not done enough to change opinions, complaining that Blair was helping the Tories under Michael Howard to turn immigration into an electoral issue.

Campbell told the Guardian: “We were more and more aware of the problem politically but there was always a tension between knowing that the economy and public services needed immigration but knowing the issue was causing real concerns. I think the fact that we won two elections in 2001 and 2005 despite the Tories campaigning on immigration may also have made us complacent. Just as in Scotland people started to feel Labour support was taken for granted so in areas of high immigration I think some Labour voters started to feel the same.

“I think deep down we always felt despite the difficulties we would be able to persuade people of the benefits of immigration and the benefits of the EU. We did to a large extent but where we are now, on both issues, suggests that we did not cement the political views we were putting forward.”

The diaries, Campbell’s fifth volume and available from Thursday, record Blair’s pollster Philip Gould raising immigration as a key factor in loss of support for Labour as early as 2004.

In one extract in January that year from the book, Outside, Inside, which is being serialised by the New European newspaper, Campbell writes: “PG was worried we had three groups of people we were losing: anti-war middle-class and younger voters shifting to the Libs because of Iraq and tuition fees; suburban middle-class, some of them moving to the Tories over more general issues; and working-class voters switching to disengagement because of asylum and immigration in particular, and the feeling they are getting a raw deal.

“Asylum was definitely linking up all our negatives now, from anything bad on the economy to crime, pressure on public services, terrorism, Europe. There was an opening for populism that we had to watch out for.”

Campbell’s diaries further reveal that in 2005 he asked the then Manchester United manager, Alex Ferguson, to join up with the Arsenal manager, Arsène Wenger, to attack the Conservatives’ hardline immigration policies under leader Michael Howard. One diary extract reads: “Alex was interesting on the Howard stuff. He felt he had a point but that every foreigner would hate it. I said he and Wenger should do a joint presser saying they would have to send all their players back to their own countries under a Tory immigration policy.”

It is also documented that it was Brown’s belief that Labour under Blair, whom he was agitating to replace, was giving the Conservatives scope to make immigration a key issue. An extract from February 2005 reads: “[Gordon Brown] said Bush was dominating world politics, a rightwing ideologue. But he was pretty down about [Bill] Clinton, with obvious resonances to [Tony Blair]. ‘He won two elections, he did some good things, and was popular at times, but he didn’t change the country’s direction for good.’ I argued that we had done a lot to change the country, not least thanks to him, but he said: ‘Come on, you don’t think it’s enough and nor do I.

“‘We have not changed the country as much as we could and should. Look at the media 80% come at politics from a rightwing agenda, and we haven’t changed that. Look at how easy it is for the Tories to make immigration the issue, and we help them.’”