It wasn’t the most lavish of birthdays for Jeremy Corbyn on Friday. Preparing for a morning speech on security was followed by preparations for a 30-minute grilling by the BBC’s most feared interviewer, Andrew Neil. It was only in the evening that he had some downtime for a typically understated celebration of his 68th birthday, at home with his family.

Yet even by the time he was consulting his speech notes over breakfast, the Labour leader had already been given a gift – a YouGov poll that appeared to not only confirm the trend that the polls were closing, but that put Labour only five points behind a Conservative party that had at one point surged into a 24-point lead.

The reactions within Labour varied hugely, from joy and vindication among Corbyn’s supporters to deep scepticism among the bulk of party veterans.

Labour’s surge to 38% in the poll, its best performance under Corbyn’s leadership, came after weeks of electioneering that had seen the party’s share of the vote slowly grow. It coincided with an astonishing narrowing in the respective approval ratings of Corbyn and Theresa May. There was a 52-point gap with YouGov at the start of the campaign; at one point last week, it was four points.

This has caused delight among Corbyn’s team, who always said his vow to do politics differently would attract the young and those who had previously opted not to vote. The volatile polls are also, however, causing deep anxiety in a polling industry that has been working overtime to try to correct problems that saw it fail to predict either the 2015 election or the EU referendum. “In some campaigns, nothing happens in terms of big movements,” said John Curtice, high priest of pollsters and president of the British Polling Council. “In this one, we’ve already had two major movements. Your guess is as good as mine as to whether we will have another.”

So what is going on? Inside Team Corbyn, the slow and steady recovery for Labour is put down to several factors – an electorate making up their own minds about the Labour leader; a disastrous Tory manifesto launch that included an unprecedented U-turn over social care policy; and a successful Labour manifesto with popular ideas such as increasing taxes on the rich and companies.

“YouGov showed how Labour’s best policies are cutting through compared the worst of the Conservatives’ policies,” said Matt Zarb-Cousin, a leading supporter and former Corbyn spokesman.

“People are also starting to see through Theresa May and are realising Jeremy Corbyn is not the person he has been portrayed as.

“Lots of people feel like the Conservatives are taking the electorate for granted, and it’s that arrogance which has led to a manifesto with no positive reason to vote Conservative, no costings, just vacuous slogans.”

For the Labour leader’s team, the message of the closing polls is to stick to a core campaign message that appears to be clawing back ground. A week of policy and attack announcements is planned, all around the “for the many, not the few” theme borrowed from Tony Blair. More attention will be placed on what the party describes as an attack on pensioners by the Tories.

Outside the leader’s office, some seasoned Labour campaigners agree there has been a partial recovery in the party’s ratings, which had fallen to a lowly 26% when the election was announced.

“It has picked up former Lib Dems, former Greens, young voters, non-voters, Labour voters who didn’t vote in 2015, voters in the north-west and north-east,” one strategist said. “This is a maximised core vote, with a few Ukip voters here and there, and Labour voters saying they will vote again this time.

“The Tories believed they could get all these Labour voters because of Corbyn. Well, that’s not really how Labour voters behave. Give them any excuse not to vote Tory, they will take it. The campaign has given people plenty of reasons not to vote Tory.

“The Tory ‘strength and stability’ message has been undermined throughout the campaign. The manifesto launch was a disaster. The foxhunting thing [pledging a vote on bringing it back] has cut through a lot more than people think. That plays into people’s concerns about the Conservative party – the party of the establishment, country estates, and quite vicious and nasty. It was not necessary. An absolute own goal.”

Inside the Labour machine, however, some are already warning it is not a close contest in the key battlegrounds – that not only are the Tories piling up new votes in key Midlands and northern marginals, but that even by Friday afternoon they were hearing negative reactions to Corbyn’s speech drawing a link between British foreign policy and terror attacks. Internal polling was showing a wider gap than five points, one figure said.

“They won’t vote for Corbyn in the end,” said an insider. “Some people on the doorstep are saying ‘He might win, it’s brilliant’. On the other hand, some are saying, ‘I’m not sure any more because Corbyn might be prime minister’.”

Theresa May holds up the Conservative manifesto at its launch. Photograph: Dan Kitwood/AFP/Getty Images

MPs around the country claimed that both Corbyn’s security speech, and reports of the narrowing gap in the polls, were having an effect. “We are seeing an immediate reaction to the fact that it is five points – people saying, ‘Uh oh, this is not what we signed up for’,” said one. “Secondly, Jeremy’s speech has gone down like a bucket of sick in working-class areas.”

Meanwhile, with the polls so volatile, there are seriously sweaty palms in the polling industry. News of the five-point Tory lead arrived on Friday just as the industry gathered in London for an event called: “Can we trust the 2017 election polls?” Coffee breaks between the bar-chart slides were filled with nervous conversations.

“Privately, [pollsters] are terrified and befuddled,” said one senior industry figure. “What the polls are telling us is that seven, eight, nine million people flipped in the last 10 days or so. The real evidence is that campaigns make very little difference when you look at the settled picture before them and the result. It would be unprecedented for there to be this level of churn in a campaign.

“My personal view is I don’t think the Tories were ever 25 points ahead and I don’t think they are only 5% ahead now. I think changes to methodology made since 2015 have exacerbated the volatility.”

One party strategist said: “These are hardworking, honest people trying to do something really difficult. They aren’t judged fairly.”

One of the big factors behind the polling concerns is what is becoming seen as the Corbyn conundrum. Will the Labour leader somehow manage to beat the form book and convince the young voters who flock to him in great numbers to cast a ballot on election day? Only about 43% of 18- to-24-year-olds voted in 2015.

Curtice said: “The Corbynista claim that they could get people who don’t normally vote to say they will vote Labour may have a germ of truth in it. But there are a lot of question marks. Whether they really will turn out is the $64,000 question.”

Other experts are trying to look at alternative factors to read the election runes. The Campaign Company, which oversaw an inquest into Labour’s defeat in 2015, has carried out an exercise shunning the usual practice of breaking down voters by age, geography and social class.

Instead, they have used psychological profiling around values, dividing the electorate into three “tribes”: pioneers, motivated by ethics and inner fulfilment, prospectors, motivated by “getting on”, and settlers, a socially conservative group motivated by security and safety.

Its most recent polling has found an unprecedented shift to the Tories among the settlers group, which makes up 32% of the electorate, with 57% backing the Conservatives and only 18% supporting Labour. The last time Labour won an election, its support among the group was almost double (35%) its current level. According to this analysis, it is this group that will deliver victory for the Conservatives.

Curtice also urges pundits to look at the bigger picture. “We are nowhere near the territory of Labour winning the election,” he said. “The reason for the excitement is that this is not an election about who wins – it is about whether someone wins well enough. We have moved from the territory where it was inconceivable that May wouldn’t get a landslide to not knowing whether she will or not. It is hard to get a landslide these days.”

The flurry of excitement came at the end of a week in which MPs had struggled to figure out an appropriate level of electioneering in the wake of the Manchester terror attack.

Wes Streeting, fighting for re-election in the marginal constituency of Ilford North, had been planning to head to a meeting of Redbridge Bengali Association in the morning to give his stump speech denouncing local school cuts and highlighting pressure on the local hospital. On the day, he headed to the meeting clutching a bunch of flowers to place on makeshift memorial to the Manchester victims set up by the association.

Jonathan Reynolds, fighting to retain Stalybridge and Hyde, went to Manchester’s central library for a quiet moment of reading and reflection. He said that, as a local MP, deciding when to campaign again was not just a matter of taste.

“It is quite hard to get back into a campaigning mindset here, not just in terms of the sense of partisan politics, but also, when you’re an MP and out canvassing, dashing from door to door canvassing, you’re being humorous and uplifting to engage people. That is not how you feel right now.

“People are right to say you can’t put democracy on hold – I totally get that – but it does require a different mindset. The tone of the rest of the election won’t be the same here.”