There was elation in Labour HQ as the election results came in, showing Jeremy Corbyn’s party had not only denied the Conservatives a majority but snatched seats such as Canterbury and Kensington, Tory strongholds even the biggest optimists assumed were well out of reach.

But in a leisure centre in Clay Cross, a former mining town just south of Chesterfield, a different narrative was unfolding at the count. Natascha Engel, the Labour MP for North-East Derbyshire since 2005, had lost control of a constituency the party had held since 1935. It wasn’t even that close: she trailed her Conservative rival by 2,861 votes.

The consequences of the defeat, along with nearby Mansfield, another former coalfield seat, have been overlooked by many amid the euphoria of Labour’s national success. But if the party wants to win power it should heed Engel’s warnings that the party has an “urgent” problem among white working-class voters in its post-industrial heartlands, far away from any university hall of residence.

That is the conclusion the 50-year-old had come to when she talked to the Guardian a week after losing her job. She had already put her constituency house on the market, had sent her CV to headhunters and had a meeting with Ipsa, the Independent Parliamentary Standards Agency, to discuss her redundancy package: just over £16,000 for 12 years of service.

Engel, a multilingual trade unionist with a German father and British mother, won a majority of just over 10,000 in 2005 in the half-rural seat, which curves around Chesterfield, just east of the Peak District.

She was defeated by Lee Rowley, a local boy done good. The son of a milkman, he was born just outside Chesterfield and was head boy of the best state school in the constituency. More worryingly for Labour, his auntie was Arthur Scargill’s PA and his grandad worked down a mine in Westthorpe in Killamarsh, on the border with Sheffield, in the far east of the constituency.

Facebook Twitter Pinterest Lee Rowley visits the April Park care home in Derbyshire North East constituency. Photograph: Christopher Thomond/The Guardian

Though he wears a tweed jacket and looks younger than his 36 years, he is a far cry from the Tory Boy pastiche. Thoughtful and modest, he claims to listen to the Guardian’s politics podcast and has nothing but praise for Engel (if not Engels). As Mark Grayling, Engel’s election agent, puts it: “He should be one of ours, really.”

It was Rowley’s second attempt to crack the nut. In 2015, when she was defending a 2,445 majority with Ukip in the ascendent, Engel was fully prepared for him to win. Yet she clung on, her majority reduced to 1,883, with Ukip polling 7,361 votes (cut to just 1,565 in 2017).

Rowley – formerly a banker with Santander in London, where he was a Conservative councillor – insists he won “the old-fashioned way – I’ve fought this twice now and we’ve knocked on every door in the constituency at least once”. He managed to persuade voters they had a choice: “Historically, people have felt this is a one-party area.”

There has been a demographic shift in his own lifetime: “I remember 35 years ago when there were coalmines, when we used to have Coalite factories on each side of Chesterfield and it was a very different place to what it is today. You can see the change in the people who live here, the aspirations they have, and that’s reflected in the changing political makeup. [The general election result] was probably the culmination of 25 years of people first loosening historical ties with the Labour party and eventually feeling confident enough to move over to the Conservatives.”

Finding Labour-Tory switchers in the many towns and villages that make up North-East Derbyshire is easy. Persuading them to put their name to it in a national newspaper is harder. At the social club in New Whittington, once a solid Labour ward in the north of the constituency, a 51-year-old woman said she had voted Conservative for the first time.

“So did my dad – and he was a miner. It goes against the grain, but I just can’t vote for Jeremy Corbyn,” she said. “We don’t trust him.”

She was among the 63% of people in the area to vote to leave the European Union and believed Theresa May was a stronger leader to negotiate the divorce. She said she knew she “should” vote Labour, with her receiving disability benefits for her multiple sclerosis. “Being disabled, I should have voted Labour, shouldn’t I? But it’s the bigger picture. We’ve got to have a stronger Brexit.”

The woman’s daughter is about to start a nursing degree and will be part of the first intake not to receive bursaries, thanks to Tory cuts. “But it still didn’t sway me to vote Labour,” she said. Why not? “Corbyn. I hate him.”

In Clay Cross, in the south of the seat, William Robinson, a retired miner who had worked in the Williamthorpe colliery, said he had voted Labour as usual while his neighbours stayed at home. “Labour voters are getting idle,” he said. “I’ll tell you what, duck, half of them can’t be bothered to get off their arses. They are getting all their little bits and bobs they are entitled to and they don’t seem to realise these handouts will go eventually if they don’t vote.”

Labour just couldn’t get the vote out in the once solid former colliery areas, agrees Grayling. “In our traditional areas we just weren’t far enough ahead. I’m afraid in our part of the world Jeremy wasn’t an asset. We had a lot of people saying to us: ‘We can’t vote for you this time.’ They just didn’t like his image and believed what the tabloids said about him: the idea of the loony left still rings true,” he said.

“If you look at the demographics we have got a more settled population, who voted for Brexit. They have had a shit time over the last 20 years. We have more carers and cared-for people than almost anywhere else in the country, a lot of people on very low wages, in insecure work, many suffering the effects of industrial diseases contracted down the mines or working in the steel mills, dying quite nasty deaths. We are like the land that time forgot.”

Harry Barnes, the Labour MP for North-East Derbyshire from 1987 until 2005, said the party began to lose its grip on the seat in the New Labour era, when Tony Blair refused to step in to stop Biwater, a pipe manufacturing plant in Clay Cross, being taken over by a French company. Within minutes of sealing the deal, in 2000, the French took the order book and announced it was closing down the factory, with the loss of 700 jobs.

Facebook Twitter Pinterest Harry Barnes, former Labour MP for North-East Derbyshire, at home in Dronfield. Photograph: Christopher Thomond/The Guardian

Along with Dennis Skinner, MP for neighbouring Bolsover, Barnes tried to get the Blairites to intervene. They failed. Seventeen years later his constituency is now Tory and Skinner’s majority is down to 5,288 following a 16.1% upswing for the Conservatives. In Gloria De Piero’s nearby Ashfield seat, the Tories were up 19.3% on 2015, cutting her lead to just 441 votes. Labour also lost Mansfield, which has never before had a Tory MP in more than a century.

It is a conundrum the Labour MP John Mann has called “the Bolsover question” – why Skinner saw his vote collapse despite him being “the one consistent advocate” of Corbyn and his policies.

In a piece for Politics Home, the MP for Bassetlaw in Nottinghamshire suggested that voters in Bolsover and beyond needed to be persuaded that Corbyn could be trusted on national security. “Jeremy needs to say, in simple words, that he unequivocally condemns all IRA violence,” wrote Mann, adding that Corbyn also needed to repeat his support for “shoot to kill” when terrorists are actively endangering life, and to accept that “if we are building new Trident missile systems, then there have to be remote theoretical possibilities that they will be used, by him”.

Engel insists she met as many Corbyn fans as foes on the campaign trail. Her real problem, she insists, was not the party leader but that there weren’t enough young people, particularly students, in the constituency. She makes the comparison with Canterbury, where she grew up, and which has just gone Labour for the first time in 160 years. “Had we had any young people in the constituency, a university or anything, we would have won it. It’s as straightforward as that. But I don’t have students. The population is quite elderly, and it’s obviously not an employment hotspot.”

Some young people did vote Labour. Georgina Marsh, an A-level student at Tupton Hall sixth form, said she voted for Engel because she worked in a pharmacy part-time “and I can see how the NHS is under pressure. We are getting a lot of people in who are really quite ill and should be seeing a doctor who can’t get appointments.” Plus she likes Corbyn: “I think he’s a genuine person. Everything he does is quite compassionate and I think we need that.”

But her classmate Josh Broadhurst went Conservative, despite Skinner being a distant relative on his mum’s side of the family (“She is still hard Labour and wasn’t happy when I put a Vote Conservative sign in the window.”) He said he just “wasn’t convinced” that Corbyn could pay for all of his policies and thought it wasn’t fair that classmates of his who weren’t going to university would be expected to pay for his forthcoming politics degree, under Labour proposals. “I see tuition fees and student loans as an investment in my future,” said the 18-year-old.

Engel, who says she may stand again, is worried that Labour just isn’t connecting in its post-industrial heartlands. “What we do need to do is reconnect with our white working-class voters. That has been a real problem for us in this election. I think we can do that, but I think it’s really urgent because if that group of people feel that now their natural home is the Conservative party then we really need to reappraise, because those are our founding roots.”

It’s not that these voters don’t like Labour, she insists. “But in those areas like the former coalfields, they feel excluded. And whatever government is in power, they don’t feel it has made a direct difference to their lives … and they feel like politics doesn’t really connect with them, whereas now when they voted for Brexit and it’s happening, they are engaged in politics. But as a Labour party we need to get into those communities much better and show people how voting Labour does have a direct bearing on people’s lives.”

• This article was amended on 30 June 2017. An earlier version said Natascha Engel’s redundancy package was just over £37,000. The correct figure is just over £16,000.