When everyone is looking one way, in politics sometimes it pays to look the other way.

And that’s what makes the towns that absolutely nobody is talking about in the wake of this election – Swindon and Reading, Watford and Milton Keynes, and Hastings and Rye – fascinating. For perfectly legitimate reasons it’s the collapse of Labour’s ancient northern and Welsh citadels that have captured the imagination in the past few days but, while it’s clearly crucial for Labour to rebuild its so-called “red wall”, that’s only half the story. Even if the party’s next leader could magically recover every one of those lost seats next time, he or she still won’t win a majority without also claiming those southern marginals through which the path to power in the UK invariably runs. And in those seats, which only two years ago looked unexpectedly ripe for the taking, it just made a big leap backwards.

Many southern marginal voters have no great love for Tory governments yet this time they recoiled from Corbyn personally

Two years ago, Labour advanced to within a couple of thousand votes of knocking out the Tories in a string of southern marginals; in theory, one more heave should have got them over the line. Yet time and time again, the opposite happened. Labour lost ground, in some cases against all the odds – as in hyper-marginal Hastings and Rye, where Amber Rudd clung on by her fingernails in 2017 before resigning the Tory whip and being replaced by a candidate now under investigation for sharing an antisemitic video – while the Tories and the smaller parties gained it. If the left’s only answer to this is arguing that the Lib Dems and Greens ruined everything by daring to exist in the same seats as them, rather than wondering why so many non-Tory voters still couldn’t bring themselves to back the obvious anti-Tory choice, then it stands every chance of going further backwards next time.

These aren’t smug suburban towns immune from the pain of austerity, and they know what a hard Brexit could do to them. I live just over the constituency border from South Swindon, another classic southern marginal that was Labour until 2010. True, it includes the posher end of town plus a swathe of Wiltshire villages, but this area has been through anxious times recently with the threatened closure of Swindon’s Honda factory. There’s genuine worry locally about public services, especially SureStart closures, and no great love for Boris Johnson; Labour’s candidate, Sarah Church, a no-nonsense former army officer, was the perfect fit for an area with a strong military presence and came within 2,464 votes of turning this constituency red last time. Yet last week her share of the vote fell by more than 4% while Robert Buckland’s rose, more than doubling his Tory majority. And while it would again have been easy to blame the Lib Dems for splitting the vote, given they were also up by 4%, Church was clear in the wake of her defeat that forcing them to stand aside wasn’t the answer in places like this.

Play Video 18:00 Anywhere but Westminster: how Labour lost, and the hope that endures – video

Having watched a startling number of orange signs sprouting this time in villages that are normally solid blue, I understand why. Around here, the Lib Dems classically attract socially liberal soft Tories unhappy about Brexit; if anything, it’s the Conservative vote they’re generally splitting, and without them Buckland’s majority could easily have been bigger. These are voters who rejected Jeremy Corbyn, and any Labour recovery depends on facing up to why they did so, not scolding them for it.

Many of these southern marginal voters have no great love for Tory governments, yet this time they recoiled both from Corbyn personally – something obvious from polling long before the election – and from an overstuffed manifesto that simply didn’t sound credible to them. So far, so familiar. But a narrative blaming billionaires for rigging the system against the downtrodden masses may also have backfired: in many communities there are no billionaires, while most people don’t self-identify as victims even in tough times or expect their leaders to do so. Any analysis of Labour’s failure that sounds even remotely self-pitying, blaming a hostile media or rival parties for ruining their chances, will only further turn off voters who long to hear practical solutions to practical problems from a confident, optimistic leader.

When I asked Church who would go down well in South Swindon she described a social democrat rather than a hardcore socialist: “Someone who comes across as credible, hard-nosed, centre left,” – free from the baggage of the past, with a vision for 21st-century socialism which includes aspirational voters and homeowners too.

Labour needs to change – but it mustn’t forget what it got right | Miatta Fahnbulleh Read more

And yes, of course that’s easier said than done. But there will be no easy answers for a party that has now lost four elections in a row and has behaved for too long as if anti-Tory voters simply had a moral obligation to vote for them, trumping their own responsibility for confronting any shortcomings. If anti-Tory votes have leaked away to smaller parties, the answer is not to demand some kind of political monopoly in which all Labour’s smaller competitors helpfully step aside but for Labour to make itself competitive again. Only by learning from this defeat in every part of the country can it once again start chasing votes that may be more ripe for the taking than we think.

• Gaby Hinsliff is a Guardian columnist