If you’re on the left, it’s easy to dislike the Liberal Democrats. Helping David Cameron into Downing Street; supporting most of his austerity policies for five years; splitting the anti-Tory vote; attacking Jeremy Corbyn more than Boris Johnson. Since 2010 the Lib Dems, supposedly a party of the centre, have done much to aid three of Britain’s most rightwing governments ever. As their leader, Jo Swinson, discovered on Question Time last Friday, some voters may never forgive them.

Last week her deputy, Ed Davey, suggested that “the most likely result” of the election was “a minority Tory government”, and that the Lib Dems could informally support it in exchange for another Brexit referendum. The Lib Dems have been much more reluctant to say they might support a minority Labour administration, even though a second referendum is already Labour policy. For many leftists, such contortions may be final confirmation that the Lib Dems have ceased to be a centre party, and have turned into one of the right.

During a polarising election, the middle-of-the-road voter whom the Lib Dems have always sought may only get scarcer

Either way, today’s Lib Dems are a puzzling political force. Despite their recent parliamentary recruits from the Tories and Labour, and their surge in the polls over the summer (which has since gone into reverse), underneath they are still recovering from their disastrous performance at the 2015 election, which reduced them from 57 to eight MPs. At their manifesto launch in London last week, the hired room was modest in size and not completely full. The manifesto itself is bold and upbeat in places – “Stop Brexit. Build a brighter future” – but underwhelming next to Labour’s radical pledges.

Yet the Lib Dems still matter. Like the parties they grew out of – the Liberals and the SDP – they reflect and influence the thinking of an important but slippery cohort of voters: moderate in their own minds, often middle-class, well represented in the media and business, and several million strong. Thanks to Brexit, the Lib Dems have even become “the party of choice for some of the country’s elite”, according to the Financial Times – and it should know.

The worldview of the Lib Dems and their supporters is a sort of centrist common sense, built out of accumulating layers of political preoccupations. This worldview is not fixed, it changes over time. And these changes can reveal much about where wider politics is heading. During the 90s, the Lib Dems were hostile to the Tories and close to Labour in ways that might baffle observers of their recent behaviour. Paddy Ashdown, formerly a Labour supporter, was party leader. Under him, the Lib Dems often acted as Labour’s accomplices rather than rivals or enemies, ganging up in parliament against John Major’s government, and seeking to replace it with a Labour-Lib Dem coalition.

Some of this closeness was personal. During the final days of the 1997 election, instead of campaigning to the last, Ashdown spent an evening in Edinburgh telling me and a few other journalists, off the record, how much he was in awe of Tony Blair. Yet the Lib Dems’ crush on New Labour and its leader also reflected a widely held view among mainstream voters: that Thatcherism, while necessary, had gone too far; and that a mildly left-of-centre government was needed to undo the damage.

Blair’s 1997 landslide meant that Ashdown’s coalition dream went unfulfilled. But his successor, Charles Kennedy – another former Labour supporter – at first maintained the Lib Dems’ pro-Labour bias, treating Major’s Tory leadership successors as pariahs. Then came the 2003 Iraq war. The Lib Dems, like many liberal Britons, opposed it, and they and Labour started to move rapidly apart.

A mock launch of the Liberal Democrat ‘Orange book’, subtitled ‘Lib Dems’ secret agenda to scrap the NHS’, by Labour MP Ruth Kelly in 2004. Photograph: Dan Chung/The Guardian

A different breed of Lib Dem politician took advantage. The party has always had two politely competing tendencies: social democrats such as Ashdown, who favour state action to address public ills; and economic liberals, who believe more in the benign power of markets. In 2004 a group of ambitious young MPs from the latter faction, including Nick Clegg, David Laws and Ed Davey, together with an older social democrat who was moving rightwards, Vince Cable, published The Orange Book: Reclaiming Liberalism. Its surprisingly sharp-toothed essays argued, among other things, that the civil service should be “drastically pruned”, the Royal Mail should be privatised and “more provision from the private sector” should be introduced in the NHS.

The rising generation of Lib Dems were turning towards marketisation even more enthusiastically than New Labour, and towards austerity well before their equivalents in the Conservative party. Cameron and George Osborne didn’t start properly questioning Labour’s spending until half a decade later.

All the Orange Book policies I’ve cited were enacted by Cameron’s coalition, in which most of the book’s authors served as ministers. Swinson did too: as an employment minister. In 2013, with characteristic energy, she defended zero-hours contracts as sometimes providing “really helpful flexibility for the employee” and “quite a good relationship with [the] employer”.

There are signs in her party’s new manifesto that she has moved to the left on economic issues since then, like many mainstream voters. “We will create a fairer economy,” it promises, echoing what Labour has been saying for the past four years. But in other respects the Lib Dems are still a long way away from their left-of-centre days. Asked by ITV recently whether she would be prepared to use a nuclear weapon as prime minister, Swinson said “yes” with a showy lack of hesitation. Ashdown, despite being a former serviceman, campaigned against nuclear weapons during his early years as an MP.

The Lib Dems still have some strengths. Despite Swinson’s sometimes gauche, currently rather besieged leadership, they articulate quite effectively two common opinions: Brexit is a terrible idea; and both the big parties are in the grip of extremists. Yet the problem with this message is that gradually fewer people hold all these opinions simultaneously. Many strong remainers will probably end up voting Labour, for non-Brexit reasons. And many voters consider one of the big parties too extreme, but not the other. During a polarising election, and in its aftermath, the middle-of-the-road voter, whom the Lib Dems have always sought and often found, may only get scarcer.

What is more, as long as this turbulent period lasts, there will probably be more hung parliaments and governments with tight majorities. To avoid crippling Commons deadlocks, the Lib Dems will regularly be forced to choose between the big parties.

If they were to side with Labour again, Labour might benefit more than the Lib Dem-haters in its ranks imagine. Leftwing parties seem more credible to cautious voters when centrist parties back them. And the Lib Dems might gain from such an arrangement too: their credibility as a pragmatic party restored after their decade-long tilt to the right.

But even a very loose alliance may be hard to construct. On my way out of the Lib Dem manifesto launch, I overheard two middle-aged activists. One was saying to the other: “Well, you know what they say, ‘If you’re not Labour until you’re 30, you’ve got no heart. And if you’re Labour after 30, you’ve got no brain!’” They both laughed loud and long. The joke’s funnier still if you’re a Conservative.

• Andy Beckett is a Guardian columnist and author of Promised You a Miracle: Why 1980-82 Made Modern Britain