During the election campaign, there were high hopes for the Liberal Democrats in the library at Goldsmiths college in London. Manoj Kerai was supposed to be finishing his anthropology master's thesis and revising, but he kept jumping up from his desk to hand out Lib Dem leaflets and buttonhole people about their policies. He wasn't even a party member, yet he says he spoke to "hundreds" of students. Sometimes he emphasised the party's opposition to the Iraq war and university tuition fees. Sometimes he simply urged people to "vote Lib Dem to stop the Tories".

Two days after the election, with a Lib Dem-Labour alliance seemingly a possibility, he finally joined the Liberal Democrats. Three days later, his party went into government with the Conservatives. Kerai's political feelings since have been "mixed . . . I am willing to give the coalition a chance. I'm happy to carry on campaigning for the Lib Dems. But I'm not a big fan of the Conservatives. The Lib Dems are closer to Labour. A lot of people on the left will never vote for the Lib Dems again. The coalition is either going to make or break the party." Has he thought about resigning his membership? "No." He pauses. "Not yet."

Kerai's anxieties do not surprise David Steel, the former Liberal leader, and the last politician before Nick Clegg to secure a place for Britain's long-marginalised third party in a coalition government, the short-lived Lib-Lab pact of 1977-8. "The Lib Dem rank and file are nervous," Steel says. "Quite a lot of the parliamentarians are nervous – about how the coalition will turn out." In the Observer last month, he described the Lib-Con deal as "collaboration with the enemy", and his misgivings were echoed by both Clegg's key predecessors as party leader, Charles Kennedy ("I felt personally unable to vote for this outcome") and Paddy Ashdown ("Am I happy about helping the Tories to form a government? No.") The Lib Dem activist and blogger James Graham adds: "The MPs I speak to say, 'It's not a question of following the coalition agreement to the letter; it's a question of: what's the issue I'm going to have to rebel on?'"

The day after the coalition was announced, a Lib Dem activist called Joe Edwards expressed this disquiet most vividly, perhaps, on the website Graham manages, Social Liberal Forum: "I did not join the Lib Dems to put David Cameron into 10 Downing Street. I certainly did not spend so much time, energy and money to see George Osborne in No 11. And most of all I did not campaign and post leaflets to see a bald Eurosceptic Yorkshireman become foreign secretary. This coalition is the worst of all outcomes and Nick Clegg and others will get all they deserve."

The resignation at the weekend of David Laws, the chief secretary to the Treasury and one of the most high-profile Lib Dems in the government, following revelations in the Tory-supporting Daily Telegraph about his expenses, will have done little to settle Lib Dem nerves. Laws was the ostentatiously stern frontman for the coalition's programme of drastic spending cuts, and yet, only three weeks into the government, he was shown to have broken the politically sensitive rules over when MPs should receive public money. The whole episode has made the Lib Dems look, at best, amateurish and vulnerable; at worst, deeply hypocritical.

And yet, mixed up with the current worry, anger and bafflement over the coalition, and sometimes warring with these feelings inside the heads of individual Liberal Democrats, there is also an enormous excitement. "The coalition is the opportunity, for the first time in my lifetime, to deal with the argument that a vote for the Liberal Democrats is a wasted vote," says the Lib Dem MP Simon Hughes. Last week, left-leaning Hughes announced his candidacy for the party's deputy leadership, a position he is likely to win. He is regarded as the champion of Lib Dems who are uneasy with Clegg and Cameron's double act. But Hughes outlines the coalition's potential almost breathlessly: "[It may lead to] the implementation of huge amounts of our policies. For example, this may well be the greenest government Britain has ever had. There's guaranteed reform of the second chamber . . . A government that is less oppressive . . . A [possible] shift in the centre of gravity of British politics away from the extreme right . . . We've just had two fantastically positive meetings in my constituency. People came with all their doubts about the coalition. But they went away excited. Now we're in government people can come and engage with us, bring us their issues, and we can do something for them. It's for real!"

A fortnight ago, Clegg gave his first proper speech as deputy prime minister. It was a perfect spring morning in north London; the venue – a college in once-Blairite Islington – showed a cheeky confidence, and the big political correspondents, before the election usually late for or absent from Lib Dem events, waited respectfully in their seats for half an hour for Clegg to arrive. Then he spoke with an intoxicating energy and optimism: "This government," he said, "is going to transform politics . . . repeal all unnecessary laws which inhibit your freedom . . . end the culture of spying on its citizens . . . reduce the power of political elites . . ." To anyone of a vaguely liberal disposition, it felt almost too good to be true. And so it proved when the speech ended and the question-and-answer session began. Clegg, as he often does when interrogated, offered wandering and equivocal responses: " . . . Of course there are several hurdles to navigate . . . We are looking at this in the round . . . What I'm establishing today is the principle . . ." Afterwards, it was hard to say whether the Lib Dems were at the beginning or the end of a golden phase in their history.

Small – or smallish – political parties may be more susceptible than large ones to sudden boom and bust. But the Liberal Democrats' recent trajectory has been especially dizzying. A year ago, Clegg was an underperforming and inexperienced party leader, nicknamed "Invisible Clegg" at Westminster and dogged by the widespread sentiment that the Lib Dems should have picked Vince Cable as leader instead. Then came the first election debate and Cleggmania; his party's seemingly historic breakthrough in the polls; the anticlimax of their election results; the even more unexpected success of their coalition negotiations; and now their prominent role in a government which commentators forecast lasting anything from decades to a few months.

So what will the coalition do to the party? The early evidence is contradictory. In polls since the election, the Lib Dems have slipped, and Labour has gained by almost exactly the same amount. But the shift has not been as large as many critics of the Lib-Con deal predicted: between 2-4%. And in the one significant electoral contest since the general election, last week's delayed ballot in the seat of Thirsk and Malton in prosperous Tory north Yorkshire, the Lib Dem proportion of the vote rose while the Labour vote sharply fell.

David Laws resigned his cabinet position after a financial scandal. Photograph: Oli Scarff/Getty Images

One guide to the Lib Dems' longer term prospects may be their Liberal predecessors' experiences of coalition government. "I had a long chat with Nick [Clegg] before the election, because he wanted to know more about [the Lib-Lab pact of] 1977-8," says Steel. Then as now, a two-party administration was forced to squeeze public spending painfully despite a sluggish economy. Successes were achieved in both areas – by 1978 Britain was no longer the spendthrift, drifting country it had been two years before – but the Liberals received little credit. The government was dominated by wily Labour veterans Jim Callaghan and Denis Healey; when their luck and judgment deserted them, in the winter of 1978-9, the Liberals, despite having no one in the cabinet and little say in government policy, were punished by voters much more than Labour. "We got tarred," Steel says, "with the brush of having kept an unpopular government in power." At the 1979 general election, the Labour vote actually rose, while the Liberal vote dropped by a fifth and effectively gave Margaret Thatcher her famous victory.

The Lib Dems have far more MPs now than the Liberals did during the Lib-Lab pact, more public support, and a much stronger role in the government. But taking part in coalitions as a more-or-less equal partner also proved disastrous for the Liberals. Between 1915 and 1922 – such dusty dates still haunt Liberal Democrat publications - the party governed with the Conservatives, in adminstrations led by the great Liberal statesmen Herbert Asquith and David Lloyd George. Yet the pressures of office in a time of war and recession, and the not-always-collegiate behaviour of the Conservatives, saw the Liberals shrink from having 272 MPs to 115, split into two factions (the Liberals and the National Liberals, one inside the government and one outside), and decisively lose their place to Labour as Britain's second political force. Over subsequent decades, the Liberals played intermittent and increasingly minor roles in a series of Conservative and Labour-led coalition governments; split again; saw MPs and members defect to the Tories; and, by the 1950s, came close to extinction as a national party.

Ideology played a large part in the Liberals' long history of failure as coalition partners. Pacifism, free trade, whether to back Labour or the Conservatives – all these divided the party at different times when, in the treacherous world of coalition politics, a united front should have been the priority. But the issue that most consistently split the Liberals, and which creates tensions in the Liberal Democrats still, is the party's attitude to the state. "There is this faultline in the party," says historian David Marquand. "There are social liberals, who want to use the state in an active way to enable citizens, and there are classical liberals, who are much more suspicious of state intervention."

From the late 70s until recently, the former group, which includes Ashdown, Hughes and figures such as Kennedy and Shirley Williams who were originally members of the Labour party, was in the ascendancy. But over the last few years the latter group, led by Clegg, Laws and Cable, has strengthened and steadily shifted the Lib Dems rightwards. In 2004 all three men contributed essays to The Orange Book: Reclaiming Liberalism, a surprisingly sharp-toothed volume which attacked "nanny-state liberalism", and called for the civil service to be "drastically pruned" and the NHS to be replaced by "a system of competing insurance schemes". The book's mixture of free-market toughness and liberalism on social issues was strikingly similar to the political recipe that Cameron and other Tory modernisers were beginning to concoct. "A fresh axis is establishing itself," wrote the perceptive Conservative commentator Ferdinand Mount, whose cousin is Cameron's mother, in the Daily Telegraph in 2005. "Cable would not look ill at ease in a Cameronian government . . . The electoral system is now so skewed [against the Tories] that . . . we might have to contemplate a Tory-Lib Dem coalition."

Yet to many Lib Dems, and to a public not accustomed to paying close attention to the third party's ideological development, the Lib Dems remained – and remain even now – a broadly anti-Tory enterprise. "The balance of the parliamentary party is left of centre," says Hughes. "We've got to keep our links with unions, council tenants, public-sector workers." Steel says: "I regard myself as a Keynesian liberal and so does a huge chunk of our electorate." Polls have consistently shown Lib Dem voters and activists defining the party as leftwing rather than rightwing, sometimes by a margin as great as eight to one.

During the Thirsk and Malton election, the difficulty for the Lib Dems of simultaneously representing both this centre-left liberalism and the centre-right liberalism of the coalition was often obvious. The afternoon before polling day, the Lib Dem candidate, a personable district councillor called Howard Keal, told me, "It's marvellous to be [campaigning] on the doorstep and to be able to point to achievements in government." But then he rather spoiled this impression of coalition harmony by giving as his example: "We've booted [the Conservative] inheritance tax cuts for the wealthy into the long grass!"

How would he sum up the mood of Lib Dems locally? His smile froze. There was a long pause. A whole verse went by on the pub jukebox. "Mostly, it's positive. There's a relatively small minority who are still coming to terms with the new political landscape . . . I would never have imagined in my wildest dreams that we would be forming a coalition with the Tories."

Having done so, the Lib Dems probably have no choice but to try to make it work, at least in the short term. "If there's another election in the next year," Williams told the New Statesman recently, "my party will be crippled – we're not rich." Clegg's initially underwhelming leadership did damage to the party's finances from which it has yet to fully recover: in 2009 the Lib Dems received a quarter less in donations than they did in 2005.

Some in the party regard the next general election with dread, whenever it is. For all the policy concessions won by the Lib Dems in the coalition negotiations, Conservatives are in charge of most of the big ministries, and control 90% of government spending, according to the Institute for Government. The readiness of the Tories to offer the Lib Dems equal billing at government press conferences and in government policy statements, despite the Lib Dems having less than a fifth as many MPs, may not be purely an act of generous and daring political "partnership", but also a political bait which the Lib Dems are too eagerly swallowing. One way for the Tories to escape their still-toxic reputation for cold-hearted public spending decisions may be to make Lib Dems such as Laws and now Danny Alexander the Treasury's best-known axemen.

And the cuts could do particular damage to the Lib Dems, strongly associated as they are with compassion by many voters. "It looks like a matter of how many seats we're going to lose, not gain," says Graham. "We're probably talking about being left with about 30." That's barely half the party's current total. To retain more, he says, the Lib Dems will need to win the proposed referendum on voting reform – no pushover in a country usually wary of constitutional change – and the Conservatives are likely to campaign against it. Ominously too, at last month's local elections, a series of well-known Lib-Con councils were swept from office by Labour after a few years in power. Even before Clegg and Cameron held their first joint, joshing press conference in the 10 Downing Street garden, there were signs elsewhere in Britain, little noticed by commentators amid the early promise and novelty of the Lib-Con experiment, that such an alliance might be electorally disastrous.

Graham is not that pessimistic. "There's a long game and a short game here," he says. "The party gets five years to advance its agenda in government. Then there is a trough of unpopularity as the coalition ends. Then a resurgence. It will be Nick Clegg's successor that gets the big chance." Or, in another optimistic scenario, the strains and messy compromises of coalition government could split the Conservatives – already divided between supporters and opponents of Cameron, and supporters and opponents of the coalition deal – and the Liberal Democrats could take advantage. Or, finally, the Lib Dems could become like the Free Democrats, a German party in which the famously Europhile Clegg takes a keen interest: small, socially liberal but rightwing on public spending, and so nimble at coalition-building that they have been in power with one of the German main parties, both centre-right and centre-left, in every decade since the second world war.

Perhaps the Lib Dems are not as naive about coalitions as generally supposed. Towards the end of our interview, I ask Simon Hughes, who has been manoeuvring to maximise the third party's leverage since becoming a Liberal in the early 70s, how he sees Labour developing over the next few years. "The Labour leadership, I hope, will be intelligent, progressive," he says. "I've shadowed David Miliband. We get on personally very well." Could he envisage the Lib Dems being part of a different coalition in the future? "Absolutely." He elaborates carefully: "If there is a balanced parliament next time, and if Labour and the Lib Dems make a majority, and the Lib Dems and the Tories don't, then the presumption [about who we go into government with] works in a different way."

Maybe after the next election, there will be a Lib Dem coalition that goes down better with Manoj Kerai and the other anti-Tory students in the Goldsmiths library. Assuming, that is, the coalition's cuts haven't shut the library in the meantime.

• This article was amended on 2 June 2010. The original referred to the Institute of Government. This has been corrected.