In 2005, he was the slick and charismatic saviour of the Tory party. Today, his bouts of red-faced anger and loss of authority have got the critics' knives out

David Cameron likes risks. On his second day as Tory leader, 7 December 2005, he took one of his biggest. In that now-distant era, Tony Blair was prime minister: not as dominant as he had been for the previous decade, but still formidable in the Commons, where he had seen off four of Cameron's predecessors with patronising politeness and lawyerly precision.

Yet Cameron used his first prime minister's questions to mount a surprise frontal assault. After a few minutes of mild back-and-forth, with Blair seeking to settle once more into his accustomed role as the only grownup in the room,a smiling Cameron, looking leaner than he does these days, switched suddenly from consensual to mocking. "It's only our first exchange," he said, "and already the prime minister is asking me the questions." Then Cameron leaned showily forward across the despatch box, and said with careful emphasis: "He was the future once."

That such a lethal soundbite might backfire one day was lost amid the triumphant Tory laughter. Blair sat listening to the uproar with a fixed smile, and for the rest of prime minister's questions spoke untypically fast. He seemed rattled. In the Commons and beyond, there was a sense that a new political star had announced himself. "David Cameron was clever and people-friendly, and I thought he had some real steel to him," Blair wrote later in his memoirs about Cameron's performance as opposition leader.

In fact, some have marked Cameron for greatness ever since Margaret Thatcher was premier. Two years ago, in a Sunday Telegraph article headlined "David Cameron: born to be prime minister", the authoritative Tory-watcher Matthew d'Ancona recalled that in the late 80s, "A mutual friend ... told me about the young hot-shot at [Conservative] Central Office who 'will be Tory leader one day'. This was taken as read in his [Cameron's] circle."

In 2003, after barely two years as an MP, Cameron began to be publicly promoted by seasoned Conservatives and political journalists as the long-awaited Tory equivalent of Blair. As Cameron became opposition leader and then, almost immediately, prime minister-in-waiting, expectations rose further. He would be "one of the most important politicians of the early 21st century", wrote the veteran rightwing commentator Bruce Anderson in the Spectator. "Cameron is the most immediately likable leader of a Conservative opposition since Stanley Baldwin [in the 20s]," wrote Anderson's centre-left equivalent, the late Alan Watkins, in the Independent on Sunday. Michael Cockerell, the revered political documentary-maker, made a 2007 BBC2 film titled David Cameron's Incredible Journey.

His failure to win the 2010 election suggested this potential might have been a little overstated. But his seemingly assured role in forming and managing the coalition, and the smoothness of his first months in office, convinced most of the commentariat that the election had been an aberration. "Like it or not, Cameron is a born leader," ran the headline of a December 2010 Independent piece by Blair's biographer John Rentoul. As prime minister, wrote Rentoul, "Cameron has two outstanding qualifications ... His convictions are adaptable ... [and] he is very polite." Thanks to "the effortlessness with which he plays the part of national leader ... How quickly we [have become] used to David Cameron in No 10."

To underline how many New Labour figures admired and feared Cameron, Rentoul quoted a recent BBC interview with Blair's former home secretary John Reid. Cameron, Reid said, was "a better prime minister than he was leader of the opposition", "successful" and "astute". With Ed Miliband – it seemed then – shakily installed as Labour leader, and voters widely assumed to be deeply weary of his party, the Westminster consensus in 2010 and for much of 2011 was that Cameron would get two terms as prime minister, if not more.

There had been hints even before the election that the Conservatives were thinking along those lines. In 2008, Francis Maude, then the head of a high-profile, faintly hubristic Tory "implementation unit", working out in advance precisely how his party would govern, told the Independent: "We have a vision of how Britain can be. It will take a long time to deliver it." In 2009, Cameron told SunTalk, a short-lived internet radio spinoff of the Sun, "We need a government acting for the long term ... with a five- to 10-year horizon." A key former Cameron adviser says: "He really admires Margaret Thatcher, rather than Blair, as an achiever [in office]. Blair didn't think about changing the national culture. David wants to change Britain."

How misplaced much of this confidence and praise, how distant his first slick phase, seems now. His political arc has turned downward. In the Sun, always sniffing for political winners and losers, he is "struggling David Cameron". In the Mail on Sunday in May, even before this summer and autumn's anti-Cameron plots and government calamities, he was "a wounded PM". In the Sunday Times last month, the well-connected rightwing columnist Martin Ivens reported: "Tory loyalists and critics alike increasingly fear that David Cameron is a loser ... real doubts have taken hold that Cameron can win the next election." In the current issue of the Tory journal the Spectator, a cover article by the editor Fraser Nelson is more definite: "Drift and disillusionment will lose Cameron the next election," runs the headline. "Dave's going down."

Some of his problems are big – the recession, the deficit, the polls, the collapse of the plan to redraw parliamentary constituencies in his party's favour, the Boris bandwagon, the maturing of Miliband, the growing popularity of Ukip, the upcoming trials of Cameron's friends and allies from News International. Some of his problems are small – his increasingly noted tendency to redden when angry, his stumbles on the David Letterman Show last month, his reputation for "chillaxing" too much with DVD boxed sets and games of Fruit Ninja. What is ominous is the way all these issues are beginning to interact and reinforce each other. Fruit Ninja, he unwisely confided to the Telegraph in January, during a brief uptick in the polls, is "quite good to get your frustration out. If you can't have a reshuffle, play Fruit Ninja."

Calm down, dear: Cameron at prime minister's questions. Photograph: PA

At prime minister's questions these days, Cameron is a less commanding presence. A kind of stutter has crept into his voice – "Let me, let me, let me explain what this reshuffle is all about ..." – and an anger chokes what used to be languid dismissals of Miliband – "Let me tell him what is actually happening in our economy!" While he leans an elbow on the despatch box with his old insouciance, his other arm often makes a downward, stabbing gesture to emphasise his point, one rigid finger out.

All prime ministers ultimately fray and weaken. But Cameron has done so fast: "His shelf life has narrowed incredibly," says Tim Bale of the University of London, a historian of the modern Conservative party and its troubled post-Thatcher leaders. John Curtice of the University of Strathclyde, a leading authority on voter attitudes, says: "People used to think Cameron was charismatic. But he is proving to be a kind of average prime minister. His ratings are not terrible, but he's not Thatcher, he's not Blair. He is not a dominant figure. Nobody loves him. That is why the Boris story is taking off."

All prime ministers have internal enemies and critics, but Cameron's are tellingly ready to speak out. From Nadine Dorries's "arrogant posh boy" jibe to Stewart Jackson putting Cameron "on notice" after the Conservatives' poor May local election results, government MPs openly attack the prime minister with a frequency that Blair – with his proper electoral mandates – never suffered. The Tory blogger Iain Dale says: "Nobody's really frightened of No 10."

Even during Cameron's ascendant period, his circle of able intimates always felt smaller than Blair's. Now, with his policy guru Steve Hilton absent much of the time on an ambiguous "sabbatical" in California, and his spin doctor Andy Coulson lost to the phone-hacking scandal, Cameron is served by a No 10 that many observers of Whitehall consider underpowered or even chaotic. "David is not an intellectual," says his former adviser. "A bit of slackness and complacency can creep in. He is coasting a bit now."

There are also wider and deeper reasons for his changed fortunes. Governments are unpopular across the west, and have been since the financial crisis abruptly ended the long 90s and noughties boom in 2008. At first, as centre-left administrations such as Gordon Brown's struggled and fell, it was commonly believed – not least by many Conservatives – that rightwing parties would benefit. That conclusion looks shakier now. Cameron's close ally, the centre-right French president Nicolas Sarkozy, lost power in May; even the formidable German centre-right chancellor Angela Merkel is beginning to look vulnerable in the polls. Austerity policies, initially popular while still largely theoretical, as they were for Cameron in 2010 and early 2011, in practice are proving electorally toxic for almost all incumbents.

For Bale, Cameron's party has a particular weakness too. The Conservatives have not won a general election for more than 20 years. "The party is still stranded too far to the right from where most voters want them to be on public services and the economy. There is an absolutely clear consensus among academics that the Conservatives didn't win the election because ... they scared too many people. In government, they have proved to these people that their fears were well-founded." Max Wind-Cowie, head of the Progressive Conservatism Project at the think-tank Demos, agrees: "In many ways Cameron has transformed the Conservatives – for example, Tory MP's now look and feel more like the rest of us – but the one thing he didn't do was rethink economically." Cameron retains a Thatcherite faith in the free market; since the financial crisis began, it has felt more and more out of date, given all the market meltdowns and "the public's quite sudden distrust of the neoliberal economic project". His desire to shrink the state, similarly, has come up against voters' stubborn fondness for big state institutions such as the NHS: "The right of the party need to accept that the public are not just one snappy libertarian argument away from telling the state to bugger off."

And then there are Cameron's well-fed features and inherited wealth, his elite education and grand family connections; his easy manner and 90s career in public relations, perhaps that more comfortable decade's archetypal business; and the fact that he spent his first three years as Tory leader arguing that Britain's economic problems were largely solved, while promising that as prime minister he would match the Labour public spending he now condemns as profligate. It is not the best persona and CV for an austerity prime minister. For all his hairshirt talk – expect more in his almost make-or-break speech to the Conservative conference on Wednesday – Cameron often feels like the last major politician of the boom years.

With fellow Tory and rival Boris Johnson. Photograph: Yui Mok

How long has he got left? "You would expect the coalition to survive until 2015," says Curtice, "except for" – he cites the famous phrase from Harold Macmillan, the previous Old Etonian prime minister, for the inevitability of political shocks – "events, dear boy". Boris Johnson is scheduled to speak at the Tory conference today and tomorrow. Next month is the Corby byelection, the first in a Conservative seat since the general election – the absence of such awkward contests has been a characteristic piece of Cameron luck – and Labour are likely to win it. If that happens, a workable Labour-led coalition government becomes theoretically possible, involving the Liberal Democrats, the Greens, Plaid Cymru and Labour's traditional Ulster allies. While Nick Clegg remains Lib Dem leader that is hard to envisage, but as Stephen Tall of the website Liberal Democrat Voice puts it: "Lib Dems are starting to look post-Clegg and post-Cameron."

Tall suggests that thoughts may become deeds if the Lib Dems are still dismally unpopular in 2014. That pre-general election year also brings the European elections: a common current prediction is that the Conservatives will come third behind Labour and Ukip. Such a surge by the latter, warns Dale, could permanently "pull away enough Conservative voters" to make a Tory general election majority almost impossible. It is already much less likely than many people realise. According to Curtice, depending on how the Lib Dems do, the Conservatives need between 6% and 10% more votes than Labour to win outright. Yet in most polls the Tories are at least 10% behind. "People in the party are very realistic about the chances of winning the next election," says Wind-Cowie. "They see the chances as marginal at best."

If Cameron is not prime minister after the general election, says Bale, "he's toast". His lack of political obsessiveness – his biographers Francis Elliott and James Hanning note scant political activity until well into his 20s – and of a large support base in the party make it unlikely he would battle on as a defeated Conservative leader.

A second Lib Dem-Tory coalition, says Bale, would probably enable Cameron to cling to office. But everyone I spoke to agreed that it would be far harder to sell to both parties than the 2010 version. A Conservative minority government might in some ways be better for Cameron: "The more precarious," says Bale, "the less likely a leadership challenge." Yet neither would be a glorious outcome for "the heir to Blair", as Cameron called himself when campaigning for the party leadership in October 2005.

And, given that the Lib Dems are likely to lose seats to Labour in 2015, both outcomes would almost certainly require the Tories to gain seats overall – not something any British government has achieved for three decades, since Margaret Thatcher's great success in 1983. That victory was made possible by the Falklands war, the foundation of the SDP, Michael Foot's frail Labour leadership, a partial but dramatic economic revival, and a crusading, crisis-hungry prime minister – an almost freakish combination of factors that feels close to unrepeatable.

Yet Thatcher's recovery from her nadir in 1981, when a struggling economy and misfiring austerity policies made her far more personally unpopular than Cameron is now, convinces some Conservative strategists to this day that any mid-term problems can be overcome. "That's the Tory party's problem – 1979-1983 is the script they're still following," says Bale.

Cameron does have some Thatcher qualities. "He is best in a crisis. He can do the high-wire business," says his former adviser. Elliott and Hanning describe Cameron's behaviour the morning after the 2010 election, as news of his party's potentially fatal electoral shortfall spread: "'He had to think very quickly how he and George [Osborne] were going to get out of this alive,'" said a friend ... But since things were in flux there was still a chance to shape events."

As Thatcher did, Cameron is also reshaping the country. "Education, welfare, the NHS – these are huge, huge programmes of reform," says Wind-Cowie. It may be that Cameron is beginning to see these as his goals instead of a second or third term. His former adviser remembers: "I asked him once what he would like to have achieved as prime minister, when he looks back, and he said: 'I would rather be in for one term if it meant we could reform welfare and education.'"

In politics, as elsewhere, people often conceive of the future as simply an extension of the recent past. Back in 2005, when Cameron became Tory leader, that meant Blair and Thatcher and whole decades of near-impregnable single-party rule. Now we are in a different world. The last time British politics was this turbulent, between the mid-60s and the mid-70s, the central figure was the Labour prime minister Harold Wilson: charming, telegenic, slippery, lazy, plotted against, despised by many in his own party. Remind you of anyone? Wilson somehow won four general elections.