In April 2006, the Conservative leader, David Cameron, was asked a direct question by the Sky News reporter Joey Jones: what exactly was it about the Tory party he wanted to change? It was the sort of opportunity Tony Blair would have relished. Instead, an exasperated Cameron said: "What are we doing here? . . . Sorry, I thought we were doing a short interview." Then he brought the conversation to an end and asked for the footage to be withheld.

Four years on, Cameron's relations with his party are troubled; in particular, many backbenchers feel let down by their leader over the MPs' expenses scandal. "There is no open-door policy with David," says one disgruntled MP on the party's right. "David is not a popular leader [internally]," says another. "If we weren't ahead in the polls, he would be in serious trouble."

There are only five people who consistently have access to the Tory leader: the influential shadow schools secretary, Michael Gove, the shadow chancellor, George Osborne, the shad­ow foreign secretary, William Hague and Cameron's two very different senior strategists, Andy Coulson and Steve Hilton.

Fault lines

Senior MPs on the party's left are worried about Cameron's direction of travel. "On every important policy area, behind the spray paint and when you look at the small print, [Cameron] has failed to move on from our disastrous 2005 manifesto," one told me. And even a shadow cabinet minister concedes: "It is true that there is an issue about David, judgement and weakness when it comes to caving in [to the right]. It comes from a deep-seated lack of self-confidence."

Cameron has done much to change the image of his party, and his desire to reach out beyond the core Tory supporters is genuine. But on policy, he and Osborne have long boasted that they need no "Clause Four moment", no dramatic demonstration of change equivalent to Neil Kinnock's expulsion of Militant in 1986. On each major policy issue, from allying with fringe far-right parties in Europe to offering tax allowances to married couples, Cameron has ultimately jumped rightwards. The Cameroon project has been to maximise the perception of change while minimising the reality of it. The approach is summed up by a sentence in a leaked document about Tory candidate selection, written by Gove. "Like a conjuror, we'll get more applause if the audience cannot see exactly how the trick is performed."

In order to understand the fault lines in the Conservative Party, as well as its recent dip in the polls, one has to revisit the period during which Cameron became leader, in late 2005. On the advice of Gove, Cameron went for the top job sooner than he had planned. Gove had been the brains behind the candidacy, in 2001, of Michael Portillo, about whom Gove, a former Times leader writer and columnist, wrote a biography and whose economic and social liberalism inspired Cameron. To win over the right, Cameron not only pledged to withdraw from the mainstream European People's Party [EPP] grouping, but also signed up to all nine principles of the Thatcherite "Conservative Way Forward" group.

Pro-gay rights, female-friendly and ruthlessly slick, the Portillo and Cameron campaigns shared an opposition to Kenneth Clarke. The former chancellor felt that both men should have supported him as the party's modernising and reforming leadership candidate. He had meetings in his office with Portillo in 2001 and Cameron in 2005 to persuade them to support him. They did not.

This is important, because the Cameron-Portillo path is different from the one Clarke would have pursued. Clarke jokes privately that Cameron has done "what I would have done, only without the fights". But this is only half true: Clarke would undoubtedly have waged war with his party to show the country it was changing, but he would not, for example, have pulled out of the EPP. He wouldn't have backed down over a policy of no new grammar schools, as Cameron did in 2007, when faced with a small backbench rebellion.

Indeed, 2007 was the turning point for Cameron, the year he became the heir to the former leader William Hague: first playing the mood music of change, only to revert under stress to a "core vote" strategy. The arrival in the summer of 2007 of the former News of the World editor Andy Coulson as communications director is said to have coincided with a partial decline in influence of Cameron's ideological "guru" and strategist Steve Hilton, who seems to understand modern, liberal Britain in a way many inside Conservative central headquarters do not.

Slash the state

Slowly but surely, the 2006 slogan "vote blue, go green" has been pushed aside, along with the strongest Tory advocate on the environment, Cameron's fellow Old Etonian Zac Goldsmith. "Family values" are back, as is the former leader Iain Duncan Smith, whose answer to "Broken Britain", as he tells Jonathan Derbyshire, is to promote marriage rather than the redistribution of wealth.

Despite this, economic liberals on the right of the party believe Cameron hasn't gone far enough and are keen for him to abandon his policy of ring-fencing National Health Service and international development spending (the Institute for Fiscal Studies says the pledge to protect spending on health and overseas aid will entail real-term cuts of 18.5 per cent for all other departments).

But, on the biggest issue of all, the economy, he has pushed a Thatcherite, state-slashing line. His recent U-turn (when he claimed that a Tory government would not make "swingeing" cuts to spending in its first year in office) is a result not of a shift in ideology but, according to one senior Tory backbencher on the Tory right, "a lack of confidence".

Some Tory MPs believe the forthcoming TV debates between party leaders will be a defining test for Cameron. Having enjoyed perhaps the easiest ride of any opposition leader of recent times, the smooth Tory frontman now faces the intense pressure of a close general election campaign, without ever having said what it is about his party he feels must change. Until he does, and until he shows voters that he has changed the substance as well as the presentation of the Tory message, he will remain, however fashionable he is in the metropolitan media, vulnerable in the country at large.