In the icy depths of late January, just as Theresa May was absorbing the latest parliamentary defeat for her Brexit plan, David Cameron landed in the sunshine of southern India. He had been hired to deliver a lecture on geopolitics, plus a few amusing cricket anecdotes, in Chennai, in honour of the late Indian entrepreneur KS Narayanan. The businessman was known in his home country for quitting while he was ahead, retiring at the height of his powers. Cameron, however, is better known for doing the opposite.

'Look at the Hagues and the Cleggs and the Osbornes, they’re all doing really well, and he’s the odd one out'

The official line is that the man who brought us Brexit, and then walked away whistling from his final farewell to the nation, has no regrets; that all will explained in the memoirs he is due to publish later this year, and that meanwhile he is enjoying an agreeable retirement on the international speaking circuit. In private, it’s not that simple. He may be more sanguine about it than his old friend George Osborne, but Brexit has divided the Camerons’ social circle just as it has divided the country, and they have not been spared the backlash. One society hostess entertaining the couple recently had to ring fellow guests in advance, checking that they would be able to keep things civil; not all the responses were positive. Even his wife Samantha’s Instagram feed, devoted mainly to holiday snaps and puffs for her new fashion business, is occasionally invaded by passersby criticising her husband.

Meanwhile, the world is moving on without him. Osborne has reinvented himself as a newspaper editor; even Nick Clegg has a swanky new life in California working for Facebook. But Cameron seems oddly stuck. Rumours that he fancies coming back as foreign secretary seem wide of the mark, but one old acquaintance describes him as being in an “incredibly reflective and self-searching” mood. “I think this is a really difficult time for him. He’s rethinking the usefulness of his time, all the decisions he made. Look at the Hagues and the Cleggs and the Osbornes, they’re all doing really well, and he’s the odd one out. He’s not been able to carve out a new life.”

When in London, his day often starts with a run, after which there might be meetings – he is president of the charity Alzheimer’s Research UK, works two or three days a month for the US firm First Data Corporation, and serves on an academic commission on fragile states – or a gossipy lunch at someone’s club. Weekends are often spent either at the Camerons’ new bolthole in Cornwall or their Oxfordshire constituency home, where he famously installed a chichi revamped shepherd’s hut in which to finish his memoirs. But he lacks a grand purpose. Even the UK-China investment fund he is leading seems unfortunately timed, with Chinese growth slowing and wariness about Chinese companies being used for political ends.

Writing the book has meanwhile forced him to relive painful decisions, and despite predictions that he will use it to settle scores with the likes of former friend Michael Gove, colleagues expect it to be reflective. “I know he’ll be very proud of what he did manage to achieve and I think he’ll put his case across, but you always wonder: ‘What if I’d done that differently?’” says a former aide. “You live it over again, and I think he will be quite honest.”

For some, Cameron will always be the man who wrecked his country in a clumsy attempt to save his party. At worst, a bad Brexit could break not just the economy, but the union – giving Scotland new reasons to seek independence, even pushing Northern Irish voters towards unification with the south – and the political party system, if both Labour and the Tories are punished by voters for enabling it. Even if it somehow miraculously delivers everything leavers promised, Cameron probably won’t get the credit. “If it’s a success he didn’t want it, and if it’s failure he was the one who didn’t avert it. Brexit is his legacy in the same way Iraq was Blair’s,” says Tim Montgomerie, the pro-leave Conservative commentator.

Had Brexit never happened, Cameron would still have bequeathed us the bedroom tax and axed Sure Start centres. He would be the man who cut tax credits for the poor while abolishing the 50p tax rate for the wealthiest, left headteachers soliciting donations from parents so they could buy basic supplies, and whose “jobs miracle” (record levels of employment) relied heavily on insecure and part-time work. And his defence for all this – that without austerity, Britain would have lost its credit rating and faced even more painful cuts – relies on something that can never be definitively proven. Whatever happened to the 39-year-old who became Conservative leader on a wave of such hope and promise?

‘Chillaxed’: on holiday in 2013 in Ibiza. Photograph: WPA Pool/Getty Images

Back in 2005, David Cameron embodied everything a party that had lost three elections could want from a leadership candidate: charisma, confidence, and a sense of being at ease with modern Britain. True, he was an Old Etonian. But he was also a father who spent nights sleeping on the floor in A&E, watching over his oldest son, Ivan, who suffered from a rare disorder that would sadly claim his life at the age of six. This Cameron argued earnestly for more understanding of young offenders, and demonstrated his eco-credentials by posing with huskies in the Arctic.

Yet much of that turned to slush when he entered Downing Street in 2010. The man who evangelised about “sharing the proceeds of growth” so everyone benefited from rising prosperity found himself in the aftermath of a banking crash with no growth to share. A leader temperamentally suited to good times would wield the axe instead.

Hints of the old sunny Cameron still surfaced periodically. For Nick Boles, a founding member of the “Notting Hill set” of socially liberal conservatives around Cameron, the legalisation of gay marriage in 2013 was “the high watermark” of modernisation in office.

“It was sort of classic Dave, in that it was a liberal progressive and even radical move that involved extending one of the most conservative and traditional institutions around,” recalls Boles, who as an equalities minister helped steer the legislation and later used it to marry his own partner. “And it rang true because he was the most uxorious person I’ve ever met. I’ve never seen somebody so totally, utterly in love with his wife, who felt marriage was the best thing that ever happened to him. It was genuine, it wasn’t calculation.” It changed minds, as well as lives. And crucially, in championing it against fierce grassroots opposition Cameron was “putting people before party”, says Boles. It was the opposite of what he would later be accused of doing in the referendum.

My general view of Cameron is that he held every belief fairly lightly… but that’s perfectly authentically Conservative Tim Montgomerie

Boles also singles out the National Citizen Service, a programme of voluntary work for teenagers which Cameron still chairs, and the academy and free schools programme as progressive achievements (although, for good or ill, academisation began under Tony Blair, and free schools were Michael Gove’s brainchild). Others cite the pledge to raise overseas aid spending to 0.7% of GDP, inherited from his predecessor Michael Howard but adopted by Cameron as a badge of compassionate internationalism despite internal opposition. “He always believed in the cause, there’s no question about that,” says Andrew Mitchell, his first international development secretary, who recalls “sitting and squirming” in cabinet as his budget rose while colleagues all around faced painful cuts.

Beyond that, however, the record gets patchier. Military intervention in Libya soon turned sour and a frustrated Cameron couldn’t secure parliamentary consent to intervene in Syria. The two biggest public sector reforms were Andrew Lansley’s structural reorganisation of the NHS, which baffled most people outside the NHS and exasperated many within it, and Iain Duncan Smith’s rolling up of six separate welfare benefits into one universal credit. The latter is now a byword for bureaucratic incompetence and unintended cruelty, inextricably linked in the public mind with food bank use, rent arrears and evictions; Duncan Smith himself eventually resigned over cuts to disability benefit, saying the project had been scuppered by insufficient resources.

David Cameron and Nick Clegg hold their first joint press conference in the garden of 10 Downing Street in London, 12 May 2010. Photograph: Christopher Furlong/AP

For many Labour voters, the welfare reforms alone would be damning. But for Conservatives who view austerity as having been painful but necessary to balance the books, the critique is different.

On the right of the party, it’s that he didn’t go far enough or fast enough in eliminating the deficit, missed a golden opportunity to shrink the state permanently, and moved Gove from education before his work was finished. On the Tory left, however, it’s that he gave ministers too much freedom to make mistakes. “He would intervene if he could see a political car crash actually happening, but [have] very little engagement otherwise,” says one former special adviser in the Cameron government. “If you look at what Andrew Lansley was doing at health, which was a massive reorganisation that nobody knew about until the last minute, or universal credit, or the education reforms – whatever their merits or otherwise, Cameron was very disengaged from all of them.”

The advantage of that “chillaxed” leadership style is that it arguably helped hold a potentially fractious coalition together. Cameron’s partnership with Clegg survived a full five years at least partly because he was unusually tolerant of dissent and pragmatic about compromises. (Were he still around now, some suspect he’d be rather better than Theresa May at brokering truces between warring Brexit camps.) His early career as a Treasury adviser during the Major years, the last time the Conservative party was this divided over Europe, left him suspicious of ideological purists in general as well as allergic to “banging on” about Europe in particular. He was insistent there was no such thing as Cameronism, that what mattered was what worked.

“My general view of Cameron is that he held every belief fairly lightly. There was nothing very much he really believed passionately in,” says Montgomerie. “But I think that’s perfectly authentically Conservative, I think pragmatism and not believing in anything in a theological way is actually quite an important Conservative instinct.” But from the beginning, the downside of that flexibility was a willingness to put tactics above strategy.

Back in 2005, Cameron was lagging behind David Davis, the initial frontrunner in the leadership race, when he gave a Daily Telegraph interview promising two things that sounded strangely at odds with his modernising agenda. One was tax breaks for marriage, and the second was an obscure hobbyhorse of the Eurosceptic right: taking the Tories out of the centre-right European People’s Party group (EPP) within the European parliament.

The latter proposal was “entirely cynical”, says one MP who played a prominent role in his leadership campaign: “I’m sure Dave could persuade himself that we should have our own grouping at an intellectual level, but it was a stunt. It didn’t matter to him at all.” But it was an early indicator of willingness to gamble on Europe for short-term gain. Years later, leaving the EPP would arguably make it harder for Cameron to build the goodwill needed when trying to renegotiate Britain’s EU membership.

His 2010 manifesto promise to reduce immigration to the “tens of thousands”, paving the way for the now infamous hostile environment for immigrants created by his home secretary, Theresa May, in an attempt to meet it, was arguably more sincerely meant. Cameron genuinely believed, says a colleague, that the British people had a “right to know that we understood” their feelings on immigration.

But in making the promise, he essentially accepted Ukip’s argument that immigration was too high. Once it was obvious the target couldn’t be met while free movement within the EU existed, what then?

Opinion was sharply divided when Cameron began thinking aloud, halfway through his first parliament, about an in-out referendum to settle the Europe question. Andrew Feldman, the party’s co-chairman and a close friend since Oxford, was against; election strategist Lynton Crosby was wary. At a kitchen cabinet meeting to discuss options, Osborne warned that losing would be catastrophic. But Mitchell, by then chief whip, raised a different concern. “George told him not to do it because he might lose. I thought we couldn’t possibly lose, but I told him he shouldn’t do it because he was using a referendum to solve a party management issue,” he recalls. The charge of putting the whole country through the mill to solve an internal Tory psychodrama has been made ever since.

Why did Cameron ignore the warnings? In his memoirs he is likely to defend it as democracy in action, giving the people the final say. But it was also surely driven by arithmetic. The splitting of the rightwing vote, with disgruntled Tories defecting to Ukip, had long hampered Conservative leaders but coalition turned a festering problem into an urgent one. Leftwing votes had also been split between two parties but were now consolidating, as Liberal Democrat voters swung to Labour in protest at their party entering coalition. His own backbenchers were jittery, threatening to oust him. If Cameron wanted more than one term in office, he needed those Ukip voters back.

Boles is convinced that without promising a referendum in 2015, Cameron would have lost the general election. “I don’t see any way that he could have remained in No 10 in any form, in coalition or in a majority government. It had been building for so long and the Ukip situation was very real.” Tactically it worked, with Cameron becoming the first sitting prime minister since Palmerston to increase both his vote share and number of seats held. Yet, in seeking to ward off what he saw as the socialist threat from Ed Miliband, he unleashed a chain of events that may yet make Jeremy Corbyn prime minister.

Cameron appears to have seen the referendum primarily as a safety valve, a means of releasing pent-up rightwing urges relatively harmlessly before getting on with what he regarded as the real business of the second term: returning to the themes that originally won him the Tory leadership.

David Cameron at an EU summit in Brussels, 2016. Photograph: Olivier Hoslet/EPA

Shortly after the 2015 election, he asked the Downing Street policy unit to draw up a “life chances” agenda, envisaged as a way of bringing the party back together after the referendum. The strategy was designed to improve poorer children’s future prospects by investing in the early years, helping families at key transition times during childhood and into adulthood, and tackling social issues such as addiction. Having already decided to quit before the 2020 election to spend more time with Samantha and their three surviving children, Cameron wanted to focus in his final years on becoming the social reformer he had always intended. The dream was to go out on a high having lanced the Europe boil, belatedly balanced the national books and returned to the sunlit uplands of his early vision.

Christian Guy was brought into Downing Street to work on the life chances programme from the Centre for Social Justice, a thinktank set up by Iain Duncan Smith that, in opposition, had taken Cameron on tours of the kind of deprived neighbourhoods Tory leaders rarely visited. “I think he met people that really moved him, and his eyes were opened to what was really going on. His instinct was to reform and change,” Guy recalls. “I never saw the Cameron that was all ‘steady as she goes’. He’d get frustrated with people who would kick things into the long grass.”

There was some scepticism within Cameron’s inner circle about all this, partly because the intended beneficiaries were considered unlikely to vote Tory. But it appealed to Cameron’s sense of moral decency, building on his earlier notion of creating a “big society” – in which people recognised their responsibilities towards the less fortunate – which had fizzled out for lack of concrete ideas. As Guy puts it: “Deficit reduction he always felt was his duty, but the moral mission was social reform. Europe was something he just wanted to get out of the way so he could do these social reforms.”

Had he won the referendum, it’s by no means guaranteed that Cameron would have got the chance. The jockeying for succession had begun the moment he admitted, in March 2015, that he wouldn’t stand for a third term; even before the referendum, Boris Johnson’s supporters were boasting of submitting letters to the 1922 committee urging a leadership contest. If he’d won the EU vote, the right might still have come for him, triggering a contest that Johnson could well have won. “Boris’s great mistake was to win the referendum, in a way,” says Boles.

But the fact that he was already in a tight corner may have helped convince Cameron he had nothing to lose from a referendum. As Guy points out, his instinct was usually to confront an argument head on. “Other politicians at a time of crisis would duck out of view; he used to come in fired up. He had that ability – in party conference, in cabinet, in places where they frankly wanted to string him up – to lead.”

Others take a less flattering view. “It was arrogance, I’m afraid, which has been his fatal flaw throughout his career: ‘Leave it to me, I’ll pull it off,’ sort of thing, because that had seen him through the Scottish referendum [on independence]. He wanted to get it out of the way and get back to governing,” says a longstanding ally. “I’m not sure he ever thought he would definitely win, but he just had that fundamental presumption that he was better than his opponents.” In his impatience to get on with it, however, Cameron spent far too little time rolling the pitch.

David Cameron drinks beer with the Chinese president, Xi Jinping, at a village pub in Princes Risborough, Buckinghamshire, October 2015. Photograph: Getty Images

In the recent BBC documentary series Inside Europe, Downing Street policy adviser Mats Persson recalled proposing a 2017 referendum to Cameron, only for the latter’s chief of staff to argue that if they left it that long Cameron would be “tired” of the issue by then. So it was pencilled in for 2016, leaving far too little time to overturn decades of British governments portraying Brussels as an enemy requiring handbagging into submission.

Before Ireland’s referendum on repealing the abortion ban, the government published specific proposals for replacing it and held citizens’ meetings exploring the issue in detail. The referendum itself was bitterly fought, but voters largely understood what they were being asked and what might happen next.

Cameron’s preparations, meanwhile, consisted of demanding concessions on immigration from Brussels which were refused, only reinforcing Eurosceptic voters’ sense of powerlessness. He failed to grasp how many Labour voters would see the referendum as a chance to hit back at the man who inflicted austerity on them. And he made far too few contingency plans for losing, leaving Whitehall exposed to a crisis for which it was woefully underprepared. On 24 June, the man who defined himself in opposition to the ideological wing of his party was duly beaten by it.

Cameron was criticised for not staying on to help clean up the mess, but the consensus inside Downing Street was that it would be counter-productive. “He emailed me and said: ‘I’ve thought about it and my heart isn’t in what we are going to have to do,’” recalls Christian Guy. “He would have had no authority, or mandate within the EU.” He resolved from the start not to meddle publicly in his successor’s handling of Brexit, delaying publication of his memoirs until this autumn lest he be seen as a backseat driver. But according to his former press secretary, Craig Oliver, he privately favours the soft Norway-style arrangement – outside the EU, but staying in the single market – championed by Boles and Cameron’s former policy chief, Oliver Letwin. At the time of writing, it looks as if he may be disappointed there too.

For his friends, the ignominiously abrupt end to Cameron’s political career represents a tragic waste of promise, a story of noble intent derailed by an economic crisis and Conservative neurosis over Europe as much as by his own flaws. Yet others will see an epic misjudgment rivalling Blair’s over Iraq, but without Blair’s redeeming record of prior achievements; a leader who thought he had transformed his party, only for it to turn on him. Cameron always was a gifted communicator. But if his memoirs are to shift that verdict, he may need to be a better writer than he was a politician.