When New Labour died, Gordon Brown’s team went to the pub. It was the evening of Tuesday 11 May 2010, five days after the general election. After days of uncertainty, David Cameron had formed a coalition with the Liberal Democrats. Members of the No 10 staff toasted the end of an era in a room upstairs at the Old Star, a low-ceilinged Victorian box in Westminster. Everyone was exhausted, haggard, looking forward to a summer of decompression. For three years, Downing Street under Brown had been a sleepless bunker, powered by the tyrannical rages of a mortally wounded prime minister and the tribal loyalty of his ministers and advisers.

At the wake, only Ed Miliband seemed to have reserves of energy. He trotted up to Stewart Wood and asked him round to his house the following morning to talk. The two men had first met in 1995 in Berlin, where Wood was studying for a PhD. In 2001, Ed secured an advisory role for his friend at the Treasury. He warned Wood on the first day to prepare for the frustration of trying to smuggle social democratic impulse into a vessel of New Labour statecraft. “We’re not as radical as you’d like,” Miliband said.

How to be radical and still win British elections was the theme of conversation in Miliband’s living room that Wednesday morning. It was to be the theme of many such conversations among a tiny group of close advisers over the coming years, as they tried to set a crumpled and demoralised Labour back on the road to power on terms that most of Westminster thought impossible.

Their project would be shaped by the constant struggle to assert authority over the party and to transmit a message in a climate of press hostility and militant public scepticism about politics. The battle to preserve the integrity of Miliband’s ideas would merge with the challenge of presenting the man himself as a candidate who might plausibly one day stand on the threshold of Downing Street, when his character and judgment were a topic of frequent ridicule. It was with some justification that he could declare, at his party’s manifesto launch this week, “I have been tested. It is right that I have been tested for the privilege of leading the country. I am ready.”

In 2010, Labour looked readier for a spell in the political wilderness. The last escape from opposition had taken 18 years and required modification of the party’s ideological DNA, making it more compatible with the bourgeois sensibilities of middle England. In the process, it had shackled itself to a form of free-market capitalism inherited from the Tories – characterised, as Miliband saw it, by short-term profiteering, job insecurity, hostility to state intervention and tolerance of grotesque inequalities. The question now was whether New Labour represented the best available compromise between egalitarian ideals and an electorate that did not seem to have a high socialism count in its bloodstream. Miliband and Wood believed there was another path: there were other ways of being capitalist without surrender to the market. That morning, Wood and Miliband talked about Konrad Adenauer and the Christian Democrat tradition that built Germany’s post-second world war economic miracle. They discussed William Howard Taft, Teddy Roosevelt and the US Republican tradition of busting monopolies – the right’s own, neglected tradition of confronting runaway corporate power. The buccaneering winner-take-all form of Anglo-American capitalism, in other words, was more negotiable than New Labour had believed.

Both men had studied Varieties of Capitalism, a 2001 book edited by David Soskice and Peter Hall that catalogued the many ways that developed countries had blended markets, labour rights, corporate regulation and tax. The authors had jointly supervised Wood’s PhD. Hall had also invited Miliband to deliver a course of lectures at Harvard in 2002 – a sabbatical from the Treasury that was a formative period of growing self-confidence and independent political thought, in which Miliband’s friends saw the seeds of a belief that he should be Labour’s leader.

Wood agreed to help run Miliband’s campaign, and has been one of the most influential voices at his side ever since. An acute intellect hidden in mild dishevelment, Wood has been a co-formulator of the intellectual arguments that underpin Miliband’s project. Wood articulated the sweeping context in which Miliband felt he was operating: that the present decade was likely to be an era of ideological and political flux, akin to the 1970s. In that decade, as now, a long period of economic consensus was coming to a close, amid talk of great political realignments and upheavals; parliament was hung. Miliband and his friends sometimes envisaged themselves shaping the outcome of this turbulent interregnum on the left’s terms, the way Margaret Thatcher took possession of the 1980s for the new right.

Not everyone was persuaded. The notion that Britain’s political centre of gravity was shifting to the left, or might be tugged there, was met with scorn by veterans of Tony Blair’s mid-90s march on power. Some blamed Wood for appealing to Miliband’s romantic leftism, in contrast to the vote-chasing pragmatism that was drilled into him as an officer of New Labour. “Stewart believes that there is a social democracy in Britain waiting to come out,” a shadow cabinet minister once told me, “and he’ll test that proposition right up to the point of losing the election.”

Wood had been Miliband’s second recruit to his leadership bid. The first was Greg Beales, a younger Brown adviser, burly and baby-faced, less susceptible to the joys of political abstraction. Miliband had buttonholed Beales one day earlier, on the margin of failing coalition talks between Labour and the Lib Dems. If Wood appealed to Miliband’s appetite for new left paradigms, Beales drew him toward kitchen-table politics. He has studied the polls and tested messages for popularity. His politics are a shade more Blairish; his manner and accent more East End than Oxford don. To unwind, Beales goes to a boxing gym; Wood writes songs on the guitar. (Miliband himself doesn’t have many hobbies: a holiday companion recalls him sitting on a beach reading a biography of Iain MacLeod, a Tory minister from the 1960s.)

Miliband’s task over the next five years would be to take the ambition that was hatched in his living room that May morning and fashion it into Labour’s agenda for government. He had to refine an intellectual proposition into something that looked like a stump pitch to the electorate – a message that would set pulses racing in Beales’s focus groups. The challenge of doing so was prefigured in his first appointments. “One way to look at the journey of the past five years is to see it in terms of Ed’s favourite adviser changing over time from being Stewart to being Greg,” one senior figure on the team told me as the election campaign got under way. Miliband himself would put it differently. As the years went past, he would talk about a balance between “the highfalutin ideas stuff and the retail end”.

Miliband would learn the hard way that British politics can be a hostile climate for intellectualism. It often looked as if the gap between theory and policy was not so much bridged as crossed in abrupt leaps of faith. Inside the advisory clique, it was obvious how capping utility bills or taxing mansions expressed a grand ambition to redress a structural bias in British capitalism. But to Labour sceptics and Tory critics, it looked like heavy-handed socialist statecraft – or simply a muscle memory of the “clunking fist”, as Tony Blair had once characterised Gordon Brown’s crudely effective technique for pounding political points home.

It was a long shadow from which to emerge as a plausible candidate for prime minister. In that summer of 2010, all Miliband really had was an abstraction – British capitalism rewired for a fairer allocation of wealth and opportunity. He had neither policies nor a plan to persuade the country that it could be done.

* * *

Facebook Twitter Pinterest Miliband in 2014, campaigning to keep Scotland in the United Kingdom. Photograph: Murdo MacLeod

At around 9 o’clock on the evening of Friday 3 September 2010, my phone rang at home. It was Ed. He had found out that the Observer was planning to run an editorial that weekend endorsing his older brother, David, for the leadership of the Labour party. As chief leader writer, I would be drafting the opinion. Miliband wanted to persuade me that we were wrong. He assured me he was going to win: “We’ve got the Big Mo.” I didn’t recognise the Americanism at the time. “Mo” for momentum; snowballing towards victory. He spoke with amiable urgency, tinged with exasperation that the message didn’t seem to be getting through.

Miliband was disappointed that the Observer, a voice of the liberal left, was taking the same line as Conservative-supporting media. “Why are you buying into their definition of what kind of leader Labour should have?” he asked. I replied with a question about how he would shift perceptions of what a Labour leader is supposed to look like. He answered obliquely by talking about the financial crisis: how it had changed the terms of political trade in ways that were auspicious for the left. It was a common argument at the time, at least in intellectual Labour circles. Globalised finance had failed; only massive state intervention had averted collapse of the banking system. If you zoomed out far enough, it looked like seismic activity in the ideological crust of British politics.

The strategy for the leadership race – formulated in large part by James Morris, a precocious protege of Stan Greenberg, one of Bill Clinton’s former pollsters – portrayed Ed as the “insurgent outsider”. It was an appeal to frustration in the ranks that Blairism had diluted the ideals of the left so heavily that they had started to taste conservative. The plan meant casting David, who was the frontrunner, as the proxy for an obsolescent New Labour establishment. But the real coup was securing support from trade-union bosses, who wanted to thwart the “Blairites” at all costs and saw the younger Miliband as a pliable instrument to that end.

When the votes were counted, Ed won by a fraction of 1%, tipped over the finish line by the weight of the unions. He had not won among ordinary members or MPs. It was a mandate with a birth defect. Ed wanted to believe that his articulation of a new political and economic paradigm had won the day. Westminster declared with one voice that it was the old left that had done it; that Labour had curled into thumb-sucking nostalgia and probably forfeited the next election. A cheer went up in Conservative press headquarters when the result was declared.

For more than a year after moving into the opposition leader’s suite of offices in parliament, business was chaotic. Experienced figures from the last Labour government bowed out of the game. Miliband was rebuffed several times in his efforts to recruit a permanent chief of staff. Discipline was breaking down. The shadow cabinet was packed with former ministers who had been senior to Ed in government and did not embrace their new subordination. In meetings, his authority was challenged openly or undermined by disdainful body language – rolling of eyes and inattentive jabbing at BlackBerrys. Downstairs in the huge glass atrium of parliament’s Portcullis House building, disdain for the new leader was spilling out from Labour MPs into the ears of political correspondents.

“There was zero honeymoon period, it was pretty horrific,” said Lucy Powell, Miliband’s campaign manager and acting chief of staff. “We didn’t have the machine on board, we didn’t have most of the shadow cabinet on board at that point. There were grenades being chucked in all the time.”

Meanwhile, the new government had successfully set the terms of debate on the decisive issue – the economy. Recession, derelict public finances and public-sector cuts were being blamed on profligate spending by Brown. The shadow cabinet was divided on whether it was best to concede the point and move on or, as Ed Balls – who would become shadow chancellor – insisted, double down on the counter-charge that austerity was hobbling the economy and delaying a recovery. Miliband’s instincts as an economist were with Balls, but he was also unhappy so much energy was being spent to defend the legacy of a government that voters had rejected.

He wanted to start an entirely different conversation about the economy. During his time at the Treasury he had grown impatient with the reluctance of his bosses to express a central demand of left politics: that government should intervene to correct the injustices that inevitably follow when wealth and power are hoarded by a minority. “Ed found the stealthiness of New Labour frustrating,” said a former colleague. “It annoyed him – the pact that seemed to say, ‘You can have all these great public services without paying more taxes.’ ”

But Miliband was also a product of the Labour generation whose formative, scarring experience had been Neil Kinnock’s defeat in 1992. Just when it looked as if the years of Conservative rule were over, the British public had recoiled from an explicitly social-democratic platform. The lesson seemed to be that the game could only be won by sticking to rules set by the Tories and Tory-supporting newspapers.

There was zero honeymoon period, it was pretty horrific ... There were grenades being chucked in all the time

What little prospect there might have been for winning over one section of the press died in July 2011. Miliband rejected counsels of caution and attacked Rupert Murdoch hard over illegal phone hacking by journalists at News International. At the end of one long meeting where all the downsides of provoking the press had been set out, Miliband responded with a wearily defiant exclamation: “What is the point of us even being here if we don’t speak out on this sort of thing.” It was a cry of exasperation at more than just the matter in hand. A year had passed and not once did it feel that Labour had set the agenda.

The episode restored depleted confidence. Miliband saw it as confirmation of his judgment that the public was ready for a more pugnacious approach to the institutions that had long irrigated the terrain of British politics with conservatism. He went into that summer determined to use his party conference the coming autumn – a moment when the opposition is guaranteed a few days’ coverage on its own terms – to press the point.

* * *

Miliband was in his Liverpool hotel room on the evening of 27 September 2011 when he caught a glimpse of what looked like a newspaper front page with the words “Thick as a Brick” on the screen of Wood’s BlackBerry. “Oh my God, is that tomorrow’s headline?” he gasped. It turned out to be an image of a Jethro Tull album cover. But the real reviews for the speech he had just delivered at the Labour party conference were scarcely better.

The big idea was “responsible capitalism”. Miliband argued that individual pursuit of profit need not take precedence over collective wellbeing; that corporate interests needed to be realigned with the interests of their employees and society as a whole. “The people at the top taking unjustified rewards is not just bad for the economy. It sends out a message throughout society about what values are OK,” Miliband said.

But the passage that got most attention was a metaphor to describe conflicting capitalist ethics – “predators and producers”. The phrase had been cut and reinserted several times over the summer in Miliband’s living room: Beales at the laptop, Wood at his shoulder; versions of the text emailed to Marc Stears, an university friend of Miliband who was teaching at Oxford and was later put on the payroll as a speech writer. They knew it would be controversial, but did not anticipate the ferocity of the backlash. The frontbench team was not properly briefed – after the speech, shadow ministers struggled in interviews to name actual predators. New Labour loyalists thought the party’s pro-business credentials, painstakingly established in the mid-1990s, had been torched. A ripple of jeers washed across the auditorium when Miliband mentioned Tony Blair.

The speech provoked a smattering of indulgent commentary, but mostly it aggravated the old tension between those who wanted to salvage Blair’s legacy and those who thought its grave was too shallow. The unions started agitating against the “zombie Blairite” faction in the shadow cabinet. New Labour loyalists suspected Miliband’s office of tolerating the attack because it deflected the left’s disappointment with the leader. Ed’s loyalists felt his Blairite critics were secretly toasting David, voluntarily exiled in New York, as king over the water. David’s former supporters drew parallels between Ed’s accession and Brown’s coup against Blair in 2007 – seizing the crown by subterfuge, then proving unfit to wear it. That dynamic generated a background buzz of briefing and counter-briefing that routinely drowned out Ed’s public interventions.

A pattern was set. Miliband’s office would spend most of the year bogged down in low-level Westminster skirmishing. Then, around June, the tiny cadre of trusted advisers would begin on the next conference speech, notched on the calendar as a chance to stand up from the defensive crouch and advance their big argument.

In autumn 2012, the theme was “one nation”. Miliband drew upon Labour’s historic ethos of solidarity – symbolised by the post-1945 creation of the welfare state – while poaching Disraeli’s metaphor for a socially conscientious strain of conservatism. It was a bravura performance that won praise even from Tory commentators. In autumn 2013, Miliband tried to execute a pivot from “highfalutin ideas” to retail. He pledged to freeze energy bills. The move was intended as a show of courage – a readiness to take on powerful “vested interests”. It was also meant to start a new conversation about the economy based on the “cost of living crisis” – as opposed to the Conservatives’ preferred topic of budget discipline.

Facebook Twitter Pinterest Miliband hugs his brother, David, after being elected Labour leader in 2010. Photograph: Martin Argles

Beales had come up with the idea after witnessing rage against the energy companies in his focus groups. He included the freeze in an April 2013 email to Miliband with a short list of headline policies that would “go to the heart of his vision for a changed economy”. (Another one from that list – abolishing the non-domicile status enjoyed by wealthy tax avoiders – was stored away for deployment at the start of the election campaign in 2015.)

The price freeze was nurtured in deep secrecy. Torsten Bell, a former Treasury adviser who was admired across the party’s factional divides for his ferocious attention to detail, stress-tested the plan for legality. Over the summer of 2013, only Beales, Wood, Bell and Miliband knew about the plan. They were counting on surprising the Tories and provoking them into an outraged defence of the despised energy companies. It worked. Downing Street was rattled by the popularity of the freeze. Miliband started telling friends he had Cameron “on the run”.

But each autumn of accolade would yield to a winter of stagnation. There seemed to be no method for turning the one-off hits into winning streaks. Voters were still telling pollsters they didn’t trust Labour to run the economy or think that Miliband was up to the job of being prime minister. Labour MPs were reporting that their constituents did not have a clear idea of what the party was for. Miliband was frustrated that the shadow cabinet was not helping to illuminate or sell his vision. “I have to be my own outrider,” Miliband would often complain to his team. The frontbench would in turn grumble that their voices were being stifled by the disciplinarian system for signing off announcements in case they ranged off-message or implied spending commitments (a stricture that gave Balls considerable power of veto). On one occasion, when Miliband was lamenting his shadow ministers’ reluctance to speak out, one of his media team replied: “But Ed, we don’t let them say anything.”

In the doldrum periods between party conferences, MPs would complain about their leader’s apparent seclusion with the “brains trust” in his office. He would listen enthusiastically to their concerns and yet they felt their input disappeared into a fog of conflicting counsel. As it happened, advisers had the same frustration: leaving a room having settled on a plan with Ed, just as someone was entering by another door to unsettle it. Under pressure, Miliband could make snap decisions and was stubborn in his resolve – the attack on Murdoch; the peremptory sacking in November 2014 of the shadow attorney general, Emily Thornberry, for a single tweet that reeked of metropolitan hauteur.

But often his modus operandi was to bring more people into the room, harvesting contradictory opinions and letting them simmer before acting. It was a system that, according to one contributor, “rarely resulted in a choice being made that was more radical at the end than at the beginning of the process”.

* * *

On 8 April 2013, Miliband was on a train from Ipswich to Cambridge, campaigning for local elections, when the news came through that Margaret Thatcher had died. He had about 20 minutes to absorb the news before meeting a Sky news crew that had been deployed to record his reaction. He called Harriet Harman, Labour’s deputy leader, to discuss the outline of a response, and then wrote a script on his phone, emailing bits of text across the table to Tom Baldwin, one of his media strategists, and Marc Stears. The challenge was to be solemn in deference to what many in the country would feel as a national bereavement, and mindful of the likely reaction of those on the left for whom Thatcher had been a despicable nemesis. Miliband praised the force of her character – the political will that had transformed a country – while acknowledging that the same traits had divided the country. This ambivalence towards Thatcher was not feigned. He may have despised her politics but he sincerely admired the conviction that had seen her defy the contempt of the Tory establishment to become an epoch-defining prime minister. A discreet shade of vanity led him to suppose their stories might even be analogous.

Watching the scene on the train with bemused detachment was Arnie Graf, a septuagenarian community organiser from Chicago. Graf had recently become an avuncular figure at Miliband’s side, preaching the virtues of grassroots campaigning, building a movement beyond the usual party machine; teaching citizens to take charge of the politics in their own neighbourhoods. Miliband was passionate about these ideas, and full of admiration for Graf personally as a veteran of the US civil rights struggle. It appealed to his affection for the crusading wing of American liberalism; Miliband’s political hero is Robert Kennedy.

It was because of Graf that I had been invited on that train. Miliband’s press team wanted me to see his work in action. They also wanted to show that, away from the airless confinement of Westminster, Miliband had a charismatic capability. It was true. Before the Thatcher news landed, I had seen Ed address Ipswich town centre from a soap box, speaking off the cuff, taking hostile questions from shoppers and winning some of them over with a self-deprecating candour that never came across in parliament.

It was the character that some of his early supporters had told me was lying dormant in Miliband, the man they knew in private as witty and empathetic. It was certainly a different Miliband to the one I had met on a handful of occasions in his London office, where vintage Labour campaign posters on the walls made the place feel more museum than engine room. That man was guarded, brittle. The common charge that his natural milieu was either the committee room or the university seminar rang false. Inside the machine, he became more mechanical. The campaign trail suited him.

Graf had been introduced to Miliband by Maurice Glasman, another academic and theorist of “Blue Labour” – a school of thought based on a double critique of New Labour as too complacent about market forces and too reliant on bureaucratic state methods for effecting social change. It urged the rediscovery of political terrain – patriotism, faith, family values – that the left seemed to have ceded to conservatism. Stears, the “bluest” of Ed’s team, had also tried to nudge his old friend in this direction.

The Blue Labour flirtation began late in the autumn of 2010. As Miliband felt the parliamentary party was an inhospitable climate for his ideas, he sought refuge in an informal Sunday afternoon salon of leftwing academics, MPs, writers and brains-trust advisers, usually convened at his home. Over the next two years, the group would debate how to turn “responsible capitalism” into a mass movement. Miliband was energetic and attentive in the sessions, but the participants started to suspect that it was a weekend affair. One called it “Ed’s Glastonbury”: “You smoke some political dope, listen to a few bands, but then Monday comes and you put the suit back on and disappear into Westminster.”

Miliband sought refuge in a Sunday salon of academics, MPs, writers and advisers. One called it ‘Ed’s Glastonbury’

In January 2011, Miliband asked Graf to review the state of Labour campaigns, and was sufficiently impressed with the result to ask that he stay on and start training organisers. Rave reviews started coming in: thronging meetings, a new volunteer army, queuing round the block. But in party headquarters there were misgivings about the use of scarce resources on a scheme with no proven record of delivering votes. The May 2013 local elections had shown no measurable uptick in support that could be attributed to Graf’s work. Doubts about the method escalated into a full-blown culture clash after the shadow foreign secretary, Douglas Alexander, was appointed chair of general election strategy in October 2013. Alexander had run the 2010 campaign, which had seen Labour strategically marshall its resources to eke out a more creditable haul of parliamentary seats than might have been expected from a dismal national vote. He believed in rigorous data and clear chains of command. Graf saw such managerialism as part of Labour’s problem.

In January 2014, the Sun ran a story about problems with Graf’s immigration status and he was forced to return to the US. His devotees believed someone at HQ had leaked the visa story. Everyone in the hierarchy denies it. “A cock-up, not a stitch-up,” according to one senior party official.

Either way, Graf’s disciples felt they were losing a battle for the leader’s soul. There was, they said, a “Blue Ed” who was ready to break the mould of conventional party politics and a “Brown Ed” who preferred the familiar path of central party command-and-control. The Brown streak prevailed, according to one Labour-left veteran and salon regular: “Maybe a lighter shade of Brown, since there was a deeper Brown group around Ed Balls, but they were all schooled in the same way of not taking unnecessary risks.”

When Miliband reshuffled his shadow ministerial team in October 2013, he promoted a number of “clean skin” MPs who had first been elected in 2010 and who, it was hoped, would be uncontaminated by the old Blair-Brown vendetta – true Milibandites. But the top positions were still held by members of the old Brown circle: Alexander, Balls, Yvette Cooper. Some of Miliband’s aides counselled against handing Balls the Treasury portfolio and, later, Alexander the campaign strategy job. Both times Ed ruled that weight of experience trumped other concerns. “People tried to portray it as weakness,” Sadiq Khan, one of Miliband’s oldest allies in the shadow cabinet, told me. “I see it as a sign of Ed’s basic decency. He sees the best in people.”

Whatever character trait was expressed in Miliband’s management of the party – some say ruthlessness, some indecisiveness, others generosity – the result was that he entered 2014, the last full year before the election, with the party held together in expedient unity. Labour were ahead in the polls. Even the many sceptics thought they might win – but in spite of Ed, not because of him. He had defied forecasts of civil war, but at a price. But it still did not look, from the outside at least, as if the Miliband candidacy would be an assertion of the argument he had made in speech after speech. Voters still seemed puzzled about what Labour was all about.

Facebook Twitter Pinterest Miliband and Ed Balls at the Labour party conference in 2013. Photograph: Matt Cardy/Getty Images

* * *

“It has to be about more than just how many microwave ovens people have got,” David Axelrod told Miliband. “You have to raise it above the retail.” It was 15 May 2014, and Barack Obama’s chief campaign adviser was talking kitchen-table politics with the Labour leader over a polished oak table at Corrigan’s, a smart Mayfair restaurant – chosen for privacy rather than grandeur.

The seeds of their meeting had been planted three years earlier, at a reception for the US president in Buckingham Palace. Miliband struck up a conversation with David Plouffe, Axelrod’s colleague on the Obama campaigns and his partner in the political consultancy AKPD. There were more contacts over the following years, although it was only in summer of 2013 that the prospect of AKPD taking on the task of sharpening Miliband’s candidacy started to seem plausible. The deal was brokered by David Muir, a former Downing Street adviser now based in Washington; in April 2014, Axelrod was signed up for a fee said to be £300,000. Plenty of Labour officials baulked at the sum, seeing the appointment as the expression of an unhealthy obsession with US campaign wizardry and questioning Axelrod’s commitment. (Even aides who extol the acuity of his advice concede that his focus has been sporadic.)

The Corrigan’s dinner came at the end of two intensive days in which Miliband, Alexander and a select group of advisers locked themselves away for a gruelling dissection of strengths and weaknesses in the project. Axelrod was enthusiastic about the ambition of a new economic settlement and blunt about the challenge of setting Miliband up as its champion against David Cameron, whose polished manner conformed more naturally to voter expectations of what a prime minister should look like. “You are someone who has a distinctive vision of how you want Britain to change. If you let the election become a competition between two conventional politicians you won’t win,” Axelrod said.

That warning was borne out in European and local elections the following month. The tone of the campaign had lurched away from big ideas. Policy announcements had been sprayed out in rapid succession – out-of-hours access to doctors; rent controls; a higher minimum wage. It had not coalesced into an uplifting story. Focus groups showed that swing voters were unimpressed by Tory claims to have sorted the country’s economic problems, since they were not feeling the benefit in their pay packets. But nor were they expecting things would be better under Labour.

The European campaign also culminated in presentational horror. On the eve of polling, Miliband was photographed devouring a bacon sandwich. Shots of a hunched, gurning Labour leader went viral. It was impossible to ignore reports coming back from the campaign trail that voters were citing doubts about the leader as a serious obstacle to voting Labour.

After winning the leadership, Miliband had not devoted any effort to constructing a public persona. Privately, he drew a distinction between his high-minded approach and that of David Cameron – who had taken gimmicky shortcuts while in opposition to advertise a new brand of compassionate conservatism, nuzzling husky dogs in the Arctic to show care for the environment. Miliband’s media team had to operate under a “no huskies” rule. He was naive about how pictures define a leader. He thought his ideas would speak for themselves.

Miliband also found the thespian demands of modern politics uninteresting. His public performances were stilted. His critics inside the party felt the hesitant delivery was a manifestation of political indecision – unreadiness for the job. Opinion polls suggested the public was taking the same view and hostile newspapers amplified it. Some of this was a reflection of prevailing opinion; much of it was cynical. After one well-delivered speech in January 2015, a political editor congratulated Miliband for his stirring oratory, and then added: “I’m sorry about all the terrible things we are now going to write about you.”

As the Labour leader’s oldest friend in the team, Stears was the one who could persuade Miliband to confront the problem. He had already helped Ed weave more biography into his speeches, telling the story of his parents’ flight from the Nazis and his father Ralph’s rise from penniless refugee to eminent LSE professor and Marxist theoretician. The backstory had been used to illuminate a favourite Miliband theme: restoring the “promise of Britain”. This was a reworking of “the American dream”, tapping into the anxiety of a generation laden with student debt, locked out of the housing market and doubting they would ever enjoy the same quality of life as their parents.

The remedy, Miliband believed, was “predistribution” – a term coined by Jacob Hacker, a Yale professor, to describe the need for government to fix structural obstacles to social mobility – low skills, patchy public services, stagnant wages – instead of just skimming cash from the rich minority and handing it down to those left behind. Miliband’s press team had banned him from using the “p” word in public, since it fed the unhelpful perception of inaccessible intellectualism. Stears’s speeches tried to personalise the concept.

In July 2014, Miliband and Stears took the personal battle straight to Cameron. In a speech at London’s Royal Institute of British Architects, Miliband depicted the Tory leader as a confection of public relations techniques, a soulless political operator from central casting: “I am not going to able to compete with that. And I don’t intend to.” He argued that an obsession with image, sustained by Westminster apparatchiks and the media, was corroding trust in politics. It was the first public expression of the frustration that had been building since 2010: the ideas weren’t speaking for themselves, so Miliband had to be their incarnation.

That was also the thinking behind Miliband’s last party conference speech in September. The plan was to apply Axelrod’s ideas while tackling the problem that voters did not believe that a change of government would make a difference in their lives. Miliband tried to weave the big themes of previous years together – enterprise and social responsibility (without referring to “predators” this time), distilled into a retail offer to show that Labour was on the side of ordinary people – culminating in a simple choice between an insouciant Cameron who served powerful interests and a bold, empathetic Miliband who confronted them: “Leadership that will always stand for the privileged few or leadership that fights for you.”

Hopes of another breakthrough autumn were quickly dashed. Miliband’s delivery was lacklustre. Worse, he spoke without notes and forgot passages on immigration and the budget deficit – issues where voter confidence in Labour was lowest. Party morale tanked. For the first time since 2010, there was serious speculation among MPs about changing the leader. There was loose talk of parachuting in Alan Johnson, a Blair-era minister with a reassuringly working-class backstory and a genial, TV-friendly manner, but there was no mechanism to trigger a smooth transition and Johnson wanted no part in a bloody coup.

Miliband tried to reassert his authority with a mini-reshuffle of the shadow cabinet team, bringing Lucy Powell back in to work with Alexander on the campaign. Powell was important as one of the original believers – the select few who had encouraged Ed to run for the leadership. She also had a healthy impatience for the rolling, ruminative meetings, and a capacity to get quick decisions from Ed. As one senior campaign colleague put it: “Lucy’s the one who makes the trains run on time.”

After the disastrous 2014 conference, Miliband applied himself more determinedly to the craft of performance. He had taken professional direction for big speeches before, but the proximity of the election – and the prospect of televised leader debates – focused his mind. He pointed his studious temperament at the science of communication, deploying some of the energy he had always spent scouring policy documents to the challenge of being persuasive on camera. He became steadier in his gaze; his voice lost some of its nasal twang.

Facebook Twitter Pinterest Miliband in 2014. Photograph: Oli Scarff/Getty Images

The Tories had always said Miliband’s image problem was their passport to victory. Ed’s allies believed the opposite: that public attitudes would soften when voters got to know him better; that the enhanced exposure of a general election would reveal the man they had always known – the one I had glimpsed in Ipswich – and banish the hapless caricature. The opening weeks of the campaign have appeared to support that view. Miliband performed better on TV and looked more relaxed in interviews. Opinion polls showed a spike in his approval ratings. Since becoming leader, there had been too many points when he appeared to have so overthought the message that he had forgotten how to be himself. At the 11th hour, he remembered.

Practice had something to do with it. More important was the leader’s unerring self-belief. He had carried the flame of his original idea across five years, when gales of internal dissent, media contempt and Conservative attacks had threatened to extinguish it. The project had necessarily been marked by compromise. It had become a hybrid of New, Blue and Brown Labour. The slate of retail policy may not, to many on the left, look equivalent in scale to the mission of rewriting Britain’s social and economic order – a higher minimum wage, apprenticeships, a gentler trajectory of budget consolidation to protect public services, more midwives, free nursery places and nurses. Some of it looked like old-fashioned redistributive social democracy: taxing the very rich to pay for the NHS.

But for Miliband these were emblems of intent – flares sent up to illuminate the motive that had impelled him to run for the leadership five years earlier. Opposition had imposed severe constraints, not least the obligation to publish a manifesto whose first page delivered a commitment to budget discipline. But Miliband felt that need not extinguish Labour’s fervour for radical change. And he has revealed enough evangelical ability at the end of the journey for the party to get behind him with more relish than it had previously shown. Miliband often struggled to be heard, but he never lost sight of what it was he wanted to say.

Whether less partisan voters see it that way is another matter. Miliband has not amassed a great army of followers along the way; at times it felt more like a siege than a crusade. But for his small band of trusted advisers, it has been a triumph of intellectual consistency over political volatility. They see a potential prime minister whose platform is a concrete extension of the idea that brought them together in 2010: a belief that Labour could win without compromising its historic determination to fashion a more equal society.

If he fails, there will be no shortage of critics ready to point out the flaws in the original idea and its subsequent execution. Yet none can deny that there is a principled resilience that has driven him on. If Miliband makes it to Downing Street, it will be a vindication of his earliest conviction – that his vision for Britain would, in time, vanquish doubts about its bearer. “If you let that side of you shine through,” Axelrod had told him over dinner in May, “the man who has a cause he believes in and is prepared to lose for – then you can win.” •

• Follow the Long Read on Twitter: @gdnlongread