To his fans, he's the fresh thinker ready to take on the coalition. To his critics, he's too quiet, too awkward and too leftwing. So who is the real Ed Miliband? And how does he still find time for Desperate Housewives and the local pub?

It is 6.30am at King's Cross station in London, early February; not a hint of dawn in the blustery streets outside. Inside, the shadow Welsh secretary Peter Hain is buying a pile of newspapers. Despite nine months out of power, Hain still has the air – expensive-looking coat, television-ready hair – of a traditional heavyweight politician. On the concourse, where the shadow cabinet are gathering for a visit to Gateshead, stands another: the shadow chancellor and ex-Labour leadership candidate Ed Balls, holding an ostentatiously enormous briefcase. Yet the station's other users pay Labour's finest little attention; the shadow ministers stand and chat slightly self-consciously, surrounded by a thin, Opposition-sized entourage.

At 6.50, with no fanfare, one final politician joins the group. Ed Miliband's status as party leader is apparent only from his late arrival and the fact that his paperwork is being carried for him. The shadow cabinet do not rush to greet or cluster round him. Like the commuters on the concourse, they carry on with their conversations.

Miliband seems happy to be barely noticed: 41, tall but slight, wearing an inexpensive-looking suit and unflashy shoes, his thick utilitarian haircut little changed from his first days, over a decade and a half ago, at the precocious age of 24, as one of Gordon Brown's advisers, he chats for a few moments with his personal staff. Then he walks to the train with them in long, flat-footed strides, the rest of the shadow cabinet straggling along behind. Only the extra police officers stationed on the platform seem excited as he passes.

Ed Miliband is a kind of political leader – and, judging by Labour's poll standing, increasingly likely prime minister – that Britain hasn't seen for quite a while. After Tony Blair's compelling mix of messiah and chatshow host, after Brown's furrowed intensity, after David Cameron's sly U-turns and retro ruling class smoothness, Miliband has followed a different path. In modern British politics, he tells me in our first formal interview a fortnight later, "There is a Blair-Cameron model. You come in [as party leader] and you keep moving your party to the centre, with a sort of shock-and-awe strategy." The reference to Blair's Iraq war, which Miliband has famously disowned as leader, is pointed. For Labour nowadays, Miliband suggests, tabloid-pleasing manoeuvres are no longer enough: "I think our problems are more complex than that."

But complexity may be a hard sell in a country as politically impatient as Britain – especially now, with the economy seemingly stuck in semi-recession and the coalition on reckless fast-forward. On Friday, it will be six months since Miliband was elected Labour leader, having beaten his brother David, the favourite, by just over 1% of the vote. Yet, apart from those last three facts, for the new leader of a major party, strikingly little about Ed Miliband has registered with the public. "A quarter of the public have no opinion of him," wrote Tom Mludzinski of Ipsos Mori in late January, "while the same proportion (37%) are satisfied as are dissatisfied with his performance. [And] the more the public get to know him, the more his negative ratings increase."

Ever since Miliband was proclaimed leader, looking a little wide-eyed, in front of a stunned Labour party conference last September, many Westminster players and commentators have seen his leadership as an unfolding or potential disaster. "Ed is Brown without the [poll] honeymoon," says a frustrated ally. "He hasn't looked comfortable in himself. On telly he doesn't look entirely honest: people think he's more leftwing than he's letting on. Actually, he's more emotionally expansive than he's letting on: he's got the emotional range not to be Brown. He has much more talent than David. But he has yet to articulate his vision."

To his critics, Labour as well as Lib Dem and Tory, Miliband is too passive; too quiet; too cerebral; too awkward; too adenoidal. He is too gauche and inexperienced; too nervous-seeming; too leftwing ("Red Ed"), and too dependent on the unions who were decisive in winning him the leadership. He lacks, it is said, a secure powerbase in a party that did not really want him as leader. "I would bet against Ed Miliband leading the Labour party into the next [general] election," says a well-connected Labour figure.

And yet it feels too simplistic to write him off. Firstly, few of those doing so gave him much chance of beating his brother in the leadership contest. Someone capable of defeating David – better-known, more experienced, more conventionally charismatic, endorsed by much of the New Labour establishment and almost the entire press – is perhaps not to be underestimated. Last summer, when David was supposedly miles ahead in the contest, I ran into a shrewd Labour MP and asked who he thought would win. "Ed," he said. "He's better at networking."

Secondly, Labour leaders are nearly always dismissed by the mainly Tory-leaning British press – whether they are frail like Michael Foot or formidable like Clement Attlee. The broader Westminster conventional wisdom can be just as unreliable: since 2007 alone, it has failed to foresee the financial crisis, David Cameron falling short of a Commons majority and the forming of the coalition.

As the son of Ralph Miliband, the late leftwing thinker who spent much of a distinguished career identifying the shallowness and short-termism in Westminster thinking, Ed Miliband can claim more convincingly than most politicians, perhaps, not to be obsessed by gossip and headlines. "Conventional wisdom can appear extremely set, and sometimes quite herd-like," the Labour leader says, sounding as dispassionate as a political science don (just before becoming an MP in 2005, he spent a year teaching politics at Harvard). "But I'm also struck by how quickly it can change." A tickle of dry wit lifts his flat voice: "… And suddenly people can't remember what their previous positions were."

Under Miliband, Labour have won their Commons byelections impressively: in Oldham East and Saddleworth in January, where they increased a tiny majority by a factor of 35; and in Barnsley Central a fortnight ago, where their share of the vote was 14% higher than in last year's general election. At prime minister's questions, the usually commanding Cameron has been unsettled by Miliband – mild-sounding and laconic, sometimes mocking – more often than parliamentary sketch writers expected. And in recent weeks, as Miliband has become more high profile and energetic outside the Commons, his ratings have improved and his Labour critics have gone quieter.

"The start was a bit slow," says Blair's former spin doctor Alastair Campbell, who supported David for the leadership. "But I have been impressed by Ed's air of calm under pressure. People who meet him find him very likable. And he is a guy who gets under David Cameron's skin." To these qualities, Miliband's advocates add an ability to think afresh – vital for a party that has just recorded one of its worst ever modern election results; a deftness at delegation and building bridges – crucial for winning over a clever shadow cabinet and, in the longer term, Liberal Democrat voters and politicians; a deceptive durability and cunning – Miliband survived 15 years at the heart of the Blair-Brown battle and emerged with his own prospects enhanced; a rare ability to listen; and, rarer still among the rather technocratic current generation of leading Labour politicians, an ability to connect emotionally and inspire. "Ed," says one prominent supporter who switched his allegiance from David during the leadership campaign, "can leave a room feeling better than when he came in."

His leadership, in short, is an experiment. Are empathy and thoughtfulness, plus awkwardness, the ingredients of a realistic Downing Street contender? Or is it now an unbreakable rule of politics that only shallower but slicker politicians become prime minister?

Our first interview is in the grand suite of Westminster offices provided for the Opposition leader. When Cameron had the job, the big rooms almost vibrated with the expectation of power: immaculately dressed aides striding about, Cameron lieutenants struggling to suppress smiles at the latest twist in the Brown government's downward spiral. Team Miliband has a more humble aura.

When I first met Miliband a decade ago, he was the geekiest but also the warmest and most open of Brown's inner circle, and he seems little changed. He is prone to slipping into policy jargon, and to producing robotic-sounding sentences such as, "Empathy is simply the end product of engagement." But he is also prepared to admit fault and properly answer questions. And he remains quite unguarded in his body language: leaning down awkwardly to tie a shoelace as the photographer snaps; later, blowing his nose on a large handkerchief on stage in the middle of a speech. It is as if he has not yet learned all the standard rules of being a party leader – or as if he never intends to.

He says Cameron's famous polish does not impress him much. "He's a very effective political performer. He's a good salesman. The question is, what is he selling?" Miliband forms the beginnings of a smile: "He got into trouble on [government cuts to] Sure Start at prime minister's questions today. When he doesn't have a response to my argument, he will tend to resort to slagging me off. It's his Flashman routine – it makes his side cheer, but I'm not sure it cuts much ice with the public."

What about his own, often criticised presentational skills? "I've been doing prime minister's questions for five months and he [Cameron] has been doing it almost six years. You get better at it. There are people who can come out and read out a speech very well – I acknowledge that is something I could probably do better. But…" – his level voice switches smoothly from defence to attack – "…I don't particularly like the formal, read-out speech. It feels very traditional and old-fashioned, to have a 'wise person' come out and read a speech to you. The times I feel most engaged with an audience are when I do a speech without notes, because it allows you to adjust to what the audience is feeling, and thinking. You're in a conversation."

At the shadow cabinet event in Gateshead, in front of a few hundred floating voters, Miliband starts in his stiffer public speaking mode. The catchphrases of his speech – "New Politics, Fresh Ideas" and "Helping Families Get On" – feel focus-grouped and bland. On stage he stands rooted to the spot, arms hanging at his sides, his posture unflattering, with shoulders slumping and hips forward. The comparison with Cameron's stage-striding energy at similar events is not favourable.

But once the audience questions start, Miliband is transformed. He is droll and self-deprecating. He draws people out: "Vanessa, tell us about your experience…" He takes the questions in batches but remembers every first name. He seems genuinely interested – much more than Cameron – in what people have to say. The event overruns. Afterwards, I ask a middle-aged local man sitting next to me, who had told me beforehand, "I wish his brother had won", what he thought of Ed Miliband now. "He's a better performer than I thought he was," the man says. "Better than on TV."

But what does he offer beside likability? Throughout Miliband's leadership and the long contest that secured it, a frequent criticism has been that he has yet to say what he stands for, apart from a vaguely defined new start for his party. In our first interview, he initially edges round the subject: "Ah, the vision," he says mockingly. Having briefed journalists for years for Brown, and having lived and breathed politics for years before that, at home with his brother and father and mother (the pro-Palestinian activist Marion Kozak), Miliband is a very confident interviewee; but sometimes he is too knowing about the rituals.

"I'm laying down some of the building blocks for where Labour is at the next election," he continues carefully. On Monday, he called for a new tax on bankers' bonuses to fund the creation of 110,000 new jobs for young people.

In the interview, he goes on to explain the thinking that will guide his other policies: "A lot of the faith that people had in markets has been shaken by what happened in 2008. The financial crisis has got to be this big moment of reassessment. Now, clearly you need markets… [But] if you're Labour, you believe that it's just not good enough to say the outcome the market produces is a fair outcome. You've got to find ways of intervening in that outcome." Shifting around on the sofa, he switches from Treasury-speak to something more concrete and demotic: "If you care about things like the gap between rich and poor, between the rich and the rest" – the "squeezed middle" has been one of his leadership's few resonant themes so far – "it's about having a critique of the injustices of capitalism… That partly defines my politics. It's the very important core of it."

Since the creation of New Labour in the mid-90s, it has been hard to imagine Brown – and impossible to imagine Blair – summing up their politics in such a clearly left-of-centre way. Is Miliband happy to call himself a socialist? "Well, the Labour party card says 'democratic socialist'."

Given his closeness to Brown, shouldn't he have done more to stand up for these ideas in government? "I think that's a fair criticism. I tried to influence things in the direction I believed in. [But] partly you are defined by your historical circumstances, and the circumstances of [the creation of] New Labour were… coming to terms with capitalism. It was hard to be a critic of capitalism. I'm not making excuses. But I'm trying to explain it – almost as a historian might explain it."

In the many accounts of the Blair and Brown governments, Miliband does consistently emerge as a left-leaning and unusually well-liked figure, more interested in tax credits for the poor than in poisonous feuds. But the inequalities of modern Britain, even after New Labour's 13 years of slightly furtive wealth redistribution, and before the coalition cuts really hurt, suggest that the achievements of his quiet brand of socialism have been modest so far.

Miliband's constituency is Doncaster North, a damp crescent of south Yorkshire countryside, commuter suburbs and ex-mining villages. It is not desperately poor, but full-time male wages are a fifth below the national average and the total number of benefit claimants a third above. His constituency office is in a small, beleaguered-looking business centre that also houses a debt recovery firm and several vacant units. In the surrounding terraced streets, scattered with litter and patches of waste ground, the residents I meet are hardly aware of their MP's existence, let alone how he is doing as leader. Doncaster North remains a safe seat, and Miliband increased the Labour vote at the 2010 election, a rare achievement nationally; but that vote has almost halved since the early 90s.

In London, he lives in a different world. He has a handsome tall house near Hampstead Heath with Justine Thornton, his partner – not wife, to the consternation of the Tory press and socially conservative parts of the Labour party. She is a rising young barrister specialising in environmental cases, and they have two sons under two. His mother lives close and helps out, and both boys are good sleepers, he says, but it is not the easiest domestic situation for a new party leader. When he took two weeks' paternity leave last November, a Labour leadership "crisis", partly real, partly imagined, rapidly filled the resulting political vacuum.

Attitudes to his broader life in north London, the archetypal middle-class Labour stronghold, can also be unforgiving. Miliband's father and grandfather, it is often forgotten, were Jewish refugees from the Nazis, Ralph Miliband was an itinerant radical academic, and his sons grew up and were educated locally (at a comprehensive) when north London was much less wealthy than it is now. Ed Miliband's upbringing was privileged in its political connections – at 16, he was already working for Tony Benn in his school holidays – but ordinary and straightforwardly aspirational in other ways compared with Cameron's or Nick Clegg's, with their private ski lodges and swimming pools and aristocratic relations.

Even as a teenager, Miliband had a striking ability to get on with people. "When I went up to Primrose Hill [where the Milibands lived], he knew everybody, even the people who sold you oranges," says his old friend Marc Stears, who now teaches politics at Oxford. Miliband was fun – "He was keen as anyone to go to the movies" – but politics framed everything for him – "After the film, he'd talk about how it represented various political issues."

Miliband says his off-duty life now is quite plain. "Sometimes people say, 'As a politician, do you lead an ordinary life?' Well, no. It's an odd life." He works and travels too much. He misses his family. At home he watches middlebrow American TV (Curb Your Enthusiasm and Desperate Housewives) and American sports, with all their geeky statistics: "I've always liked numbers," he says, suddenly animated. He and Justine, like most new parents, do not go out much in the evenings; at weekends when they are in London (they also rent a house in his constituency), they keep to a small orbit: a nearby bookshop, local pubs, a takeaway round the corner. "He's at his happiest in a group of friends, usually people who've got some connection with politics, or people he knows from America, telling jokes, being funny. He's very quick at working out people's eccentricities," Stears says. Miliband has not yet decided how much of this private life, potentially quite politically marketable, he is willing to publicise: "You want people to know who you are," he says, "but you don't want to expose your family."

I wait until our second interview to ask about David. We are on a train to Barnsley, where a walkabout is scheduled with Labour's victorious candidate. Miliband is more relaxed than in Westminster, relishing the byelection result, suit jacket off. But his caffeinated voice slows right down when we talk about the fallout from the leadership race. He says he and David are still talking; but, "I won't pretend it's been an easy year in our relationship. It's been difficult. But I sort of feel we're getting through it. The more that time passes, the easier it becomes. He gets on with other things. I set out my stall as leader. People adapt." Could David still return to the shadow cabinet? "Yes. It's absolutely conceivable. I think it's very much in his hands… whether he wants to come back in, or do something else, or continue to be an MP but pursue other interests."

Despite the brothers' studiedly benign public statements about each other since the leadership contest began, there is a common view that they will never be reconciled. The uncertainty of David's plans does not help. Ten days ago he suddenly resurfaced publicly, to make an impressive speech in London about how to revive the international centre-left. Predictably but also provocatively, he offered a much more New Labour set of remedies than his brother.

Yet those who foresee an endless fratricidal struggle ignore the fact that the brothers have a lot of prior experience at keeping family relationships afloat despite political differences. By their father's death in 1994, Ed and David had long left behind his fierce brand of leftism – his best-known book, Parliamentary Socialism, was an attack on Labour's instinctive caution – for the compromise politics of the Blair-Brown era. "We had disagreements, both of us, with our dad," Ed says. But he points out that all three of them remained very close. And for more than a decade and a half after Ralph's death, despite the brothers' positions on opposite sides of the Blair-Brown divide, and their potentially rivalrous upward trajectories, Ed and David never publicly fell out.

But even if they can gradually make up, for Ed to win over David's ex-supporters – or continuing supporters – may be another matter. "I think my leadership victory was a shock to many people," Ed acknowledges. "I totally understand a lot of people had supported David and felt very… bad for him." Given the last six months' near-constant sniping about Ed's leadership by some "David-ites", you could see this as Ed extending his celebrated empathy a bit far.

And yet sometimes he can be more sharp-edged. After a speech at a London thinktank last month, during which he had made some typically professorial but potent points about the roots of the current "cost of living crisis for ordinary families in Britain", a journalist in the audience asked Miliband about some criticisms of him that Peter Mandelson had made in the new paperback edition of his memoirs. "I think he's a writer not a quitter these days," Miliband said. The whole room, filled with Westminster insiders, got the reference to Mandelson's famously embarrassing "I'm a fighter not a quitter" speech in 2001 and dissolved in sudden laughter.

Miliband is tougher than his unlined, soft face and mild manner suggest. Two of his heroes are Geoff Boycott and Jimmy Connors, both bloody-minded and hard to beat, the patient constructors of immense sporting victories. Miliband has yet to say much about what he would actually do as prime minister – sensible opposition leaders don't so far from a general election– but his ambitions for Labour and the country are large. He wants his party to wield power more confidently: "When we are in government, we act like squatters who don't really have a right to be there. There is a sense of ideological confidence the Tory-led government have – I think misplaced – which is instructive."

He wants to reform the British economy, so it generates mass prosperity rather than social problems for the state. He wants to keep the welfare state affordable in a "cold fiscal climate", while also making it more responsive to citizens: "The New Left [including his father] were talking about that in the 60s. I don't think we've cracked the problem." He wants Britain to be more like Germany and Scandinavia and less like America.

Wouldn't all this take decades?

He looks into the middle distance from his Westminster sofa and says calmly, "It's not a political project for one year, one parliament… or even one political leader. Governments overestimate what they can do in the short term, but they underestimate what they can do in the long term."

There is already a little evidence that Miliband might be effective at this sort of big-picture government. Between late 2008 and last May, the fag end of the Brown government, he held his only senior ministerial post so far, as secretary of state for climate change. According to many environmentalists, in this short period he achieved more on the issue than any minister before or since.

But before he can start trying to transform the country, there is the small matter of the coalition, and its own, very different vision for Britain. Next Saturday, he is scheduled to address the TUC's big anti-cuts rally in London. It will be a stern test: for months, he has struggled to establish a clear position on the anti-cuts protests. When we speak in Westminster, he still hasn't. "I'm not distancing myself from the important protests there are against the cuts," he says stiffly. "[But] the judgment on me is not about how many protests I go on. It's more about…" He pauses. "Do I articulate the [anti-cuts] arguments in all the settings I have available to me… as leader of the Labour party?"

At moments such as these, his leadership feels a huge missed opportunity – or, at best, a work in progress. His allies justify this caution by saying Miliband is haunted by what happened to Neil Kinnock, the 80s Labour leader (and Ed supporter) who built up a huge poll lead in opposition and then lost it because, beneath his impassioned anti-Tory rhetoric, he did not have an alternative plan for governing properly thought out. They say that Miliband, by avoiding fiery posturing and instead working out his policies patiently and quietly, is sacrificing short-term political momentum for a momentum that will last. Yet it's a risky trade-off: sometimes voters outraged by a government need to hear fire from the opposition; sometimes Miliband is at his best when he improvises rather than calculates.

But when we reach Barnsley, such problems recede for an hour. He does his walkabout with the newly elected Labour MP, Dan Jarvis. Jarvis is a 38-year-old former army officer with a touch of Ralph Fiennes about him, and as the two of them, both youngish, tall and slim, stroll through the town centre, a town centre where even the most expensively dressed old ladies virtually spit when you mention the government, Miliband is commanding and spontaneous and charming to passers-by and party activists alike. This, in a tiny way, is what a Labour general election victory in 2015 might feel like.

Then two middle-aged men come into the square where Miliband has stopped to make a brief speech, and spot the Labour leader. One man turns to the other, and says in a voice of almost infinite south Yorkshire scepticism: "So he does exist!"