‘The trouble with Ed is that he is just too miserable.” So a senior member of the shadow cabinet once said to me. It wasn’t a judgment on how the former Labour leader was dealing with defeat. It was said before the general election had even been called, offered as a critique of the draft 2015 Labour manifesto.

It was one of the more interesting charges levelled against Miliband. But it was also almost entirely wrong.

Labour under Miliband, the argument went, had a lot to say about the travails of our economy – growing inequality, the inadequacy of the minimum wage, the insecurity of zero-hours contracts – but too little about the party’s successes. The manifesto was, as a result, too full of gloom. It lacked the sunny uplands, the promise of a bright new tomorrow, which have been associated with Labour’s more successful electoral moments.

The economic order we have now simply does not work for millions of people and it needs to change

It was an argument we heard a lot in the first few weeks of Labour’s elongated leadership election this summer: that time when candidates launched their campaigns in bright, shiny meeting rooms with City backdrops and talked about aspiration and ambition, about the need to reposition the party in tune with business, about the desire to be the people who step boldly into tomorrow. It was an argument, I used to think, stuck in the time of Labour before Corbyn, until it reappeared in a slightly new guise in Peter Hyman’s call for fission in Labour ranks in this newspaper last week. But it was wrong before the general election, it was wrong in the leadership election and it is wrong now.

And it is wrong for a very simple reason: the economic order we have now simply does not work for millions of people and it needs to change.

That economy does not work because of the desperate inequalities it has created. Some people may be tired of hearing it but the top 1%’s share of our nation’s income has more than doubled since 1980 and their share of wealth now exceeds the poorest 50% of the entire population. But it is not just inequalities of income and wealth that blight our country, but of opportunity and power too.

The ever accelerating erosion of good-quality jobs – those that are dependent on high skills and that come with proper security – is not just bad for people’s financial prospects. It is bad for the economy as a whole, as talent goes to waste and innovation is held back. And it is bad for people’s chances to shape their own lives. Zero-hours contracts are just one of the most egregious examples of working practices that leave millions at the beck and call of companies that put short-term profit ahead of the wellbeing of their staff and communities. What kind of country is it when we now casually talk of the “gig economy”, referring not to the music industry but to young, ambitious people left to move from one low-paying, low-skill job to another?

Raising these issues is not miserablism. It is realism. And all the signs are that the problems will get worse. Trends suggest that technological advance is likely to eradicate still further well-paying, high-skilled jobs. And even if productivity increases, the rewards might not reach those who work. What’s more, the public services that both protect the poorest and promote a healthy economy in the longer run are set to reach breaking point in communities across the UK.

All of these trends might be altered or ameliorated by political action, of course. But we won’t achieve that with another project full of grand phrases symbolising little, but with a programme for far-reaching and concrete change: a programme that can help us collectively rework the fundamentals of our economy and our society. And that is true not just for Labour but for all political parties in Britain and across the world.

We do not yet know the full details of such a programme. No one would claim that the last Labour manifesto, or any of the others, possessed them. But we do know the basic building blocks. It is a programme that would put the enormous commitment and talent of our people together with the technological advantages of our age. This would halt the trend to ever greater inequality, create more collaborative and rewarding patterns of work, give employees and entrepreneurs a sense of real power in the economy, and reverse the priority given to short-term profit over employment that offers long-term, sustainable wellbeing. All parts of the Labour party and many others beyond ought to be able to agree to the essentials of this programme.

There are nonetheless many obstacles to designing and delivering it. The most obvious is the need for an electoral coalition to support it. Labour’s failure at the last election is testament enough to the scale of that challenge. But the causes of that failure are more complex than are usually imagined, residing far more in people’s deep and pervasive distrust of politics, politicians and their plans than in an attachment to the economic status quo.

Anyone involved at close quarters with the general election campaign saw evidence of that. I was privileged in April and May to meet thousands of people all over our country, both Labour supporters and supporters of other parties or none. The vast majority shared an intense anxiety about the future of our economy, our society and our planet. But what they did not share was any understanding of the causes of the problems or of the potential shape of the solutions. Instead, people widely agreed that no one associated with party politics knew what was going on in their lives or how to even begin to change it.

A feeling like that has occurred before, so we know it can bring surprising political results. Sometimes it has the paradoxical effect of encouraging millions of people to stick with what they know. Stanley Baldwin’s Conservative-led national government triumphed in the midst of the Depression not because of voter satisfaction of how things were but because of a deep anxiety that they could still get even worse. Alternatively, it can lead people to gamble with their nation’s future, pushing them into the hands of populists and demagogues, nativists and narcissists. In this year’s election, David Cameron reaped the rewards of the first trend and Nigel Farage the second.

In the face of this challenge, the right response for reformers is neither to abandon the aspiration that things could be different nor simply to dig in and hope somehow that people change their minds. It is instead to begin to close the gap between people and power.

The gap between the so-called elite and everyone else has grown too wide

It is one of the most wonderful features of our age that people long to have power over greater and greater aspects of their own lives, even if that comes, in part, from not trusting anyone else to do it for them. The days when a party elite, or a civil service, or a thinktank, could come up with a plan for reform and then put it to the nation in the expectation of passive acceptance are over. The gap between the so-called elite and everyone else has grown too wide.

Real change in the 21st century will only come when the programme for the future is authored by people themselves and when it resonates with the experiences of everyday life in the here and now. That is why the change we need is not best conceived in the hackneyed or grandiose terminology preferred by orthodox politicians of both left and right – the language of “a new Britain” or “hardworking people” – but in the way that impacts on the day to day. We require a politics that speaks with real familiarity of daily experience: young people unable to find decent housing, mothers and fathers unable to spend time with their children because they are working two jobs, the creeping loneliness and isolation in our communities, the endless frustration of faceless and unaccountable bureaucracies in both public and private sectors.

Our economy stands in need of fundamental reform. It will not get better by itself

To meet that call, political parties of all colours will need to learn to look outwards and not turn inwards. There is no future in the politics of faction or deselection any more than there is in the politics of splits. There is hope only in parties that are rooted in the diverse communities of Britain; parties that are willing to challenge settled opinion in some areas – entrenched attitudes to migration, for example – but who also confidently accept that they possess no monopoly on either wisdom or virtue.

What all of this means is that the challenge facing all of us who believe in a better, fairer, more socially just future has two dimensions. It starts with an ability to stare the stark realities of our current world in the face, not to hide from them or to wish them away. Our economy stands in need of fundamental reform. It will not get better by itself. Millions of people’s lives will worsen, not improve, if we do not act in ways that previous generations have not. But even though the stakes are that high, people will not simply sign on to a programme of reform designed for them by someone else. They will need to be involved at every stage, with an approach to policy formulation more open, more genuinely participatory, more grounded in real-life experience than anything we have seen before.

This is an unenviably hard task. It would be naive and dishonest to pretend otherwise. But we must hope that it is not impossible, because it is derived from a reality that cannot be wished away.

Marc Stears is former speechwriter for Ed Miliband, professor of political theory and fellow of University College, Oxford