I was a teenage Corbynite and grew up to be an employee of Ken Livingstone and John McDonnell, as well as Tony Blair and Gordon Brown. Because of this chequered past, I have seen, up close, their virtues and their vices. Having written a book entitled Politics in an Anti-Political Age many years ago, I am not surprised by this latest eruption of hostility to the political class and I welcome challenge to conventional wisdoms and the breaking of taboos, especially in economic policy. Parties need periodic upheavals to remember what they are for. But they also need the humility to learn from the world around them and an ability to empathise, not just with their own side but also with those who do not automatically support them.

I first became involved in politics in the constituency of Hornsey, where Jeremy Corbyn was the agent. I doubt he remembers me but I spent a fair amount of time in his genial company. I enjoyed helping to organise jumble sales (an underrated but essential political skill, though not one he was all that good at) and canvassing often angry and reluctant voters. I was then on Labour’s far left and took part in feverish discussions with him and others in the Labour Party Young Socialists that echo today’s arguments.

Then, as now, we discussed the betrayal of the Parliamentary Labour Party and what we considered to be the moral ambiguity and occasional corruption of the previous Labour governments (of Wilson and Callaghan) and their failure to change the system. As we sat talking earnestly in our damp houses and flats, piled high with books and parcels of the unsold weekly papers that were an odd fetish of the Trotskyite left, we put our faith in Tony Benn as the standard-bearer of a more decent and radical politics and, despite our tendency towards Groucho Marxism (“Whatever it is, we’re against it!”), we were serious about changing the world for the better.

Fairly soon, I was brutally expelled as a heretic from the group that I had joined. Academic politics is notoriously vicious because the stakes are so small, and the same was true of the Trotskyites (which is perhaps why so many ended up as academics).

A few years later, I found myself in another milieu that has also just returned to centre stage against the odds. My first proper job was as a civil servant with Ken Livingstone’s Greater London Council (GLC), then a bastion of progress facing off against the reactionary power centres over the river in Westminster and Whitehall. Looking back, I think it was ahead of its time in taking equal opportunities seriously, deepening democracy and pursuing economic ideas that became conventional in business schools a decade later.

It also accumulated vices. It was often poor at management and, at times, flagrantly wasteful of money. Its leaders seemed more comfortable with giving speeches and holding rallies than with making things happen and appeared to prefer a fantasy world out of Eisenstein films (in which they would be carried to power on the shoulders of cheering crowds) to the hard grind of achieving change. To my surprise, I came away from the GLC with a greatly enhanced respect for the boring virtues of good management and administration.

The GLC was lively but also chronically factional. I worked for a time in John McDonnell’s staff and saw at close quarters his cleverness and his keen interest in power, which led him to take on the unglamorous but strategic finance committee. Before long, he attempted a putsch against Livingstone, whom he seemed to think of as rather wayward and indulgent, and there was always some plot under way.

Luckily I was fairly removed from the knives being stuck into backs. Instead, my work at the GLC focused on the creative industries, jobs and investment. I was part of a group that became rather heretical advocates of markets, since it was obvious that markets were giving more opportunities for small-scale creatives in fields such as music, publishing and TV than public subsidy, which was usually captured by the London elite. (An extraordinary proportion was being spent on the opera, for example.)

The ideological commissars hated this conclusion. Yet there were enough freethinkers around to keep them at bay and we embraced ideas of flexible specialisation and post-Fordism that pointed to a much more dynamic view of how a future economy could work. It seemed clear that the old-left arguments for wholesale nationalisation and planning were anachronisms.

Indeed, it felt as if an entire approach to politics was on its last legs and, in the years after the GLC, I was involved in two very different attempts to bring Labour kicking and screaming into the late 20th century, both of which have been cast into a new light by Corbyn’s victory. One was Red Wedge, a collective made up of musicians such as Billy Bragg and Paul Weller and comedians including Lenny Henry and Ben Elton. I had a weird mix of roles ranging from van driver to company director, along with Bragg and Weller.

The stars’ main motivation was to persuade young people to vote and displace the all-conquering Thatcherites. The concerts were also combined with open meetings – in the hope that Red Wedge could help reanimate Labour’s then stagnant political culture, which was, paradoxically, symbolised for us by people waving around their tiny-circulation Trotskyite newspapers.

We had a great time and Red Wedge ran fantastic events. We were, however, victims of hope over experience. We failed to dislodge Margaret Thatcher. We also failed to shift the Labour Party’s culture of stultifying meetings, overseen by serried ranks of ageing white men in suits and woolly cardigans making formulaic speeches. Like so many before and since, we mistook the exhilarating enthusiasm of the thousands of people who came to our events for a shift in the mood of the tens of millions of other people who weren’t there. On the morning after the election of 1987, I was not

just disappointed that Labour had been crushed at the polls. I felt foolish for not having understood better what was happening around me.

The other attempt at transformation was intellectual: the attempt by the journal Marxism Today to reach out far beyond the comfort zones of the left with a deliberately promiscuous exploration of ideas and theories and, above all, a curiosity: about why the Conservatives were triumphing, why they were winning the battle of ideas and why the British working classes appeared so unwilling to fit the roles prescribed for them by the traditional left.

I had spent a year at the Massachusetts Institute of Technology learning about ­telecommunications from the people who were then inventing the internet and took it for granted that digital technologies would be a more revolutionary force than the trade unions or the newspaper sellers. I tried, unsuccessfully, to persuade the intellectual gurus – Eric Hobsbawm, Stuart Hall, Anthony Giddens, and so on – to think in terms of networks and different ways of organising the state. Marxism Today was at least open to these ideas and, although it was very much of its time, its spirit of openness and curiosity has much to teach the parties today.

My experiences made me a ready convert for New Labour, working first for Gordon Brown and, later, for seven years in government under Tony Blair, including as head of the Downing Street Policy Unit and the government’s strategy team. I saw at first hand the many strengths and weaknesses of that kind of politics. I took little convincing that there was no sense in being a pure but powerless opposition and had lost all patience with indulgent rallies and meetings. I wanted government to use its considerable power to do good: to cut poverty, rebuild public services and give people justified hope. I didn’t believe then – and don’t believe now – the facile claim that governments no longer have any power. They do, which is why politics matters.

I also believed that government should encourage entrepreneurship and innovation and that many of the best ideas would come from people taking risks, often on the margins, and not just from senior officials, politicians or professors at grand universities. It was obvious then, as it is now, that governments play a decisive role in innovation, funding many of the fundamental technologies that end up in smartphones or new drugs. It was also apparent that governments are pretty hopeless at turning those technologies into useful products and services. Corbyn and McDonnell accept the first proposition but not the second, which is one of many reasons why their economics risks looking implausible and incomplete.

Most of what Tony Blair promised was achieved, beyond my expectations, with the longest period of sustained economic growth in history, poverty reduced, huge improvements to public services and fundamental devolution. But, as often happens, what were once virtues turned into vices. The virtue of discipline became the vice of numbingly bland conformity. The necessary adoption of parts of the Tory/Saatchi model of centralised parties constructed around national advertising campaigns displaced the direct, movement-building politics that had kept Labour – and the Conservatives – alive at the grass roots. In government, a healthy focus on getting results reinforced old-fashioned silos and overcentralisation. Labour’s success in attracting bright new politicians became a vice when too many jumped straight into advisory roles and ministerial posts, in which the highest ambition was to get a well-honed soundbite on to the evening news. It is not surprising that so many appeared ill-prepared and inauthentic when they offered themselves up for leadership.

More than a century ago, the sociologist Robert Michels wrote of the “iron law of oligarchy” that he perceived in Europe’s social-democratic parties, as movements that were set up to change the world became machines to keep their leaders in power. I continue to be amazed at how many politicians today speak as if it were obvious that the only purpose of their parties’ existence is to win elections. I can understand why that is so important for them. For everyone else, however, politics is a means and not an end and holding power is desirable only if you know what to use it for.

***

This is why part of me welcomes the turmoil of a Corbyn victory. Nietzsche’s comment that what doesn’t kill you makes you stronger isn’t exactly true. But politics does need challenge and crisis to rediscover its inner core, and what is true of Labour has often been true of the Conservatives and Liberals, too. It is through argument – robust, passionate and often bad-tempered – that new truths are found. Labour had forgotten how to have these

authentic, open arguments.

But the other source of renewal is practice: listening and learning from the doers. Movements such as Podemos in Spain have their roots in civic action rather than in the residues of Marxism-Leninism. One of the many odd features of Corbynism is that it appears rather uninterested in what every­day radicals are doing – the grass-roots pioneers in fields such as food or recycling, mental health or elderly care. This could be a fatal weakness.

After a few years in government, I became deeply sceptical of the idea that policy is best designed either by small groups of experts in London or by composites and committees. Both approaches privilege words over practice and often generate policies that come badly unstuck when implemented. I followed the example of Michael Young, the author of Labour’s 1945 manifesto, by turning back to the grass roots and working with innovators and entrepreneurs creating real-life answers, rather than solutions on paper. Through the Young Foundation and the National Endowment for Science, Technology and the Arts (Nesta), I became part of the extraordinary global movement for social innovation that is offering a much more enlivening approach to politics than the ones that I was brought up with.

Its underlying ethos is that if you see a problem, you should try to act on it yourself, instead of just waving placards telling someone else to act, and that if you want to change the world, you should test out your ideas on a small scale before taking them big. This should be natural territory for Labour. Yet David Cameron has often looked more in touch with this movement than the Labour of either Ed Miliband or, so far, Jeremy Corbyn. Here in Britain, the “big society” idea may have been at best incoherent but it did reflect an eagerness to learn from society and a recognition that practice would often be far ahead of theory. Angela Merkel has also spotted the importance of this space and, over the past two years, has attempted to integrate the ideas of social innovation

as well as happiness as a societal goal into her Christian Democratic programme.

The core left has been slow to understand this shift. Yet it could choose what the Brazilian thinker and now minister Roberto Mangabeira Unger describes in a recent book, New Frontiers in Social Innovation Research (published by Palgrave Macmillan), as “maximalist”, as opposed to “minimalist”, social innovation. The minimalist view puts social innovations squarely within the third sector and: “Its resonance is with solidarity and communitarianism . . . with the tradition within classical liberalism that prizes voluntary associations as well as with the strand within socialist thinking that proposes a non-statist socialism.”

By contrast, the maximalist view, Unger writes, is concerned with “the whole of society, of its institutional arrangements, and of its dominant forms of consciousness . . . At its maximalist best, the social innovation movement [undertakes] the small initiatives that have the greatest potential to foreshadow, by persuasive example, the transformation of those arrangements and of that consciousness.”

This is a debate that has scarcely started in Britain. Yet it offers a different approach to linking local devolution with national policy, the creativity of the radical margins with the power of the centre. It also offers ways to advance thinking on some of the big questions. For example, experiments on basic incomes in Finland and the Netherlands may illuminate whether this truly is part of the future of welfare. The many attempts to adapt schooling to fit the lives of the future better may prove more fertile than tired debates about local authority control and free schools. Similarly, the global movement to put self-management and peer support in health care on a much more equal footing with care in hospitals offers far more energy than yet another argument about top-down reform.

Some issues cannot be addressed in this way. Constitutional reform is hard to do in small steps. Perhaps the biggest social issue of the next decade – how to achieve a fundamental redistribution of assets to reverse the vast recent shift towards more unequal wealth – will also require the power of national governments. In most of the fields that dominate political argument, however, the iteration between micro and macro offers a better route to solving complex problems than drafting resolutions or passing new laws.

***

So what might we expect now? I doubt that a Corbyn leadership will need much external conspiracy or pressure to buckle. Political leadership is a craft, like plumbing, nursing or teaching, only more so, and it takes time to learn its many dimensions. The best leaders often have long apprenticeships before they become good at the job. It is bad enough that so many of today’s leaders had never run anything and jumped straight from being special advisers to becoming ministers responsible for budgets of billions. It is not much easier to leap from being a backbench MP to becoming a party leader. Even a genius would fumble and stumble.

I am also sceptical about whether the party can be run for long as a movement. Parties periodically need to regain some of the energy of movements but movements are different in nature from parties that aspire to govern. Movements are about opposition, anger and mobilisation: they derive their energy from the tension between them and the world around them. They are hot, whereas parties aiming at government have to be cooler, more solid, designed for compromise, for programmes and for action. Internal democracy makes perfect sense for a movement. But unless the membership reflects the public, it can be disastrous for a party seeking wide support from an electorate concerned about its interests, or the competence of its prospective leaders. That is why – as parties as varied as the SNP and Syriza have discovered – movements have to mutate when they turn into serious parties, losing some of their vital qualities but gaining others.

Similar considerations apply to readiness for government. You can be great at being a movement or party and lousy at running a town or nation. That requires many pretty boring, practical skills, such as handling budgets, or thinking through the realities of how people might react to policies. It is a very long way from running a demo and has always been undervalued by the far left and the populist right, whose metier is writing, talking and often shouting. Indeed, both are even less interested in the experiment that I also see as vital for good government – trying things out to find out what works, an approach almost opposite to government by diktat, decree or resolution. This is well understood by Labour’s many foot soldiers embedded in local government.

Yet the Trotskyite milieu in which Corbyn, McDonnell, Livingstone and others were formed took a different view. Much was made of the “transitional programme”, an idea developed by Leon Trotsky in the late 1930s. This was a manifesto or platform put forward by a left-wing party in elections that was not designed to be put into practice but was rather meant to be impossible. Having such impossible demands rejected would energise the workers, disorientate the enemy and accelerate the contradictions opening the way to revolution. Any incremental gain or compromise had to be rejected, as things had to get worse for the revolution to have any chance.

I do not know if Corbyn’s team still believes this. It is possible that they may not really care if their programmes are plausible or coherent; what matters is the effect. Yet “impossibilism” also has another unwelcome consequence – it militates against learning. If you do not care whether your programme is realistic or not, you have no need to grapple with the big and difficult questions: what to do about jobs and automation; how to transform the health services to cope with ageing populations; how to handle migration; how to overhaul the state. Like many of the academics advising the new Labour leadership, you can enjoy analysing the problem but avoid having to answer the question: “So, what would you actually do?”

In the 1990s, Tony Blair and Gordon Brown, like David Cameron and George Osborne in the late 2000s, were hungry to understand what made their opponents tick and where the world was heading. There is not much evidence of any comparable hunger now. The new group Momentum could be one source of change but, so far, it has offered only vague rhetoric, rather than showing any appetite for unsettling ideas and practice, or empathy for the unconverted. What that may presage is a dumbing down at the precise moment when Labour, like any opposition party, should be encouraging a thousand flowers of creative imagination to bloom. After all, no “transitional programme” ever led to a revolution, or an election victory. Trotskyism turned out to be one of the narrowest cul-de-sacs of 20th-century politics. British politics will be the poorer if the Labour Party has just turned down one of its own.

Geoff Mulgan is the chief executive of Nesta and a senior visiting scholar at Harvard University in Cambridge, Massachusetts. He writes here in a personal capacity