Work – and the lack of it – is one of the most explosive devices on the political road ahead. In all economies, whether complex or basic, workers at every level from unskilled to highly professional must increasingly struggle in, or benefit from, globalisation. Mostly, collective organisation in unions and representation in leftist parties do not help much and their influence even in European welfare state societies continues to diminish. New technologies are creating a labour market in which the only certainty is the destruction of jobs.

In the 2005 book New Division of Labour, Frank Levy and Richard J Murnane write that the coming rift will be between computers and people – and in the latter group, between “those who can and those who cannot do valued work” in the new economy. That is the vast and complex problem that every government must face and for which any opposition must prepare.

The current Labour leadership’s interests – as expressed in the speeches and writings of Jeremy Corbyn and his allies in the past decades – have not focused on this problem, instead trying to reconstruct a society that has collective values in work, expressed in state-owned industries and services, in strong trade unions and in pulling Labour away from New Labour and, above all, Blairism. Yet, absent a siege economy, it will be impossible for a left-of-centre party to recreate these conditions. The only route open to progressive, democratic politics is to understand the world that is being created, and to intervene in it as forcefully as possible to secure employment, greater equality and both good and adequate minimum wages.

It is hard for parties of the left to change to meet the challenges of the contemporary world and especially its labour market. And it is particularly hard in the UK, where the economy is so dependent on services – real driver of the UK’s present relative economic success – while the manufacturing sector is dominated by the closely linked sectors of defence and aerospace: largely repugnant to the new leadership.

The roots of left-wing parties were in collectivism. Their support came mostly from trade unions (which is especially true of the Labour Party) and their instincts, reflexes and myths were centred on the creation of a society stably resting on pillars of collective provision and organisation: raising the scarlet banner high, building a new Jerusalem. Few join the Labour Party to propose or debate ways of running a capitalist economy better. Yet that is precisely what – for the immediate future, at least – a responsible Labour opposition party must do, if it is to govern in the interests of labour.

As Labour goes sharply left, France is going, more slowly, right. The French Socialist government is performing, albeit less spectacularly, the same U-turn that François Mitterrand made in 1983, when he dropped large nationalisation and high tax plans and followed the grain of the market economy. François Hollande’s programme when he was elected in May 2012 was less ambitious than Mitterand’s plan to enlarge the public sector but he ran from the left, with promises of high taxes (a 75 per cent marginal tax on incomes over €1m), while saying that he disliked the rich and would bring in tighter regulation of the finance sector. His switch to economically liberal policies – his €41bn of tax breaks on companies and individuals over three years, together with public spending cuts – is a bid both to win back middle-class votes in preparation for the presidential elections in 2017 and to boost the business sector.

This approach has a new champion: Emmanuel Macron, the 38-year-old economy minister, described in a Nouvel Observateur profile last year as “jeune, beau et brilliant”. At an investors’ event in London at the end of last September, he claimed that France was embracing disruptive innovation, by which he meant that entrepreneurialism would be privileged and the state pulled back to allow private-sector experimentation and growth.

Macron has been explicit about the nature of the emerging labour market. He told the Economist in October 2015, “The left has built its history around the extension of collective rights . . . [Now] the transformation of the economy risks bringing to an end this adventure of collective progress.” The challenge, he said, is to “build a form of neo-progressivism, structured around the idea of individual progress for all, in a way that combines agility with security”. Blairism lives in the unlikely ambit of the French Socialist cabinet room, as it does in the reformist cabinet of Matteo Renzi’s Democratic Party administration in Italy.

Many of Macron’s comrades don’t like his talk and don’t like him – he was previously an investment banker at Rothschild & Cie and made a fortune arranging Nestlé’s $11.9bn purchase of Pfizer’s baby food manufacturer; he has also said, among other things, that the French 35-hour week, aimed at increasing employment, is “a false idea”. Still, he has a big cabinet job. There would be no chance of a British Macron getting into the present Labour shadow cabinet, even if the party attracted such a figure. Andrew Adonis – the former head of Tony Blair’s policy unit and an ex-cabinet minister with similar ideas to Macron – whose “what works” approach resembles Macron’s, has accepted an offer from George Osborne to chair his new national infrastructure commission, resigning the Labour whip.

Yet Labour desperately needs to attract people who understand the ways in which business and communications are now moving. What happens when driverless cars, intelligent homes, “faxed” production, “droned” delivery systems and robotised factories fully arrive? Levy and Murnane argue that policymakers must ask four fundamental questions:

1) What kind of tasks do human beings perform better than computers?

2) What kind of tasks do computers perform better than humans?

3) In an increasingly computerised world, what well-paid work is left for people to do, both now and in the future?

4) How can people learn the skills to do this work?

John McDonnell, the Shadow Chancellor, has shown most interest in this – arguing in a speech on “Creating the Economy of the Future” in November last year that “Labour in government will bring together business, unions, and scientists in a new Innovation Policy strategy, with a mission-led goal to boost research and development spending, and maximise the social and economic benefits from that expenditure”. The speech could have been a landmark statement, putting Labour firmly in the camp of the technology entrepreneurs and making out how the state could be directed to assist breakthrough developments.

Professor Marianna Mazzucato, a smart recruitment by McDonnell as an advisor, has written extensively on this – pointing out, in an essay on the Innovative State in January last year that many of the innovations coming out of Silicon Valley were pioneered by state aid – as the smart phone, which began life in the work of Joseph Licklider and his colleagues in DARPA, the Department of Defence’s research arm. But most of the speech was a routine attack on the Tory government in which the few lines on innovation and the state were lost, while McDonnell put off the real work of bringing together diverse interests to brainstorm and develop the issue till Labour comes to government, thuis wasting the possibilities for creative thinking that opposition grants.

In their 2014 book The Second Machine Age, Erik Brynjolfsson and Andrew McAfee write: “Technological progress is going to leave behind some people, perhaps even a lot of people, as it races ahead . . . There’s never been a worse time to be a worker with only ‘ordinary’ skills and abilities to offer, because computers, robots and other digital technologies are acquiring these skills and abilities at an extraordinary rate.” The people with “ordinary” skills are those whom the Labour Party should want to assist most, providing routes for preparation for employment in the emerging economy.

There is nothing like the concentration of new technology development and fortune-making that is Silicon Valley anywhere else in the world. It comes with a certain kind of arrogant naivety but just as other states had to send envoys to Manchester and the City of London in the 19th century to see why Britain had the leading economy, so every serious political force has to understand what is streaming out of the valley’s campuses and from the heads of entrepreneurs such as Hoffman, as foolish as their vision of universal enrichment through start-ups is.

As the campaign for Labour’s leadership was entering its final week in early September 2015, the annual Ambrosetti forum in Milan heard a slate of American speakers warning Europe that it was shirking a confrontation with hyper-modernity. Vivek Wadhwa from Stanford University showed that robots are transforming manufacturing; tech companies with the most data about citizens’ health are elbowing aside established health systems; solar panels are challenging the energy industry; global wifi is carving into the markets of telecom companies.

The Ambrosetti speakers were describing the future, through which the Labour leadership has the responsibility to guide one of Europe’s major political parties. The Ambrosetti crowd seeks to understand the future, because their businesses demand it. Labour’s future claim to good governance depends on it as heavily but there is too little evidence that the new party leadership is trying to understand – and to acknowledge the deals and reforms that have to be made to mitigate or benefit from the wave of change that is coming.

There are few in the UK whom Jeremy Corbyn could summon to advise him, even if he wished to – as Barack Obama has, with a range of Silicon Valley leaders and entrepreneurs. Yet there are several in or close to the Labour Party who both think and act on the same imperatives. Not surprisingly, perhaps, they are former aides or advisers to Tony Blair (and thus may be automatically disqualified from speaking to Corbyn). Their views surfaced in the speeches and interviews given by Liz Kendall in the leadership campaign and one could argue that they were decisively defeated because she was. Kendall, as brave as she was, had too little of a track record as a visible reformer with a body of coherent ideas to carry conviction, even among moderates.

Blairism, now identified in the eyes of its critics on the left as almost wholly discredited by Britain’s involvement in the invasion of Iraq, was, in essence, an attempt by Blair and Gordon Brown to mould social-democratic outcomes to the real economic inputs of the British economy – which, for much of Blair’s period in government, were led by the City of London.

For Matthew Taylor, head of Labour’s policy unit from 2003-2005 and now the chief executive of the Royal Society of Arts (RSA), there was an empty space in the Ed Miliband leadership where curiosity should have been and Corbyn shows no more desire to understand the challenges of the 21st century. Instead, in the past few months, formulaic denunciations (of bankers) and utopian declarations (of equality) from Labour have filled the halls and studios. Ian Leslie, the author of Curious: the Desire to Know and Why Your Future Depends on It, says: “One of the advantages of Blair was that he combined curiosity with optimism. Where many on the left say, ‘This is happening, it will be terrible,’ he said, ‘Look at all these new great things that are coming!’

For Geoff Mulgan, the head of Blair’s strategy unit between 2002 and 2004 and now the chief executive of the pro-innovation charity Nesta, the lack of curiosity was complemented by an absence of “granularity”: the necessary recognition that bricks build new Jerusalems and the electorate has been left short on details as to how these will be supplied, or how the space upon which to build will be found.

In May 2015, Mulgan wrote a letter to the incoming Conservative government advising it to reshape the administration around innovation – downgrading the Treasury and upgrading a department of business and growth. “It’s encouraging,” he once said, “that in the election, all three main parties committed themselves to experimentation. Unfortunately it was the election in Finland, not in Britain.”

Matthew Taylor believes that those currently at the top of the Labour Party seriously underestimate the diversity and the intellectual energy that went into the creation of New Labour, a work of more than a decade before the election of 1997. Policy development, team building, public relations and the job of opposing the Conservatives in the late 1980s and 1990s happened because of a stream of influence swollen by new thinking. One of the currents into this came, curiously, out of the death throes of the Communist Party of Great Britain, a last flaring of creativity and openness to new movements and ideas in a party that had not been noted for an interest in either.

In a 1989, Marxism Today published a series of essays under the heading “New Times”. The contributors – led by the journal’s then editor, Martin Jacques, and the cultural theorist Stuart Hall – argued that the central state should no longer be seen as the main engine for social change. Instead, politics should take explicit account of consumerism, individualism, ecological concerns and more fluid class and work structures, framing progressive politics around them.

Both Mulgan and the journalist-turned-social entrepreneur and freelance government adviser Charles Leadbeater were active in these debates and spoke up for the recognition of the power and usefulness of market mechanisms and consumer-oriented public services. Mulgan now says that “New Times” was devoted to “understanding the right, not merely dismissing it; connecting with the leading edge of social science; and inquiring how the world was changing to be able to shape it – very important in preparing the ground for future work”.

Blair, a steely pragmatist clothed in amiability, came to government committed not to rule from the old left. His New Labour thus sought a “third way”, between ideological (“old-fashioned”) leftism and Thatcherite neoliberalism. To theorise this, he called on the services of one of the world’s leading sociologists, the then director of the London School of Economics, Anthony Giddens. Giddens was and is a workaholic who wrote a book on Durkheim over one Christmas holiday (I should say that he is a personal friend). In his 1998 work The Third Way, Giddens viewed “old-style” social democracy as having lost both its working-class base and its belief that it could make a fundamental change through gradualist means.

He dismissed conservative neoliberalism (at least in the Thatcherite version) as in thrall to “unfettered market forces”, anti-egalitarian, inimical to the welfare state and imprisoned within potentially warlike nation states. The world – globalised, individualist, its industrial working class shrunken, with “post-materialist” values, especially ecological ones, increasingly prominent – needed a social democracy that transcended its old nostrums and practices. It could no longer pretend to remake the world but could at least influence and attempt to socialise its remaking by larger forces than itself.

The Third Way made Giddens a political celebrity for some years. He remains closely attentive to politics and believes that “an era of market fundamentalism may be coming to an end”. An urgent necessity, he says, is to “do something about the huge inequalities”. If he does enter the arena again, however, he would do so in much less propitious circumstances, with a Labour leadership that, unlike New Labour’s, does not appear to share his bias towards new thinking for new circumstances.

Like the other innovators, he is both admiring and critical of Blair. In the latter mode, he believes that Blair should have been more proactive in making the case for closer union with the EU and that he was too accommodating to capital.

The Third Way debates and conferences – a strong feature of the first New Labour government (1997-2001), when it could count on the strong support of President Bill Clinton – fell away in a second administration (2001-2005), dominated by the effects of 9/11, and had only an anaemic revival in the last (2005-2007), truncated Blair government, when he was harried by a chancellor within sight of his ambition’s goal. Giddens drew on his earlier academic work to show globalisation as involving “the retreat of custom and tradition from our lives” – a “moral transition”, as well as an economic one, much more than “just the influence of markets”.

Though he argued that the old social-democratic agenda relied on institutions no longer present, he always stressed the need to reduce income inequality, sharing nothing of Blair’s insouciance on the issue (summed up by the prime minister telling Jeremy Paxman, “It’s not a burning ambition for me to make sure that David Beckham earns less money”).

All of the former policy advocates, not just Taylor, emphasise curiosity as essential. Curiosity about what technology, globalisation and the new communications are bringing to the world; and curiosity about how these movements may be directed – or, at times, subverted – to create “a larger life for the ordinary man or woman” to quote the Brazilian philosopher and government minister Roberto Unger, a guru for the former aides. Or, in words of the old Blairite slogan, for the benefit of the “many not the few”.

Leadbeater, free of executive responsibilities and a compulsive producer of new ideas, laments that this necessary curiosity passed, in the Miliband era, from Labour to the Conservatives – as radically minded advisers such as Steve Hilton, Danny Kruger and Rohan Silva took up positions as economic or technology advisers to David Cameron. All have now left to pursue their own initiatives, usually expressing the disappointment common to innovators enfolded into government (Hilton has been harshly critical of the turn towards an authoritarian China, for example).

In a piece for the Observer in the last weeks of the leadership campaign, Leadbeater described Labour as marooned in a “cage . . . of its own making”, while the Tories, especially George Osborne, its probable next leader, were happily heterodox, intent on “borrowing from across the political spectrum”. His article rapped out a series of ideas for Labour – a “massive expansion in creative, vocational learning”; an eco-wealth fund, investing in ecological solutions; more shared solutions such as Airbnb, using unused resources and giving space for living and working at low cost; a universal basic income; street parties to encourage meeting, play and sociability.

Matthew Taylor has turned the RSA – with its roots in the Enlightenment’s “curiosity” about the possibilities of practical experimentation, as well as theoretical advances – into a shill for people’s power, a quest that runs through the work of his staff. Anthony Painter, the RSA’s director of policy, wrote in Prospect magazine that devolution should not stop at city halls or regions – but at people. He cited popular resistance to the closure of Brixton Market in London and its recasting as “a thriving hub of community regeneration”; or Liverpool’s Toxteth, where the community’s determination to salvage the semi-derelict area led to the creation of a community land trust; or the Alliance Scotland initiative, in which people with debilitating conditions are supported, provided that they design and implement new projects to aid and engage them.

Taylor, imposing and restless, insists, “You don’t put policy first. You think about what people do.” He adds: “And you listen to how people think about things. So inequality isn’t so much a worry as injustice. People don’t mind so much about unequal incomes, they do about unequal ownership of capital. Equality isn’t an end – it’s a means to an end, of self realisation and self-expression.”

Mulgan may qualify as the most curious (in the sense of having more curiosity) of all the former Blair aides. Nesta has helped to create of a network of “studio schools” that conform to Leadbeater’s idea for creative vocational learning, bringing the students into the working world early. He has also investigated “democratic disappointment” – the defection of the young from the voting booths, the perceived hopelessness of politicians – looking at the citizens’ consultation strategies of Spain’s Podemos, as well as parties such as Italy’s Five Star Movement and DemocracyOS in Argentina. Nesta works with Podemos and parties in Finland and Iceland to develop new ways in which parliamentary systems can combine with use of the internet.

As with the others, Mulgan sees little in the new Labour leadership to excite optimism. “If you are to campaign against inequality, you must have a coherent account of how it is to be reduced,” he says. “Such declarations, in policy development, quickly get granular and that’s where the real problems lie. There is a genuine debate on austerity and it’s not clear who is right. What is clear is that raising the productivity in the public sector is central to any government.”

Giddens, mulling over the present possibilities, believes that Labour has no choice but to put itself in the vanguard of technological and global change. For him and the others, the world is not infinitely malleable by political will: voluntarism, any more than populism, is not an option.

These views are outside the narrow church that the Labour leadership appears to wish to create, even though Giddens, Leadbeater, Mulgan and Taylor address ways in which society and employment can improve and give the working majority more power over their lives – an aim that should be common to all. It is impossible to believe, however, that their ideas will remain subterranean as the big state, pro-nationalisation, forward-to-the-1970s policies of what is the new Labour unroll – and unwind.

John Lloyd is a contributing editor of the Financial Times and a co-founder of the Reuters Institute for the Study of Journalism, where he is Senior Research Fellow.