It has long been argued that workers will be replaced by machines, but now the threat is real. How will we bring about a revolution in both work and leisure?

Oscar Wilde dreamed of a world without work. In The Soul of Man Under Socialism (1891) he imagined a society liberated from drudgery by the machine: “while Humanity will be amusing itself, or enjoying cultivated leisure … or making beautiful things, or reading beautiful things, or simply contemplating the world with admiration and delight, machinery will be doing all the necessary and unpleasant work.” This aesthete’s Eden prompted one of his most famous observations: “Is this Utopian? A map of the world that does not include Utopia is not worth even glancing at.”

In Wilde’s day the future of work was the first question that every aspiring utopian, from Edward Bellamy to HG Wells, needed to answer. Everything else, from gender relations to crime prevention, flowed from that. But proponents of the more attainable goal of drastically shorter working hours have also included Benjamin Franklin, Bertrand Russell, AT&T president Walter Gifford and John Maynard Keynes. When the great economist coined the phrase “technological unemployment” (“unemployment due to our discovery of means of economising the use of labour outrunning the pace at which we can find new uses for labour”) in Economic Possibilities for Our Grandchildren (1931), he focused on the potential benefits a century hence.

Written as a sprightly provocation rather than a detailed prophecy, Keynes’s short essay has since become a set text for the diverse thinkers, such as Benjamin Kline Hunnicutt and Nick Srnicek, known as “post-workists”. The prospect of disappearing jobs has at least created a few new ones for economists and polemicists, some of whom make Keynes look stingy. In his future-drunk Fully Automated Luxury Communism, Aaron Bastani maps out a post-scarcity paradise in which, by way of solar panels, asteroid mining and 3D-printed bacon, “luxury will pervade everything as society based on waged work becomes as much a relic as the feudal peasant and medieval knight”.

Running parallel to this bullish tendency has always been the dystopian analysis, in which robots toss workers on to the scrapheap, dooming them to empty lives of poverty and despair. The New York Times ran the ominous headline “A Robot Is After Your Job” in 1980 but it could have done so at almost any point in the last century. The title of Daniel Susskind’s book, therefore, is either a threat or a promise, depending on your point of view.

Susskind, an economics scholar and a former government policy adviser, has come up with an explainer rather than a polemic, written in the relentlessly reasonable tone that dominates popular economics: the voice of a clever, sensible man telling you what’s what. He always has a helpful graph to hand and a greatest hits collection of anecdotes about technology and society, from Ned Ludd to Deep Blue via the 1890s horse manure crisis.

Susskind guides the reader through a boneyard of discredited assumptions about technological unemployment. Throughout the 19th and 20th centuries, automation tended to replace human labour in “routine” tasks without destroying entire occupations. Even when certain professions were eliminated, new ones were created. The somewhat technophobic utopian socialist William Morris saw this happening in 1890: “All their devices for cheapening labour simply resulted in increasing the burden of labour.”

Facebook Twitter Pinterest Is a robot really after your job? Photograph: Rodrigo Reyes Marin/Zuma Wire/Rex/Shutterstock

But AI, Susskind argues, has changed everything, starting with the definition of “routine”. Time and time again, it has been assumed that a task required a human being until a machine has come along and proved otherwise, and without needing to mimic human cognition. It used to be argued that workers who lost their low-skilled jobs should retrain for more challenging roles, but what happens when the robots, or drones, or driverless cars, come for those as well? Predictions vary but up to half of jobs are at least partially vulnerable to AI, from truck-driving, retail and warehouse work to medicine, law and accountancy.

That’s why the former US treasury secretary Larry Summers confessed in 2013 that he used to think “the Luddites were wrong, and the believers in technology and technological progress were right. I’m not so completely certain now.” That same year, the economist and Keynes biographer Robert Skidelsky wrote that fears of technological unemployment were not so much wrong as premature: “Sooner or later, we will run out of jobs.” Yet Skidelsky, like Keynes, saw this as an opportunity. If the doomsayers are to be finally proven right, then why not the utopians, too? Committed to neither camp, Susskind leaves it late in the day to ask fundamental questions. The work ethic, he says, is a modern religion that purports to be the only source of meaning and purpose. “What do you do for a living?” is for many people the first question they ask when meeting a stranger, and there is no entity more beloved of politicians than the “hard-working family”. Yet faced with precarious, unfulfilling jobs and stagnant wages, many are losing faith in the gospel of work. In a 2015 YouGov survey, 37% of UK workers said their jobs made no meaningful contribution. Susskind wonders in the final pages “whether the academics and commentators who write fearfully about a world with less work are just mistakenly projecting the personal enjoyment they take from their jobs on to the experience of everyone else”.

That deserves to be more than an afterthought. The challenge of a world without work isn’t just economic but political and psychological. To paraphrase the children’s author Richard Scarry: what will people do all day? You can only contemplate the world with admiration and delight for so long. Joblessness tends to create loneliness, lethargy and social dysfunction. But is relying on work to provide self-worth and social status an inevitable human truth or the relatively recent product of a puritan work ethic? Keynes regretted that the possibility of an “age of leisure and abundance” was freighted with dread: “For we have been trained too long to strive and not to enjoy.”

The state, Susskind concedes with ambivalence, will need to smooth the transition. Moving beyond the “Age of Labour” will require something like a universal basic income (he prefers a more selective conditional basic income), funded by taxes on capital to share the proceeds of technological prosperity. The available work will also need to be more evenly distributed. After decades of a 40-hour week, the recent Labour manifesto, influenced by Skidelsky, promised 32 hours by 2030. And that’s the relatively easy part. Moving society’s centre of gravity away from waged labour will require visionary “leisure policies” on every level, from urban planning to education, and a revolution in thinking. “We will be forced to consider what it really means to live a meaningful life,” Susskind writes, implying that this is above his pay grade.

The virtue of his reluctance to take a firm political position on an inevitably political issue is that it makes pragmatism and idealism seem to point in the same direction. While other writers make a strongly socialist, feminist or environmentalist case for a post-work world, he says simply that the jobs will go and we’ll have to make the best of it. In light of the current state of political leadership, his optimistic sign-off feels more dutiful than persuasive. Still, if AI really does to employment what previous technologies did not, radical change can’t be postponed indefinitely. It may well be utopia or bust.

• A World Without Work is published by Allen Lane (RRP £20). To buy a copy go to guardianbookshop.com. Free UK p&p over £15.