Danyl McLauchlan writes an epic and extraordinary essay drawing threads from the past and present of political history and discourse and various books that, he says, ‘might be peripherally relevant to the 2017 General Election but to be honest probably won’t be’.



This story first ran in the leadup to the 2017 election

‘Because you are like lukewarm water, neither hot nor cold, I will spit you out of my mouth.’ – Revelations 3:16

Bill English is a political conservative. He gets a certain amount of respect for that. ‘At least he has values and stands for something,’ the sentiment goes, unlike certain popular-at-the-time but already weirdly forgotten former prime ministers.

But what is political conservatism? I want to try and explain this in somewhat good faith and how it relates to neoliberalism – which I will also attempt to explain, also in somewhat good faith – and why I think conservatism is a useful approach to politics in theory, but became a perversely radical doctrine when it merged with the also useful-in-theory neoliberal belief in free markets, and, also too, about how the ‘social investment’ public-service model Bill English promotes might even lead to the death of conservatism and a new ideological contest of ideas. But to make sense of all that I have to talk about baseball statistics, the French Revolution and scientific agriculture in 18th century Prussia.

Let’s start with the baseball. I never know whether to recommend Michael Lewis’s book Moneyball or not. On one hand it makes a very simple, very powerful point; on the other it is mostly about statistics and baseball, and unless you’re into those subjects, a lot of it is pretty boring.

Very briefly: professional baseball in the US is a multi-billion dollar industry. A lot of the money goes towards recruiting star players, and there’s a whole industry of veteran scouts that tour the country looking for up-and-coming players, judging their potential worth based on the scout’s extensive experience of the game and competing to sign the most promising players by offering them huge sums of money to go play for them.

Moneyball tells the story of Billy Beane, head coach for the Oakland Athletics: a very poor team – they only had $44 million to spend on salaries, unlike say, the New York Yankees who had $125 million. Beane decided that the scouting system and all the value criteria that scouts and teams across the entire professional industry had used for the last 150 years was just wrong.

Turned out there was this huge body of statistical baseball data, pored over by amateurs but mostly ignored by the industry until Beane and his advisers used it to determine the true indicators of what made a successful player and which metrics lead to successful games. The scouts rated players on how fast they ran, or calculated simple metrics like batting average. Beane and the other quants looked at much more sophisticated indicators: things like ‘on base percentage’, or ‘slugging percentage’, which research indicated were far more significant.

These metrics were undervalued in the market – the scouts and executives at other teams didn’t know or care about statistical analysis, and they thought Beane was an upstart idiot – so Beane was able to hire an incredible team for a very low price, and they went on to beat rival teams with double the Atheltics’ salary budget.

I’m not even slightly interested in baseball but I still love that story, because Beane got to live out a rational progressive fantasy: he found a situation in which the humans were doing something stupid and had been for decades, and he challenged the status quo with reason and facts, and won.

Transforming professional baseball is not a very meaningful form of activism, but it’s a proof of concept. It’s a metaphor for what most progressive activists want to achieve: look for problems – like a housing bubble leading to children sleeping in cars; or the planet we live on warming up because we’re changing the composition of its atmosphere – and then find rational solutions to them.

Seen from this perspective, political conservatism seems either baffling, or stupid or evil. There are obviously problems in the world. Why wouldn’t anyone want to fix them? Change is clearly possible, we’re surrounded by the proof of it. So why would anyone acting in good faith argue against progressive change? Political conservatives have to be like the old fashioned scouts and executives in Moneyball, right? They profit from broken, obsolete systems so they fight to preserve them. What other justification could there be for conservatism?

Seeing like a State: How certain schemes to improve the human condition have failed by James C Scott, a professor of political science at Yale, takes a shot at answering that question. Scott opens his book in Germany in the 18th century, when the governments of Prussia and Saxony decided that the huge forests in their kingdoms were horribly inefficient from an economic and bureaucratic point of view.

The primary purpose of a forest – as the state and its administrators saw it – was as a source of firewood and lumber. How much usable wood was in each area of woodland? They surveyed the woods and drew maps of them, but even with the maps it was still hard to say because the forests were untidy, chaotic places: hundreds of different types of trees along with shrubs and bushes and underbrush and so on, all planted at different ages in uneven distributions and filled with all sorts of animals and insects and other pests. You can’t plan anything with forests like that. So they cleared them and planted regimented rows of Norwegian spruce, a rapid growing, very valuable hardwood.

This worked brilliantly for the state because now it knew exactly how much wood it had in its forests and what the yearly yield and revenues from that wood would be. It worked really badly for the people living in villages who relied on the old, untidy forests for hunting and gathering, fishing, pasturage, charcoal making and trapping, and whose local economies promptly collapsed. And then it worked terribly for the state because after some decades of phenomenal yields from the Norwegian spruce trees the soil became exhausted and a new term – Walsterben, meaning ‘forest death’ – entered the German language as the challenged ecosystems collapsed and the forests died out en masse.

Scott calls this story a parable of what he terms ‘high modernism’ and its failures. What is ‘high modernism’? His definition is: ‘A strong, one might even say muscle-bound, version of the self-confidence about scientific and technical progress, the expansion of production, the growing satisfaction of human needs, the mastery of nature (including human nature), and above all, the rational design of social order commensurate with the scientific understanding of natural laws.’

In other words, it’s what people like Billy Beane in Moneyball do: look at existing systems and try and improve them using rational principles. Beane succeeded but Scott’s book looks at failure: he examines a number of schemes to improve the human condition by applying rational principles to agriculture, the design of cities, taxation systems, property rights, planned economies, the standardisation of languages and colonial occupations, citing Communist China, Le Corbusier’s urban planning, Lenin’s farm reform in Russia, village resettlement in Tanzania. All of these schemes were disasters, at least from the point of view of the people whose conditions were supposedly being improved by the wisdom of the high modernists and the power of the state.

What went wrong with all these noble progressive plans? Scott puts it down to lack of metis, a concept from Greek thought roughly analogous to tikanga. Before the state can act in the world it needs to make the world legible to it, which almost always involves simplification. It draws a simplified map of its territory then uses its power to make the territory look like that simplified map. So forests become rows of regimented trees, cities become geometric grids, economies become centrally directed plans. What this simplified, centralised, rational way of doing things misses out is metis. ‘The way things are done around here because that’s the way we do things around here.’ Traditions like ‘don’t cut down the forests and replant them because it’ll anger the Forest God,’ which might sound absurd from a rational perspective but turn out to make a lot of sense when the rational ways of doing things lead to terrible unforeseen consequences that the local, traditional solutions avoided. Like the forests dying out.

This suspicion of experts and their grand plans to change everything for the better is the intellectual basis for political conservatism. Progressives might think they know what they’re doing, the conservatives say, and they might think they’re making things better for everyone, but a lot of the time they know an awful lot less than they think they do and what they’re really doing is making everything worse for everyone.

Political conservatives trace the origin of their movement back to Edmund Burke, a liberal MP in the British House of Commons during the period of the French Revolution. Most 18th century liberals were very excited about the revolution: the people were rising up and getting rid of the corrupt, evil monarchy and the corrupt, evil, demonstrably-wrong-about-everything church and replacing them with a society based on enlightenment rationalism with equality and liberty for all. What wasn’t to love?

What’s not to love, Burke told them in Reflections on the Revolution in France, published in 1790, was that the revolution’s commitment to abstract intellectual ideas – like rationalism, or the rights to freedom and equality that guys like Voltaire and Rousseau advocated – ignored the complexities of human nature, human society and the traditions that sustained them. It ignored metis, in other words. The ancien régime might look irrational, he argued, but its structures and traditions reflected centuries of accumulated wisdom, a depth of knowledge that surpassed the superficialities of rationalism.

Moreover, Burke warned, the ability to radically remake society using the might of the state was too much power for individual humans to wield. ‘When men play God, presently they behave like devils.’ It would all end badly, he prophesied, ‘plots and assassinations will be anticipated by preventive murder and preventive confiscation,’ and the nation would descend into war, bankruptcy and bloody anarchy until ‘some popular general’, seized power and turned the Republic into a military dictatorship.

All of this made Burke incredibly unpopular with his fellow liberals. How could anyone who claimed to believe in individual freedom stand up for monarchical despotism and religious oppression over the freedom and equality that the citizens of France now enjoyed? Burke’s pamphlet was widely mocked and attacked by such luminaries as Mary Wollstonecraft and Thomas Paine.

But then the September Massacres happened, followed by the Reign of Terror, the Committee of Public Safety, Civil War, the Revolutionary Wars, the Thermidorian reaction, followed by Napoleon’s coup d’état and military dictatorship, and some people agreed that Burke might have been onto something. The modern conservative movement – especially in the United States – was a reaction to the revolutions of the early and mid 20th Century, in which rational progressives overthrew governments all over the world – Russia, China, Cambodia, etc – and tried to build new societies free from religion and oppression, based on reason and enlightenment and it was literally the worst thing that ever happened in all of human history. They felt that Burke was definitely onto something.

Was he though? Really? Did the societies of pre-Revolutionary France or Tsarist Russia actually represent ‘accumulated wisdom’, like the ecosystems of those forests in early modern Prussia? Or were they just evil, corrupt, oppressive regimes that collapsed when they could no longer sustain themselves, and which Burke et al were mere apologists for? When modern conservatives argue against gay marriage are they really worried about the limits of enlightenment rationalism and the millenia of wisdom allegedly encoded in the traditional family structure, or are they just being cruel and intolerant bigots and dressing their prejudice up in pseudo-philosophical rhetoric?

I think a lot of the time it’s the latter, but I also think the motive matters less than the fact that conservatives are occasionally demonstrably correct: progressives do sometimes want to change society in ways that they haven’t thought through, and this does sometimes lead to catastrophic outcomes. The conservative critique of progressives – that they don’t really want to make the world a better place: they just want power so they can play God with everyone’s lives – is mostly wrong, but is right often enough to justify conservatism as a valid and even vital critique.

The problem with contemporary conservatives – from my perspective – is that they aren’t conservative enough. Contra Burke, they’re happy to advocate major, radical changes to our society so long as this change is market-driven rather than centrally planned. Gay marriage is bad because it challenges the traditional concept of the family which encodes centuries of wisdom, yadda yadda yadda, but a speculative bubble in the housing market that has families living in garages is all okay because it’s the wisdom of the market and the state shouldn’t interfere with it. Conservatives should, in theory, be opposed to the destruction of ecosystems; the atmosphere of the planet should be something conservatives are reluctant to mess with. But modern conservatives don’t see it that way. To understand why we need to talk about neoliberalism.

Most people on the left encounter neoliberalism in books or articles by people like Noam Chomsky, Naomi Klein or George Monbiot. In this context neoliberalism is not so much an ideology as it is a malevolent global criminal conspiracy that exists to undermine democracy, enrich the wealthy and destroy the planet. It’s the reason behind everything that is wrong with the world. It’s based on obvious logical fallacies: that humans are always rational, that greed is good, that there’s no such thing as society, that the invisible hand of the market is always correct, that the wealth will always trickle down. Things that only monsters or idiots could believe.

This is a great way to get people angry and to sell a lot of books but it doesn’t actually explain the ideas advanced by people like Milton Friedman or Friedrich Hayek, which are more substantive than the usual left-wing critiques allow. So I want to start with a more generous yet indirect explanation of neoliberalism, and a hopefully more constructive critique.

One of my favourite books is Francis Spufford’s Red Plenty, a hard-to-classify non-fiction novel with fictional components and almost a hundred pages of footnotes citing historical sources. It’s set in the Soviet Union in the 1950s and 60s, and it asks a fascinating question: why did Soviet Communism fail?

There are plenty of glib answers to this, ranging across the political spectrum: Because it was Godless. Because the revolution failed to follow the principles of historical materialism. Because Communism lacked incentives. Because it failed to understand human nature. Because it wasn’t true communism, but rather ‘state capitalism’.

Spufford starts from the premise that Soviet communism in the late ‘50s and ‘60s was definitely communism, and definitely not failing. Stalin and Beria were gone, and the KGB had stopped killing people at random to meet Beria’s monthly execution quotas; workers were being housed and fed instead of marched at gunpoint into the middle of Siberia en masse to build mines or canals until they starved or froze to death. Economic growth was phenomenal, and so was technological progress. In 1957 the USSR launched Sputnik, the first artificial satellite, into low-Earth orbit, scaring the crap out of all the decadent capitalist countries. Living standards soared. In just seven years life expectancy increased by 25 years! Communism was killing it, but in a good way this time.

Which made sense. It’s hard to live in a capitalist society without contemplating just how insanely inefficient and unfair and wasteful it all is. Coke spent an estimated $4,000,000,000 last year persuading people to buy their arguably useless product instead of other nearly identical useless products. Wouldn’t it be better for everyone if we lived in a fairer society without all that competition and overconsumption; a society in which everyone had what they needed to get by instead of some having so much and so many others having so little? Wouldn’t it be more rational for the profits of everyone’s labour to be reinvested back into the economy to make everyone better off instead of being poured into speculative bubbles or frittered away on palaces and private jets for the wealthy few? Red Plenty starts out in a country in which the answer appeared to be ‘Yes!’ but turned out to be ‘No!’ and tries to figure out what went wrong.

Spufford tells a story about Solkemfib, a remote chemical plant on the shore of the White Sea (a composite, Spufford explains, of several factories) which manufactures tire cord – a vital commodity in the industrialising Soviet economy. They don’t have to make a profit because there’s no market economy to profit from, and thus no competitors, because everyone is working together, because the Soviet economy is run on rational scientific principles not the chaos of the free market. Instead the Central Economic Planning committee (Gosplan) tells them how much tire cord to make because that’s how many tires they’ve predicted the Soviet economy will need.

Unfortunately Gosplan predicts that the Soviet economy needs 14,000 tons of tire cord over the next year and the plant’s machinery just can’t produce that much. No one at the factory wants to tell Gosplan this because Gosplan are basically what Treasury would be like if they gave public servants holiday homes on the Coromandel if they met their forecasts but relocated them to the Chatham Islands if they failed.

The good news is that another factory that produces the plant’s machinery has a new model available, and this new, improved machine can easily meet the Gosplan quota. But the tire cord factory can’t just buy a new machine – they have to be assigned one by Gosplan. So the plant’s executives drive a massive earthmover to the top of the hill outside the factory and take the handbrake off: the earthmover speeds down the hillside, through the brick wall of the factory and into the viscose machine, spilling sulphuric acid everywhere, destroying the factory and the outdated machine.

Gosplan is forced to give the Solkemfib factory a new viscose machine, and we’re treated to a brief vignette showing us how this single reallocation will ripple through the Soviet economy, disrupting plans and supply chains across the entire system. But at least the boys at Solkemfib will make their quota.

Or not, because it turns out, Gosplan has also told the factory that produces the viscose machine that their quota for the year is based on the weight of machinery they produce – which is the easiest metric for Gosplan to track. Because their new, better model is lighter than the old one they’ve switched back to manufacturing the older, less efficient machine – because they don’t have any competitors, or need to make a profit – which they ship over to Solkemfib, replacing the identical machine the plant’s executives deliberately destroyed. Solkemfib’s executives wrecked their own factory and disrupted their country’s economy for no gain whatsoever.

Towards the end of the book we get another insight into the workings of a planned economy: a famous case of a pig farmer put on trial for buying a load of timber from the factory next door to his farm. The farmer hadn’t been allocated any wood, but he needed it to build stys for his pigs so they didn’t freeze to death in the winter. The factory next door was going to burn their surplus wood, but instead the farmer traded goods with the factory for it, got caught doing so, was put on trial and sent to prison pour encourager les autres, because if everyone just traded for whatever they needed the entire planned economy would collapse.

These stories should, hopefully, sound familiar. They’re the same kind of problems James Scott wrote about in Seeing like a State. The individuals in the stories have metis: a depth of local, practical knowledge, but the state, in order to make its rational, planned decisions needs to simplify the map of the territory it governs and then enforce its simplified map onto the messy and complex territory of reality, with disastrous consequences.

All of the problems the planned, rational economy routinely encounters are avoided with the de-centralised, distributed, metis-powered decision-making of a free market economy. Dumb, awful stuff still happens under capitalism, but failure is localised: it doesn’t become so endemic it leads to systemic collapse. This is the idea at the heart of neoliberal ideology: that millions of individuals making informed choices about their own circumstances will make far better decisions than central planners acting on limited information making those decisions on their behalf, while being removed from the consequences of the decisions. If you take away people’s ability to communicate and choose via the market, neoliberals say, you take away much of their freedom.

One of core tenets of Marxist theory is that markets are intrinsically evil because they reduce humans into commodities to be traded, and transform abstract non-human things like prices or companies into quasi-real beings that enslave us. Neoliberalism argues that markets are inherently liberating: they’re a means for humans to communicate with each other about our needs and values: they free us to make decisions about what those needs and values are and what we’ll exchange to meet them. Outlaw the market and you’re left with a powerful state dictating to powerless individuals what they should value, and what they need, and what they have to do to meet those needs. It’s the state that commodifies people, the neoliberals, say, and the market that liberates them.

Not only do markets free us from the coercive power of the state, they also free us from poverty. Because markets are such a powerful mechanism for communicating information about scarcity and value they allow free market economies to become vastly more complex, and therefore more prosperous than non-market economies. Remember, the Soviet economy was doing fine while it was industrialising: mining coal and manufacturing steel and building stuff out of concrete, because that’s fairly simple stuff. Once it reached the prosperity and complexity of late-industrialism, with chemical plants, consumer-goods, plastics, micro-electronics etc, the scale and intricacy of the system – the sheer volume of information – overwhelmed the state’s ability to co-ordinate and plan it all.

By the end, Spufford wrote, the Soviet economy ‘produced goods that less and less corresponded to human needs, and whatever it once started producing, it tended to go on producing ad infinitum. The control system for industry grew more and more erratic, the information flowing back to the planners grew more and more corrupt. And the activity of industry, all that human time and machine time it used up, added less and less value to the raw materials it sucked in. Maybe no value. Maybe less than none. One economist has argued that, by the end, it was actively destroying value; it had become a system for spoiling perfectly good materials by turning them into objects no one wanted.’

By the ‘80s neoliberalism seemed like the right ideology at the right time. The Communist states were totalitarian basket-cases, while developing economies which abandoned central planning and adopted liberal democracy and market solutions were experiencing phenomenal growth and enjoying the huge increases in living standards correlated with prosperity and political freedom. Meanwhile, established western economies with extensive planning and state intervention – like the UK and New Zealand – were on the brink of insolvency, borrowing money from the IMF and World Bank to keep their faltering economies propped up (similar stories to Red Plenty’s Solkemfib exist about the Muldoon government’s ‘Think Big’ industrial projects in the ‘70s and early ‘80s). Neoliberalism seemed to explain everything that was happening in the world, and it claimed to have answers to the low growth and high inflation crippling the developed countries: lower taxes, free trade, deregulate business; disempower the state and empower the market.

All this was deeply attractive to many conservatives. Conservatives tend to be religious, because they feel ancient faiths represent millennia of accumulated wisdom; the Communist states were militantly – often murderously – atheistic, because rationalism. And here was this persuasive philosophical alternative to the planned, rational, atheistic society, that also indicated a very strong policy direction. Political conservatives became devout believers in the transformational powers of the free market. The contingencies of history lead them to the point where they convinced themselves that the cautious, sensible thing to do was unleash the market and embrace change.

And we all know how that went. There were a number of flaws in the neoliberal model and its predictions: (a) the links between free markets and democracy weren’t nearly as strong as they hoped: China liberalised markets without liberalising politics; instead of becoming ‘free’, Russia became the world’s first postmodernist dictatorship; also (b) it turns out having billions of people making supposedly rational decisions at the individual level generates non-rational outcomes with existential consequences at the global level, and many less apocalyptic but still terrible outcomes at the national and social level, (d) the minimalist governments the neoliberals prophesied never arrived. Instead, as capitalist states grew wealthier the size of central government expanded. Contrary to theory there were lots of things centrally planned government could deliver better than free markets, like build and maintain the increasingly sophisticated social and economic infrastructure the markets needed to function properly, (e) failure in the financial sector of free market economies is not localised and the costs of that failure are socialised, leading to outcomes like the 2008 financial crisis; finally (f) humans are a lot less rational than the neoliberals – or anyone – thought.

Often when someone tells you that ‘people aren’t rational’ what they’re really saying is that they, an educated smart person, is rational, while everyone else – especially those with different beliefs or values – is ignorant and irrational. But this is a delightfully irrational way to think about human decision-making.

In 2002 the Nobel Prize for economics was awarded to Daniel Kahneman, an Israeli-American psychologist who had never formally studied economics. What Kahneman and the psychologists and behavioral economists who built on his work found is that humans rarely resort to the kind of probabilistic, objective, rational decision-making that neoliberal economists based a lot of their assumptions about human behaviour on. In most situations – and for most of our species’ evolutionary history – that was an inferior way to make choices, no matter how smart you were. If you were foraging for berries and thought you saw a snake behind the leaves, you really wanted to go with your gut instinct on how to react rather than conduct a rational Bayesian analysis of the situation.

We aren’t living in the paleolithic era but we still make most of our decisions intuitively, and that’s not a bad thing, most of the time. If someone offers you a job paying 1% more than your current salary but they seem like a deeply untrustworthy and dishonest person is it ‘smart’ to ignore your feelings, act like a rational economic maximiser, quit your old job and take the new one? Probably not. Our instincts are pretty good at making small-scale personal judgements. The problem is, the further we transition into large, complex, technologically sophisticated societies, ie the kind that capitalism makes possible, the less reliable our intuitive, evolved decision-making capabilities become, and the more dangerous the flaws in those capabilities: Kahneman refers to these flaws as cognitive bias.

Not only are the smart educated people not immune to cognitive bias, they’re highly prone to what Kahneman refers to as ‘the most pervasive and potentially catastrophic’ form of bias: overconfidence. We all want to be taught and led by confident, optimistic people who seem like they know what they’re doing. But such people routinely, massively overestimate their own abilities, their understanding of the world, their capacity to change it and their chances of success or risk of failure in any endeavour. Overconfidence, Kahneman argues, is why most wars and revolutions start, why most lawsuits happen, and why most businesses fail. James Scott thought that hubristic over-reach was a feature of high-modernism; the neoliberals thought it was a feature of central-planning. Kahneman’s book is called Thinking Fast and Slow – it’s weirdly unreferenced in political circles; I think people think it’s a self-help book – in it he argues that failure through overconfidence is ubiquitous across our species.

The flawed nature of human decision-making is good news for conservatives who always suspected that the educated smart people knew a lot less than they thought they did, but bad news for neoliberals who thought that people interacted with markets in mostly rational ways leading to efficient utilitarian outcomes. The empirical and intellectual failures of the neoliberal project have seen political parties backing away from it, slowly at first and then rapidly ever since the 2008 Global Financial Crisis, but without rolling back its core reforms.

Instead most western democracies spent the past few decades trying to find a happy medium between the market and the state via the Clinton/Blairite ‘third way’ in which the prosperity from free markets funds large public sectors and welfare systems. Some intellectuals on the left still advocated for a return to the rational planned economy without solving, or even seeming to acknowledge its flaws. Neoliberals and their champions in the business community continued to advocate markets and individual choice: they never claimed markets were perfect, they say, but since the advents of their reforms markets have lifted more than one billion people out of absolute poverty: free markets and globalisation have been the most powerful emancipatory tools in all of human history.

But most successful political parties in the developed world embraced the third way. It felt like a sensible, moderate solution. It wasn’t the utopia progressives dreamed of, but neither was it the dystopian nightmare they tend to plunge everyone into if given free rein. Politics in the late 20th and early 21st century became about litigating the balance between the state and the market. Getting the balance just right.

And now that seems to be going wrong.

There are lots of takes out there on where we’re at this moment in history. ‘Why Brexit?’ ‘Wither Trump?’ ‘What (does) Corbyn (mean)?’ Maybe one of them is correct. Maybe we’re too close to our own history to see it properly. Maybe the current system is so complex and ideologically incoherent it compels intellectuals and political activists to make simplified maps, transforming it into whatever their ideology teaches them to oppose: neoliberalism if they’re left-wing or a vast socialist welfare state if they’re a neoliberal; or, maybe there is no trend, or deep cause?

My not-very-original theory is that when you have a strong market and a strong state you inevitably end up with collusion between sympathetic factions of the two. The most effective way for businesses and the wealthy to advance or protect their interests is to buy political influence and use the power of the state to interfere in the market: capture their regulators, externalise their risks and losses, privatise the profits. Socialism for the rich, capitalism for everyone else. None of it is ever illegal because political leaders get to write the laws that regulate all of this behaviour, but people know the system is corrupt, that business leaders and politicians are crooks, that the game is rigged: the outcomes are inherently unfair. They get angry and look for alternatives.

What are the alternatives? How do you deliver more just social and economic outcomes in a mixed post-industrial economy? How do you reject neoliberalism if you don’t have solutions to the problems neoliberalism addressed? If you just go back to the old days of politicians setting prices and wages there’s every likelihood you’ll recreate the same old problems of high inflation and low growth, crashing the economy and justifying another round of neoliberal reforms. How do we move forwards?

There are two ideas I want to talk about: both partial solutions, both originate from the intellectual right; one has been embraced by the left – with modification – the other thus far rejected: the Universal or Unconditional Basic Income (UBI) and social investment. Let’s do the UBI first.

Look at it like this. You don’t have a job. You’re on welfare. You’re getting paid, let’s say $200 a week. Someone offers you a job working 40 hours a week. It pays $200.

The state would love for you to take that job, partly so it can save some money by abating your welfare payments but mostly so you’ll start working and generating economic value, and gain experience and knowledge and skills and thus generate even more value, earning more, paying more in taxes and so on. However, you have little rational incentive to take the job because you end up working 40 hours for the same amount of money you get paid anyway. The rational option – in the short term, at least – is to remain on welfare, gaining no skills and experience and generating no value. This is where the right-wing argument that ‘welfare is a poverty trap’ comes from.

But what if everyone got paid $200 a week and it didn’t get clawed back if you started working? Now you have an incentive to take the job because you double your income. And – here’s where this is attractive to neoliberals – it also lets the government abolish the entire apparatus of the welfare state. No more bureaucracy or anti-poverty programmes or means-testing or job schemes. Everyone gets just enough to get by and the market takes care of the rest. That’s the right-wing version of the UBI.

There are various progressive versions of the UBI, most of which are similar only the payments are much larger – usually at the level of our current superannuation scheme – and there’s still a large public sector and welfare state. These versions of the UBI are very expensive. Like, its hard to deliver a comprehensive UBI that doesn’t cost more than the current welfare system, education system and health system combined, levels of expensive. (One left-wing economist has explained to me that I’m wrong here: a comprehensive UBI is just a system where everyone gets a lot more money but it doesn’t cost anything, and I’m an idiot if I don’t understand this. But most concede that a comprehensive UBI is eye-wateringly expensive).

It’s a policy idea that seems so far outside the current parameters of what’s politically possible it wouldn’t even be worth talking about, but its advocates insist we have to talk about it because the comprehensive UBI is the solution to the looming problem of the mass-automation of jobs and the jobless future.

Self-driving cars and other forms of machine intelligence are going to change everything, this argument goes. Just as machines have automated agriculture and manufacturing, the robots and algorithms are now coming for service sector and white-collar jobs. You can’t rely on a minimal UBI to supplement income from the labour market because in the near future there won’t be a labour market. Instead the wealth generated by automated mass-production will fund the comprehensive UBI, which is what we’ll all live on while the machines do all the work. Everyone will fish in the morning, read poetry in the afternoon and criticise at night, just like Marx promised.

If that’s how it plays out, great. But I’m skeptical of the end-of work prediction. Is the self-driving car going to be more transformational than… the car? The machines have been coming for our jobs for 250 years and almost everyone still has a job: people are software developers or yoga teachers instead of blacksmiths or ploughmen. I understand the arguments about why this time its different, that we’re at the bottom of the S-curve of AI, expert systems and widespread human economic obsolescence. My problem is these predictions sound awful similar to the forecasts people like Keynes made in 1930 about the fully-automated 15 hour work-week utopia we’d all be living in today.

Industrial revolutions take longer than you think. Maybe those things happen in the future, sometime, and justify sweeping policy changes like a comprehensive UBI, and maybe they don’t. Right now we’re living through a period of average – currently declining – unemployment and very low productivity growth. The opposite of what we’d expect to see if super-efficient robots and AI systems were taking everyone’s jobs.

I think there’s a good progressive case for a minimal UBI. Because the scope of the welfare state is a political decision we’re gradually seeing the system drift upwards to support more and more middle-income earners, who happen to also be swing voters. The people who really need the system but are far less likely to vote are falling through the no-longer very effective net at the bottom and there’s not much political incentive to catch them. A single universal payment solves that problem, so long as you still have a proper welfare system bolted onto it.

But when it comes to a comprehensive UBI funded through futuristic technology scenario, I’m more interested in what we can do with actually existing technology. I’m more interested in social investment.

Let’s jump right back to the start of this essay and recall Billy Beane, the guy in Moneyball who used statistical analysis to beat the professional baseball industry. Billy Beane wasn’t a guy with metis. The people with local, informed decision-making knowledge in that story were the scouts who Beane outperformed.

What Beane had on his side was something humans had very limited access to until the late 20th century: data and analytics. A huge volume of aggregated data that no amount of local knowledge could match, analysed with algorithms which enabled him to make more informed decisions than his local-knowledge-based competitors. Businesses – most notably tech companies and the finance sector – have been using big data and big analytics for a while but governments – in the policy space at least – have been more cautious.

Currently the parameters of the welfare system are determined by politicians making guesses about what might be best for large groups of people who meet the criteria for eligibility; guesses that they are naturally very confident in making, but that aren’t immune from political calculations. Welfare recipients get paid and sometimes delivered services based on those criteria and guesses, and then they make market-based choices about how to spend the money. There are some restrictions and it’s not much money so their choices are incredibly limited. The outcomes measured are vague: total working-age unemployment; Not-in-Education-Employment or Training (NEET) rates; average benefit duration.

It would be weird if there wasn’t a better way to do things. National’s social investment approach proceeds on the basis that the state has a lot of data about welfare recipients and their individual outcomes, and they can analyse it to see patterns and indicators that aren’t apparent to political decision-makers or even the recipients themselves. Subgroups appear in the data: people whose demographic profiles indicate they need much higher levels of support than the state currently provides them with, delivered in a targeted form they might not choose even if they’re given the kind of substantial help a more generous welfare policy – a comprehensive UBI, say – would provide. Let’s look at some examples.

The welfare system is designed to help people who are on welfare. Get a job and you’re not really its problem any more. But what the analytics reveal is an ongoing trend among a sub-group of people who are on welfare: they get a job and then lose it again shortly afterwards and remain on welfare for a much longer period of time. 70% of people who register for a benefit on any given month have been on a benefit before. Most of them want to be in work: why can’t they stay there?

Illness is one reason; kids getting sick is another; school holidays yet another. The possible interventions are pretty obvious: help with navigating the healthcare system, subsidies for time off work, subsidies or assistance with child care. It’s counter-intuitive because being in work is supposed to mean you get less support, not more, but the investment approach means you can justify the higher upfront cost if it leads to the person staying in paid work instead of returning to the greater cost long-term cost of a long-term welfare recipient. And being in paid work also correlates to much better life outcomes than being on a long term benefit.

Take a second group in the data: the 1% of children aged five who are known to the Ministry of Vulnerable Children, have been supported by benefits for more than 40 months and have a parent who has had contact with the prison system. Statistically they’re likely to have very poor life outcomes: leaving school without a qualification, becoming a long-term beneficiary and have a 24% chance of having been in prison by the age of 35. On a year-by-year basis this is only about 600 kids; cumulatively this adds up to 12,000 over 20 years, more than the current population of the prison system, which currently costs about a billion dollars a year. Because their total cost to the state is probably going to be huge you can justify spending an awful lot on interventions to improve their outcomes. Send a registered nurse, a social worker and a tutor to all of their homes every week for a few years and you’re still going to come out billions of dollars ahead. And the actual children will get nicer lives too.

Those are all hypothetical interventions. I couldn’t get any officials involved in delivering these programmes to commit to describing definite solutions; this has less to do with bureaucratic cowardice and more to do with the empirical nature of the programme which is based on piloting a number of different interventions, monitoring them to see if they work, broadening them if they do; discontinuing them if they don’t then moving on to try something new. Again, this is different from the way a lot of policy has been formulated in the past, especially in the welfare space which has traditionally been based on the politicians-make-confident-guesses-and-never-back-down model. It remains to be seen how this class of policy will interact with our politics.

Social investment has not been well received on the left. Partly this is a tribal thing: the right are the bad guys, so their ideas are ipso facto bad. Also, social investment has the word ‘investment’ in it. People aren’t investments, the argument goes, and we shouldn’t treat them as though they are. Especially not children: we shouldn’t put a price on a child’s life. What we really need is to change our society into one in which all children are valued and supported because that’s the right thing to do, not because they’re going to incur costs on the state’s balance sheet.

I’m sympathetic to this argument. Not everything that counts can be counted; people are a means unto themselves, not the means unto an end. But it’s easy to be sanctimonious and talk about valuing children and changing society. It’s hard to actually change society by coming up with practical solutions to value and coordination problems that aren’t prices or something very similar.

One of the heroes of Red Plenty was a real life Soviet mathematician called Leonid Kantorovich, a Nobel prize-winning polymathic genius who invented linear programming as a solution to the resource-allocation optimisation problems of the Soviet economy. But it used something called ‘shadow-prices’ in contradiction of Marxist-Leninist doctrine so it was abandoned. They never found another way to solve the coordination problem and their economy collapsed. Prices are information. If you want to communicate information about value in an advanced economy you’re stuck using something that looks an awful lot like prices, even if what you’re valuing are people.

The bigger problem, I think, is that social investment returns us to the familiar dynamic of a powerful state deciding and dictating the needs of the individual: a relationship that already suffers from massive power imbalances and is made even more unequal by the amount of information the state will possess about us. Funding the state through taxation compromises our property rights, but most of us are happy to pay taxes to live in a civil society. Maybe operating more effective welfare states is going to mean compromising privacy and other rights on the same grounds? Politics and economics are about trade-offs: maybe we have to give the state more power to give others more help? Bearing in mind that this trade-off has gone horribly, horribly wrong in the past, and so needs to be made with great caution. Conservatism, even.

Some people in National gloat about the prospect of social investment rendering their adversaries on the traditional left obsolete. ‘If we can deliver a more effective welfare system for less money, why would anyone vote for them?’ A fair question, but here’s another one: what does it even mean to be a right-wing party or a conservative politician in a society in which the state can make better decisions about price or resource allocation than the individual or the market? What kind of economy do we end up with once the state no longer needs to simplify its model of the world, and its maps can be as complex as the territory?

Right-wing political parties will still be sympathetic to business and the social investment approach doesn’t need to be restricted to the welfare system. It’s applicable to most economic decisions and this will conflict with the interests and values of the right and their donor class in fundamental ways. It already is: treating housing as an untaxed speculative commodity is good for business but insane from a social investment point of view. So is the punitive approach to welfare which sees families dependent on benefits punished with huge cuts to their income if they fail to jump through the systems’ complex and often degrading hoops. What will the criminal justice system look like from the perspective of evidence-based policy and measuring outcomes? Nothing like the contemporary prison system, which is politically palatable yet incredibly expensive and ineffective at either deterring crime or rehabilitating prisoners. What about those sectors of our economy which are currently served by the market but seem vulnerable to price-gouging or other conspicuous forms of market failure? The supermarket and finance sectors, say? If technology leads to a rebalancing of the role of the state, where does the rebalance end?

I’m not going to pretend the public will vote on any of these issues during the election. Most people make good-old metis based decisions, voting for perceived economic stability or social identity affiliation or the leader they’d like to have a beer with. And that’s fine: it probably leads to better outcomes than decisions made by political obsessives like me with our confidently argued ideologies and simplified maps of reality. But the policy nerds and other thinkers inside the parties – whatever collection of them ends up running the country – will need to engage with these issues.

They take us outside the traditional territory of 20th century political debate, which is a good thing as we head towards the third decade of the 21st century. But it is going to disorientate some people who are deeply invested in relitigating the old debates and have entrenched views on the role of the state and the market. If new ideas like social investment are successful then many current politicians – both left and right, progressive and conservative – will be rendered obsolete, but the new contest of ideas will just be beginning.

This content is entirely funded by Simplicity, New Zealand’s only non-profit fund manager, dedicated to making Kiwis wealthier in retirement. Its fees are the lowest on the market and it is 100% online, ethically invested, and fully transparent. Simplicity also donates 15% of management revenue to charity. So far, Simplicity is saving its 7,500 members $2 million annually. Switching takes two minutes.

The views and opinions expressed above do not reflect those of Simplicity and should not be construed as an endorsement.