Finland is not planning to scrap its existing benefit system and give everyone an unconditional grant of €800 a month – contrary to what some recent headlines may have told you.

What it is planning promises to be an interesting policy experiment involving a sample of the population, which may or may not include some form of basic income paid to all participants: which in turn may not be unconditional, and may be worth a lot less than €800. Still the general excitement was testimony to widespread interest in the basic income idea.

That idea has been around for a very long time (at least since the aftermath of the first world war), attracting support from both left and right. Versions have been supported by economists as divergent as Friedrich Hayek (who inspired Margaret Thatcher) and Tony Atkinson (who didn’t). Another variant was part of the Green party’s general election manifesto.

Perhaps the most widely advocated version is the proposal for a UBI – unconditional basic income – to replace most existing social security benefits. A UBI is an income provided without conditions to every adult and child (or, in some versions, only citizens) to provide at least a subsistence level of resources. It is not means-tested, although it is subject to income tax. It is usually assumed to provide an equal payment to everyone, with a special rate for children.

The idea of replacing benefits and tax credits with UBI has huge intuitive appeal. No means-testing of benefits, and thus no families stuck in poverty traps where benefit withdrawal erodes any increase in earnings. No out-of-work claimants afraid to take up short-term job offers for fear of losing benefit entitlement. No intrusive testing of benefit eligibility, no punitive sanctions regime, no jobcentre advisers hassling people to apply for the lowest-paid jobs. No fraud and no gaming the system. Most of the bureaucracy of the welfare system swept away. And the possibility for everyone to take time out of employment as and when they want to and for as long as they want to, answerable to no one.

If this all sounds too good to be true, that’s because it is. To be clear, it needs to be said that some of the more obvious objections to UBI are misplaced. The argument that it is “too expensive”, for example, makes no sense given that UBI is a scheme of income redistribution where gains and losses across the population add up to zero. The same applies to the objection that UBI is politically impossible to deliver: a lot of policy proposals have been politically impossible right up to the time a government implemented them.

But on the journey from initial concept to practical implementation UBI encounters a host of problems, most of which can be encapsulated in a single criticism: it promises a division of labour between government and market that is neither feasible nor desirable, in which the government’s role in ensuring economic security is to redistribute income and then stand back.

This might work in a world of perfect markets inhabited by perfectly rational individuals with perfect foresight and perfect mobility. But that is not a world we live in or will ever live in, and adapting UBI to a more realistic universe undoes most of the advantages claimed for it. We can see this by looking in turn at UBI’s main selling points: no conditionality, no means-testing and equal payments to all.

The current UK government’s punitive sanctions regime makes the idea of unconditional benefits attractive. However, conditionality does not need to take such a severe form, and the risk that unconditional payments would encourage some people to drift into long-term worklessness can’t be discounted. The question is whether having the right financial incentives, as promised by UBI, is enough to prevent this happening.

A world of perfect markets inhabited by perfectly rational individuals is not a world we live in or will ever live in

Single parents in the UK offer a test case, as up to 2008 they were effectively in receipt of something very like an UBI, when not in employment. They had no obligation to actively seek work while tax credits ensured that most would be significantly better off in work. Employment rates had increased since the 1990s in response to improved incentives but remained relatively low, and from 2008 obligations to look for work were imposed. By 2014 the employment rate outside London had risen from 57% to 61%. In London the increase was dramatic from a lower baseline: from 45% to 57%.

The lesson is that incentives matter (as shown by the rise in employment prior to 2008) – but in the absence of conditionality, some parents who would otherwise have been working remained out of the labour market. Even allowing for the fact that many parents moved into low-paid part-time work, it is hard not to see the change since 2008 as on balance an improvement. By not having conditionality in place and relying solely on incentives, the system prior to 2008 led to more children living in workless households, a situation associated with poorer outcomes in later life.

It is hard to see why this lesson would not apply equally to a basic income scheme. Unless we are completely relaxed about long-term worklessness – and all the evidence tells us we should not be – some form of conditionality seems to be essential. But if UBI were subject to conditionality much of what it aims to eliminate would reappear: sanctions, eligibility testing, welfare bureaucracy.

Eliminating means-testing also runs up against the problem that has haunted UK social security policy since Beveridge: housing costs. In the absence of means-testing UBI would have to include a flat-rate component to meet rents. This would imply windfall gains for those without housing costs. Worse, a housing element that would meet London rents would lead to huge windfall gains for people renting more cheaply in other regions. If set at national average rents, London and other high-rent cities would become no-go areas for anyone who ever expected to have to rely on UBI. As geographical mobility is not perfect, a steep rise in urban poverty would be likely.

Finally, equality of payments becomes much less attractive once long-term sickness and disability are brought into the picture. If all involuntary interruptions to employment were short-term in nature, a subsistence level income might be enough to tide people over with the help of savings and borrowing. When people are going to be out of the labour market for months or years, with savings exhausted and no access to credit markets, the proposition is much less attractive. Hence benefits for long-term sickness have historically been higher than for short-term unemployment. But paying a higher rate of UBI to people who can’t work because of disability brings us back to some form of work-capability assessment.

In a world of perfect markets, people could take out insurance to top-up UBI in the event of long-term inability to work. But markets aren’t perfect: most risks are not insurable in private markets – that’s why we have social security systems.

The ad hoc changes to UBI needed to overcome these problems bring it much closer to what existing social security systems already do, however inadequately. And this may be no bad thing: it may be better to think of UBI and other basic income proposals as thought experiments, allowing us to test our intuitions about subjects such as fairness, inequality, property, markets and risk. Outside the confines of the thought experiment, UBI needs to be made much more like existing social security to be feasible. But the thought experiment might also lead us to conclude that existing social security systems need to be much more like UBI to be equitable and efficient.