The title of the recent housing white paper suggested a new urgency to government policy. It’s not hard to see why that’s needed. Homelessness is rising fast and a whole generation finds itself priced out, yet unable to get one of a dwindling number of social homes. The sudden return to a world where working families must rent privately at eye-watering rents is crippling finances, productivity and economic growth alike. Levels of home ownership have been falling sharply since 2003, to only 51% of households today. The dream of a property-owning democracy, that cornerstone of conservative ideology, is almost dead. No wonder the government is anxious.

But the white paper is only the latest in a long line of policies, strategies and initiatives that promise to solve the housing shortage. All before have failed, for the same fundamental reason. For decades now we have been unable to address, or even understand, the role of land in the economy.

Land is obviously important for housebuilding, but the land problem goes much deeper than our housing shortage. It lies right at the heart of many of the economic problems we face today. Financial instability, mounting inequality, debt overhangs and the puzzle of stagnant productivity are all direct results of our failure to properly account for and manage land in the modern economy.

Land first began to be treated as tradable, private property in the 16th century – triggering the birth of modern finance and parliamentary democracy. Yet the economic transformation created by the privatisation of church and common land impoverished millions, as vast numbers were driven off it.

There was something manifestly unfair about a minority turning what had previously been a common source of food, shelter and security into a source of private wealth: those who had to rent land found that any additional value they could generate was quickly absorbed by higher rents, leaving them no better off even as the economy grew.

Since the start of the liberal political and economic project, thinkers have wrestled with these contradictory consequences. John Locke strove desperately to justify private property in land, precisely because it was so essential to the classical liberalism he espoused. Adam Smith was more clear-eyed about landowners’ ability to extract undue levels of rent from workers and capitalists alike – and worried it would limit productivity and undermine the political legitimacy of property itself.

Smith believed that the state’s job was to protect the institution of private property from democracy by minimising its worst effects, while Jean-Jacques Rousseau thought property should be sacrificed to save democracy. The anarchist Pierre-Joseph Proudhon summed up the paradox in two pithy statements: property is theft; and property is liberty.

The 20th century saw the same story played out all over again. This time, the focus was not on farmland but on housing. Today Savills estimates that the UK’s housing stock is worth £6.8 trillion – compared with £190bn for all agricultural land.

For a while, council housing provision, state provision of land for housing development, tight mortgage regulation and taxes on home ownership had kept supply up and house prices under control.

By the late 1960s so many homes had been built that political attention could shift away from supply and on to boosting the slackening growth in home ownership. Taxes on home owners were replaced with subsidies for buyers. The new towns programme was stopped, ending the implicit guarantee of state land supply. Councils were forced to sell their housing stock and barred from building more. Mortgage lending was deregulated, unleashing credit into the housing market and creating a new model of “residential capitalism”. The era of housing booms and busts was born.

‘To really fix the problem of land, we need to do more than increase the number of houses built.’ Photograph: Bloomberg/via Getty Images

Today, the dark side of the land ownership paradox has reasserted itself with a vengeance. Homes routinely “earn” more than people do, and the banking sector is hopelessly addicted to mortgages rather than lending to productive enterprises.

Even the relatively well-paid struggle to house themselves adequately within commuting distance of jobs, and businesses say employee housing costs are the biggest brake on their expansion. The problem of rent is alive and well – and yet we seem unable to let go of the dream of owning a home that will always go up in value, liberating us to spend our income on consumer goods and leisure. It seems Proudhon was right on both counts: property is both theft and liberty.

To solve these problems, we have to recognise that land has special characteristics that make the normal rules of supply and demand inoperable – and make market exchange a poor way of handling it. Land is inherently scarce, and its control is inherently political. Housing policy should therefore seek to level the playing field between tenures (in terms of taxation and subsidies) so that people are not incentivised to overinvest in home ownership. At the same time, we should grow the stock of non-market housing, like social housing and community-led schemes, to lessen dependence on the volatile market in land and homes.

The housing white paper, and the rhetoric around it, hints that government may be moving in this direction. There are moves to allow councils to create development corporations and to compulsorily purchase sites with planning permission that the owners seem reluctant to build on – which could lower the cost of land for new homes.

But to really fix the problem of land, we need to do more than increase housebuilding. We need to break the positive feedback cycle between the financial system, land values and the wider economy, and to capture more of the unearned windfalls private landowners currently pocket at the expense of society at large. And that will require bold financial regulation and tax reforms, and more direct intervention into the dysfunctional land market. Only this will allow us to better balance the benefits and harms caused by what Churchill called “the mother of all monopolies” the ownership of land.

• Rethinking the Economics of Land and Housing (Zed Books) by Josh Ryan-Collins, Toby Lloyd and Laurie Macfarlane, is out today