I recently took part in a gloomy debate about London’s housing crisis with the celebrated modernist architect Kate Macintosh. All the previous speakers had referred despairingly to the demolition of dozens of the city’s housing estates – a process described by the group Architects for Social Housing (ASH) as “the London clearances”. Macintosh talked about her experience in the 1960s designing the south London landmark Dawson’s Heights, and the success of recent community projects at north London’s once-troubled Broadwater Farm Estate.

“Does she not know that Broadwater Farm is also under threat of demolition?” I wondered to myself.

It turns out she did: she broke down in tears while describing what the community, despite all their efforts, now faced. She composed herself and continued, but her response certainly underlined the impact of London’s housing crisis, which in the run-up to the city’s mayoral election has emerged as Londoners’ top concern.

There is a widespread view among the political class that the death of council housing is a triumph to be celebrated

For postwar Britain’s best and brightest young architects, like Macintosh, the hottest tickets were designing public buildings: modernist housing estates, Royal Festival Hall, the National Theatre. Today, in a city with the second-highest house prices in the world (after Monaco), only listed status ensures these modernist buildings remain, with the housing estates themselves usually repurposed for a new world. In east London, for example, all of the flats in Ernő Goldfinger’s Balfron Tower are being sold on the private market, once each of the 146 interiors is stripped out and reconfigured to appeal to investors. Goldfinger presumably is turning in his grave at the idea of his spacious and light-flooded apartments converted into multiple bedsits to be sold at hugely inflated prices.

Dawson’s Heights estate in south London. Photograph: UrbanImages/Alamy

Along with Goldfinger’s Trellick Tower and Berthold Lubetkin’s Hallfield Estate, these are, to some extent, the ones that got away: listed status at least keeps the image of the modernist vision on the skyline. Down the road from Balfron, Peter and Alison Smithson’s Robin Hood Gardens, also internationally acclaimed, failed in its bid to gain listing, and is now one of dozens of estates, housing tens of thousands of people around London, facing imminent demolition.

Much of the architecture – and housing – of the postwar period failed to inspire. In many cases they were built shoddily and imposed in a top-down and undemocratic manner – for us, rather than by (and with) us. But the relish with which it is being torn down smacks of a desire to remove a pillar of the postwar welfare state: the notion of decent housing for all. Glasgow city council decided to broadcast the demolition of the Red Road flats live on television as part of the opening ceremony for the 2014 Commonwealth Games. Following an outcry, the council backtracked. But the very fact it was considered at all reflects a widespread view among the political class that the death and defeat of council housing is a triumph to be celebrated.

Although I have written a lot about the “residualisation” and marginalisation of social housing, I still find it hard to understand how the ideal of social housing has made the journey from Aneurin Bevan’s postwar vision of “the living tapestry of a mixed community” to David Cameron’s depiction of sink estates that blight the lives of their residents. Growing up in the 70s and 80s, I didn’t live in council housing. Some of my friends did, and their houses weren’t so different from mine. I “played out” and rollerskated around estates, which were hardly alien places. This was a Britain of street parties and skinheads, punks and power cuts. It was no socialist utopia, but for many people, including those on council estates, life was OK – certainly much better than the fight for survival that it is today.

The London skyline as seen from the Heygate Estate, which was demolished in 2014. A luxury apartment complex is being built in its place. Photograph: Elephantpix/Alamy

Somewhere along the line this all changed. The impact of Margaret Thatcher’s right to buy and the sell-off of millions of council homes is well known, although the incremental privatisation of every aspect of social and so-called “affordable housing” has been obscured by a media interested in property supplements ahead of housing. For the journalists who have tried to tackle the subject, the jargon and complexity that privatisation always brings hasn’t helped. But as with so much that happens in this country, the roots of what was to come for housing had all started a decade or so before, in the US.

In 1972 an architect called Oscar Newman published a book called Defensible Space: People and Design in the Violent City, which was based on a study of crime on three New York public housing projects. The core of his argument was that crime, which was rife in US housing projects, was not the result of social problems but of opportunism, and could therefore be dealt with through design. Conveniently, this would also be far simpler – and cheaper – than addressing the root causes of social and economic deprivation. His conclusion – which foreshadows today’s demolitions – was that tower blocks produced crime (Manhattan skyscrapers notwithstanding) and should be knocked down and replaced with low rise housing, where private territory and boundaries could be marked out, thereby deterring criminals from entering.

Peter Reigert as ‘defensible space’ champion Oscar Newman in the HBO series Show Me a Hero. Photograph: HBO

Newman’s simplistic view, chiming with an increasingly individualistic political culture, spread like wildfire in US policy circles before arriving in Britain in the 1980s, heavily promoted by the geographer Alice Coleman, who became Margaret Thatcher’s advisor. The consequence was a niche but heavily contested debate between Newman’s acolytes and those who argued that his approach was environmentally deterministic. What really dates this debate is that neither side wanted to do away with social housing, although Newman’s did act as a nail in its coffin, undermining the idea of decent housing for people on lower incomes by linking it indelibly with urban decay.

As a more or less direct result of Newman’s influence, the London Metropolitan police have developed Secured by Design, a set of guidelines for “designing out crime” that new developments must meet to get planning permission. The upshot is that the majority of British housing estates (and schools and hospitals) are now surrounded by gates and high, forbidding fences. Because Secured by Design guidelines state that security must be higher in high-crime areas, which correlate with poverty, deprived parts of Britain are taking on an almost militarised feel. It feels alienating and intimidating. And the rise of “defensible space” has paralleled the privatisation and marginalisation of social housing, with the result being heavily defended estates with high concentrations of poverty and unemployment.

But even that is insufficient explanation for the weight of invective against “sink estates”. This, too, is the result of a well-worn media narrative. It is burnished by the ample resources of PR companies working for the councils and developers engaged in “joint ventures” to pull down these communities and replace them with luxury apartments, marketed to foreign investors at exorbitant prices.

Margaret Thatcher, left, after handing over a copy of the deeds of a flat to right-to-buy purchasers. Photograph: PA Archive

Deprived parts of Britain are taking on an almost militarised feel. It's alienating and intimidating

In Elephant and Castle, the Lend Lease development Elephant Park is emerging from the hole in the ground on the site of the former Heygate Estate, which was demolished in 2014. The Heygate was home to 3,000 people, the majority of whom lived in social housing; but of the 2,535 new apartments at Elephant Park only 79 will be social housing. One quarter of it is supposedly “affordable housing”, though that term has undergone an Orwellian redefinition: it now means 80% of market rent. At the time of writing, the cheapest one-bedroom apartment at Elephant Park is listed for £608,000 while two-bedroom properties start at £750,000.

As investors cash in, residents have been forced out. This market is no kinder to Thatcher’s right-to-buy owners than it is to rental tenants: leaseholders whose properties are compulsorily purchased are routinely offered far less for their properties than London’s soaring market values. The upshot is that many have been priced out of the capital entirely, now living as far out as Sidcup, Thurrock, Sevenoaks and Rochester.

‘Many British housing estates are now surrounded by gates and high, forbidding fences.’ Photograph: David Levene/The Guardian

The property market drove a similar forced population movement under New Labour’s Housing Market Renewal Pathfinder project, in which tens of thousands of people were evicted across the north of England amid similar accusations of social cleansing and state-led gentrification. That market – a familiar unholy alliance of local authorities, housing associations and developers – was no kinder to home owners than it was to tenants, with elderly people often forced out of homes they had owned for decades. Pathfinder, which ran from 2002 until the coalition government abandoned it in 2011, was a misconceived and ruthless project. But at least its original aim was to renew housing markets in areas where property prices had collapsed. In London the opposite is the case, yet although the property market is out of control, no action is being taken to halt its rise.

In 2008 the financial crisis was caused by the sub-prime property crash in the US. Today London is facing a super-prime crisis. The two problems are similar: in the same way that the credit default swaps and CDOs (collateralised debt obligations) of the sub-prime crisis lost all touch with a reality where no actual people were in a position to afford mortgages, London’s property market has entirely lost touch with the needs of ordinary Londoners. It is not uncommon for new luxury apartments to be bought by investors and sold on to other investors, “off plan”, multiple times before they have even been built.

Politicians frame the housing crisis as an issue of supply and demand. They claim that house prices are high because we don’t build enough homes, and that if we simply lift planning restrictions and build more, prices will go down. But the market in London and parts of the south-east does not respond to local demand, because it is fuelled by the demands of global capital flooding into London, much of it from highly dubious sources, as the Panama Papers reveal. If London has been shown to be a hub of global money laundering, the property market is its driving force.

The super-prime market is inaccessible to ordinary Londoners, but it is completely reconfiguring the city. Photograph: Dan Kitwood/Getty Images

The fate of council housing is inextricably linked with this picture, because of the hugely inflated price of the land on which housing estates sit. Paul Watt, of the urban studies department at Birkbeck, University of London, estimates that in London “as many as 90 council-built housing estates are potentially facing demolition as a result of what is called ‘regeneration’”. It is by no mean only the Conservatives who are actively championing the policy: the Labour council in Lambeth, for example, has identified six estates for regeneration, with decisions made to demolish or part demolish three of them.

While David Cameron has heavily promoted the sink estate narrative to justify “estate regeneration” – essentially a euphemism for demolition – Lord Adonis, the former Labour minister appointed by George Osborne to chair the national infrastructure commission, is given to blunter statements, having made it clear that a central reason to knock down London’s estates is that they are sitting on “some of the most expensive land in the world”.

Those losing their homes are far from the only ones affected: other businesses and institutions are feeling the heat as well. London Metropolitan University, for example, plans to sell the Cass School of Architecture in east London, known as the “Aldgate Bauhaus”, for £50m and redevelop it as luxury flats.

Property developers look at a scale model of the city of London at the 2014 Mipim conference in Cannes. Photograph: Valery Hache/AFP/Getty Images

The super-prime market in London may seem unreal and disconnected from the needs of Londoners, but it is completely reconfiguring the city, with over 300 towers under or awaiting construction across the city. Many of these are luxury apartments going up in privatised, gated enclaves, stretching from Nine Elms, around the huge new Battersea Power Station development and US Embassy quarter, to Southwark and Blackfriars.

Who is going to live in these places? Who are the figures populating the developers’ hoardings? Who can afford to live in these fantasy worlds? Particularly unpalatable is the link between local authorities, who should serve the public interest, and the gilded lifestyles these properties promise. This is where the PR companies who oil the wheels of this new globalised property industry come in, hosting champagne receptions for councils, developers and foreign investors property industry shindigs. At last autumn’s event at London’s Olympia, sessions aimed at local authorities included: “Are you sitting on an untapped goldmine?” and “London – from social housing to super-prime.” You couldn’t make it up.

Anna Minton is course co-director of Reading the Neoliberal City at the University of East London, which along with Birkbeck is hosting a conference on London’s housing crisis on Saturday 23 April.