Here is something you might try if you live in Britain. Go to your favourite urban place, whether it be the centre of a large city or a small market town. Close your eyes, turn around three times and walk in that direction for 15 minutes (or an hour if you’re in London). I can predict with a reasonable degree of confidence that the place where you end up will be crap.

You may be stuck in the no-man’s-land around the ring road, or in a brutally functional industrial estate, or among the endless rows of parked cars in a retail park, or lost in a tangle of suburban cul-de-sacs. Wherever you are, the environment will generally be bewildering, illogical and ugly.

Planning has gone too far. It has dissolved the pattern of the traditional town

It is likely that your favourite place, where you started, was built before 1947 – quite possibly long before – and the place you ended up has been created since. This is not an argument against modern architecture. I love modern architecture and, with the possible exception of the council estate, it can’t be blamed for the problems of the place where you are now standing.

Town planning is a different story. My reason for choosing 1947 is that it was the date our modern planning system was conceived, the result of the Town and Country Planning Act – part of a raft of postwar legislation to, among other things, establish the NHS, set up the welfare state and tackle the problems of our cities: chiefly congestion, slums and pollution. It is a sad indictment of this system that most of the places it has created have been a disappointment. They may be healthier, safer and more efficient – but they lack the character, diversity and human scale of a city street, market town or village.

Facebook Twitter Pinterest Retail parks: the unwitting result of a set of planning rules that have led to a result no one foresaw. Photograph: Michael Heath/Alamy

We can’t blame designers for this, because the reality is that most of these suburban estates, retail parks and highways schemes were never really designed at all. They are the unwitting result of a set of planning rules that have led to a result no one foresaw.

Most of the rules were well-intentioned. They were designed to stop us getting killed by cars while allowing traffic to flow, to improve the quality of our environment and give us access to green space, to separate housing from polluting industry, to make business more efficient, to encourage light and healthy housing, and to give places for our children to play. All worthy aspirations, and all the responsibility of different professionals who have likely done a good job in their particular area of responsibility. It is just that no one has been looking at the bigger picture, at what sort of environment is created by the combined effect of all of these well-meaning rules.

Take the late 20th-century housing estates which encircle most of our towns and cities, and of which the sprawling Bradley Stoke district of Bristol is perhaps the most infamous example. These were largely the result of rules set out in the innocuous-sounding Design Bulletin 32 from 1977. Design Bulletin 32 stipulated, for example, that all new estates should have a hierarchy of roads, or that houses could only front on to the most minor roads. Mix in rules about every house having two off-street parking spaces, privacy distances between windows and the complex arrangements for ongoing maintenance, and hey presto: you get the cul-de-sac.

Facebook Twitter Pinterest A cul-de-sac in suburban London. Cul-de-sacs became more prevalent in the UK as a result of 1977’s Design Bulletin 32. Photograph: Dan Chung/The Guardian

In the early 2000s, government did try to reform Design Bulletin 32, first publishing a companion guide called Places, Streets and Movement, and then a new version called Manual for Streets in 2007. But research last year by the Urban Design Group showed that fewer than 20% of highways authorities had updated their guidance.

I don’t deny that many people like suburbs. But it is undeniable that the UK suburb – a tree with cul-de-sacs branching off the stem, unconnected to each other – is confusing, makes walking inconvenient and unpleasant, suffocates local shops and makes public transport unviable.​

Planning incentives 'lead to housing estates centred on car use' Read more

Contrast this to that favourite urban place of yours, where we started. Its spaces are likely to be human in scale, with walkable streets that connect to each other and make it easy to get around. It is probably quite busy, supporting all manner of shops, bars and restaurants. It is made up of lots of different buildings, many of which are not particularly interesting in themselves but frame streets and spaces that are a pleasure to be in. Some of this may have been “designed”, but there are many attractive urban places that never were. The Academy of Urbanism gives out awards for these good urban spaces each year: last year, the shortlist included places as diverse as Barnsley, Chelmsford, Paisley, Kelham Island in Sheffield and Hackney Wick in London. These are not pretty or quaint, but they have a satisfying, self-organised urbanity that we humans used to be pretty good at.

Facebook Twitter Pinterest Kelham Island Quarter in Sheffield: not pretty or quaint, but with a satisfying, self-organised urbanity that we humans used to be pretty good at. Photograph: Alamy

In Britain, any piece of land left undisturbed will first be colonised by small plants, then shrubs and then trees. After a hundred or so years, it will become mature broadleaf woodland – and, having reached that steady state, will remain that way for ever – a state known as “climax vegetation”.

Do human societies go through a similar process when building cities? People come together to live in a particular place, negotiate their relationship to each other, build homes and businesses that combine to create streets and squares that are just the right size to accommodate urban life. We might call this climax urbanism.

In the 1960s the great urbanist Edmund Bacon, describing the Umbria hill town of Todi in Italy, wrote:

The collective mind of the citizens … must have conceived the space volumes of the two squares as abstract entities, and then brought them into being by the construction of individual buildings over many years.

It is a process that can be seen at work in Syrian refugee camps in Jordan, as documented by Husam Alwaer at Dundee University. In the Zaatari camp, residents have invented a contraption with wheels that allows them to rearrange their UN-provided cabins. What started as regimented rows of cabins has gradually been reconfigured into family groupings, with those cabins on the “main street” turned into shops, so the result is not unlike a traditional Syrian town. The same process can be seen in Indian slums, which, despite their often dreadful living conditions, could be seen as a form of proto-urbanism: they may be built out of corrugated sheeting and tarpaulin, but their function and form is not so different from a historic Indian city such as Jodhpur.

Facebook Twitter Pinterest Zaatari refugee camp, north Jordan, close to the border with Syria. Photograph: TASS/Barcroft Images

Did that beautiful English market town or city street where we started our journey go through a similar process before it reached its climax state? Is the problem with the modern planning system that it just doesn’t understand this process of city growth? Does it think new neighbourhoods can be designed on paper and wished into existence like the new towns of the 1960s, the council estates of the 1970s, the Docklands development of the 1980s or the clearance of terraced housing in the 1990s? Maybe planners need to understand that urban areas are grown, rather than wished into existence under their blue utopian skies.

There is an academic field that explores these ideas. Complexity theory was originally developed to explain the “design” of insect colonies or slime mould communities, but groundbreaking work in the 1970s by people like Lewis Thomas and Christopher Alexander sought to apply these ideas to cities. Ants live in hugely complex and beautifully regulated colonies without any form of leadership. A small number of pheromone signals by which ants communicate, combined with a millennia of natural selection, have fine-tuned these colonies to perfectly meet their needs. What if the interaction of humans over many years worked in a similar way to create settlements that are ideally suited to the needs of the community?

It is an appealing thought, though there are a few problems with the idea. The first is that humans no longer seem to be very good at optimising their settlements. Entirely unplanned development is more likely to create a slum, barrio or shantytown than an attractive neighbourhood. The journalist Peter Popham, in the book City of Darkness, summed it up well when he wrote about the Walled City settlement in Kowloon, Hong Kong:

All this intensity of random human effort and activity, vice and sloth and industry, exempted from all of the controls we take for granted, resulted in an environment as richly varied and as sensual as the heart of the tropical rainforest.

Except, as he points out, “it was so obviously toxic, full of brothels and drug dens, pollution and poverty.”

Facebook Twitter Pinterest The slums of the Walled City, Kowloon, Hong Kong. The settlement was demolished in 1992. Photograph: Alamy Stock Photo

Maybe the Walled City just needed more time to reach its idealised or “climax” state. But what is perhaps more likely is that humans are no longer very good at acting in society’s (or even their own) best interests. The terrible conditions in the unregulated city were the reason that planning was invented – and why it is still needed.

The second problem with the application of complexity theory to cities is the fact that humans fail one of the basic tests of a complex system. As Stephen Johnson, whose book Emergence is one of the best overviews of the field, writes: “The relative stupidity of the individual ant is, as the computer programmer says, a feature not a bug.” If a group of ants (let’s call them planners) were to become sentient and start trying to direct and shape the colony, they would cause havoc and probably destroy the system.

It may be true that planners do not understand the natural process by which traditional places were built. But that is not the lesson to be drawn here. There have always been authorities with an overview of human settlements. As my colleague Shruti Hemani shows in her research, even in slums there are “founder families” that exercise a degree of control. The point is that planners do not sit above the system, bending it to their will: cities are complex, emergent patterns that result from the interaction of a huge number of variables, including society’s norms and values, the working of markets and the impact of technology. Planning is just one of the inputs to the system, and probably not the most important one.

But planning has gone too far. In attempting to “create” cities by regulating every tiny facet of them, it has unwittingly caused what complexity science calls a “phase transition”: it has dissolved the pattern of the traditional town and turned it into the retail park, the suburban cul-de-sac and the industrial estate. These are not places that anyone designed, but a new, emergent climax state that the people drawing up the rules never even envisaged, let alone intended.

Calling for a master plan in urban living – archive, 19 December 1959 Read more

There is a place in Holland where an experiment is taking place to rewrite these rules. The Homeruskwartier district of Almere in the Netherlands has been divided into plots that are being sold to individuals and groups to build their own homes. The rules of the plot, like the pheromone signals of the ant, are very simple: they can be written on one side of A4 paper regulating the position, size, and use of what can be built on the plot. The neighbourhood is being built one house at a time and no one knows quite what the result will be. There is a masterplan that gives it structure, but provided they stay within the parameters of their “plot passport”, and are able to negotiate with their neighbours, residents can build what they want.

This is a neighbourhood that has been coded rather than planned, and ideally over time the programme parameters would be tweaked to create the neighbourhood residents want. It may seem radical, but actually it is the way that we used to do things, before 1947, when the well-meaning planning system planned away the art of building good places.

David Rudlin is currently director of URBED, chair of the Academy of Urbanism, honorary professor at Manchester University and co-author of Climax City: Masterplanning and the Complexity of Urban Growth

Follow Guardian Cities on Twitter, Facebook and Instagram to join the discussion, catch up on our best stories or sign up for our weekly newsletter

• This article was amended on 11 April 2019 because in an earlier version, the caption to a picture of a London cul-de-sac incorrectly implied that the houses shown were built after 1977’s Design Bulletin 32. This has been amended to say that the picture shows an example of the type of street that became more prevalent after 1977’s Design Bulletin 32.