Whitechapel Underground Station in the East End of London is a long, wide trench, an unexpected burst of sunlight that comes just a couple of minutes after your train leaves the City. Being mostly subterranean, the Tube does not generally foster window-gazing, but here the raised, curious eye is magnificently rewarded. The train passes through a chasm of tens of millions of bricks, not one of which is truly intended to be seen: the canyon’s arched retaining walls, the plain huts and outbuildings used by the Underground’s operators, and the rears of the Victorian terraces on Whitechapel Road and surrounding streets.

This brickscape is just a backdrop. It is painted over by an impossible multitude of stains and seepages, deeply overgrown by pipes and cables. In places, an unplanned arrangement of steel I-beams suggests mismatched forces and structural quandaries. Overpasses bear streets above us. Creaking clapboard walkways carry our fellow passengers. More trains pass below – paradoxically, it’s the Overground beneath the Underground. While many other Tube stations have criss-crossing routes and rumbling suggestions of deeper lines, here we can actually see those other trains and platforms; the whole station, in fact, has an eerie sense of unintentional exposure, as if the surface city has been peeled back in layers like one of Gunther von Hagens’s plastinated corpses, urban viscera laid bare for inspection.

There is no single vantage point at which one can take in the whole of the scene: it reveals itself in turns and blockages. On the eastbound District line platform, the underside of a concrete stairway emerges from a wall and disappears into a ceiling, a hidden and separate space intruding mysteriously into our own. Whitechapel station is one of Giambattista Piranesi’s imaginary prisons, colonised by frantic electrical engineers and watched over by CCTV. A new line, the Crossrail heavy-rail link, is now forcing its way through this extraordinary knot with the odd combination of tact and ultraviolence so characteristic of civil engineering.

A tower crane squeezes between platforms. There are glimpses of deeper shafts and workings, new Morlock worlds. The ground vibrates. Robotised surveyors’ sights continually swivel and nod, alert and panicky, checking that nothing is moving that shouldn’t be. There will be new connections, new systems, new platforms, new lifts, new overpasses. The station will be improved, but it will not – cannot – be simplified.

As a human landscape, Whitechapel station is magnificent. Everything in it is deliberate and by accident. Everything is unconcealed but enigmatic. It belongs to no style or era, nor could it – it is the encrustation of successive waves of technology. It is not even unique: there are many places like it on the Underground, and countless others in different cities and countries. When urbanism reaches a particular pitch of intensity and remains there for decades uninterrupted by catastrophe or blank-slate utopianism, this is where we find ourselves. It is here that we come face to face with a fundamental urban quality: the play of layers, tunnels and bridges, the meeting of different forms of transport, the chaotic revelation of normally hidden systems and services, an accumulation of improvisations that marries rush with desuetude. Let’s call this quality ‘tangle’.

I grew up in the university city of Oxford, a place profoundly blessed with architectural monuments and urban set pieces: the ascending curve of the High Street, the generous expanses of Broad Street and St Giles’, the near-perfection of Radcliffe Square – all the product of centuries of aesthetic stewardship. These are urban spaces, and they are exquisite.

But there were other, peripheral parts of the city that elicited a different and novel thrill. The complex of multi-storey car parks and glass walkways behind the Westgate shopping centre, while unlovely and inhospitable, had a kind of uncompromising stare toward tomorrow that spoke directly to the spine – in particular, a smoked-glass geodesic dome that enclosed part of the stair tower. Better yet was the frosted glass conveyor bridge over the city’s eastern bypass, which carried half-finished cars from one half of the Cowley Rover works to the other. Both of these structures fascinated me as a child. Both were demolished before I became an adult. They are so unlamented that it is nearly impossible to find photographs of them.

In creating a city of layers, tangle made the moral problem of the city three-dimensional

More of these moments of urban excitement could be found on our regular visits to London relatives. We always came from the west, via the dizzying elevated sections of the A40 or M4 that weave through the spires of satellite office developments. At the end was the fabled Westway, passing over a vast dairy and its huge sign promising FRESH MILK FOR LONDON. London’s inner roads offered moments of unforgettable drama, particularly their underground sections: Upper Thames Street in the City, Beech Street under the Barbican. These layered roadways announced that we were in a real city, a metropolis, a terrain so congested and contested as to demand these complexities and multiplications of use.

Why should tangle be so intrinsic to the idea of the city? Or, perhaps I should say, to my idea; for there is, of course, another tradition, older but still very much alive, that sees things in a different light. Ever since St Augustine’s City of God (AD426), urban order has stood for virtue. This vision of a more than merely aesthetic purity shines through the imaginary streets of Filarete’s 15th-century Sforzinda, Ebenezer Howard’s early 20th-century Garden City, Le Corbusier’s 1920s Ville Radieuse. It is regular, logical, symmetrical. It is utopian. And it is too good for this world.

It was modernity itself that put paid to Augustine’s heavenly geometries. As new systems and services entwined, the city became increasingly impervious to simple solutions, relying ever more heavily on patching, adapting, retrofitting, bypassing, underpassing and overpassing. In creating a city of layers, tangle made the moral problem of the city three-dimensional: it dug through the virtuous bedrock, subverted solid walls and cast godly streets into shadow. Even as the Victorians were setting up the home as the inviolable ideal building block of society, their technologies were undermining it. ‘[T]he simultaneous appearance of advances such as central heating, sewerage, intercoms, elevators, and, a little later, electricity ensured that from the 1870s on, the interior of the building was criss-crossed by a complex of pipes, cables, and shafts,’ writes Andreas Bernard in Lifted (2014), a fascinating history of the elevator:

Beneath the visible surface there arose an invisible network that organised the circulation of energy, data, and people. In the end, this process of mechanisation and electrification made it necessary for the formerly independent unit of the house to become networked with its surroundings, for only the connection to external power sources and centrally regulated reservoirs and generators ensured the functionality of its technical installations.

The backs (and sometimes fronts) of houses sprouted new pipes, extensions, cables, and – later – television aerials, satellite dishes and air-conditioning units. This could be tangle’s flag: a disused brick chimney supporting half-a-dozen TV antennae, some unplugged and forgotten.

What went for the house went for the city: tangle-lands of rail junctions and highways can be traced around most larger cities, often handily indicating the urban boundary circa 1850, as new systems of circulation and supply started being plugged in. South of the Thames river, from Clapham to Bermondsey, London is a warren of railway bridges: brick arches, sometimes cyclopean in scale, seem to frame every street, an urban feature as consistent and distinctive as Venice’s canals. Even after its Olympic makeover, the Lea Valley in east London is a reference library of different modes of circulation: passenger rail, freight rail, light rail, river, canal, canalised river, urban motorway and a pedestrian ‘Greenway’ that is in fact the raised conduit of the Northern Outfall Sewer: the city’s cloaca.

I draw these examples from London because it’s the city I know best, but the same or similar can be found in any of the great northern British cities, often aided by dramatic terrain, and in Chicago, Berlin, New York, Hong Kong, Moscow. If tangle has a world capital, it is probably Tokyo, where, writes the Australian architect Julian Worrall: ‘In the densest parts… even the ground on which you walk has become a constructed thing. It is often not clear where you stand in relation to the surface of the earth – whether you are underground, elevated, or securely grounded.’

This sense of disconnection from our usual plane might be a basic ingredient of tangle. It is hard-edged, to be sure, but it is also lacks certain familiar boundaries and certainties. We lose ourselves in its coils.

‘The form of the metropolis,’ the American critic Lewis Mumford announced in his landmark study The City in History (1961), ‘is its formlessness.’ He did not intend it as a compliment. Mumford condemned the Morlock tunnelling, inhuman scale, anthill frenzy and snarled networks of the post-Victorian city, a pell-mell catastrophe pregnant with atomic obliteration. He believed the city-dweller had become a slave enchained by delusion to the machinery of consumption:

The metropolis, in its final stage of development, becomes a collective contrivance for making this irrational system work, and for giving those who are in reality its victims the illusion of power, wealth and felicity, of standing at the very pinnacle of human achievement. But in actual fact their lives are constantly in peril, their wealth is tasteless and ephemeral, their leisure is sensationally monotonous, and their pathetic felicity is tainted by constant, well-justified anticipations of violence and sudden death. Increasingly they find themselves ‘strangers and afraid’ in a world they never made: a world ever less responsive to direct human command, ever more empty of human meaning.

Mumford belongs to the ranks of the utopians who, though they disagreed violently about almost everything, shared a dream of a city freed at last from tangle: a lineage stretching from the ‘garden city’ and ‘city beautiful’ movements of the 19th and early 20th centuries through the Corbusier modernists to the ‘new urbanists’ of today. But tangle is forgiving: it happily absorbs the dog-ends and stumps of the utopian systems that intended to sweep it away.

Architects relished the grand aesthetic possibilities that tangle presented, seeking to build on the knot rather than unravel it

After the modernist urban project foundered in the late 1960s and ’70s, before the prissy, reactionary ‘new urbanists’ reared their heads – preaching a return to small-town scale and (less loudly) traditional values, hanging baskets and the Neighbourhood Watch scheme – something remarkable seemed to happen. Architects and writers began to look directly at the half-planned city of layers and freeways and conflicting, intertwining systems and appreciate it for what it was – to find the enchantment in it. There was a precious moment of urban realism in between long bouts of utopian wishfulness and sterile escapism.

Architects such as Paul Rudolph in the US and the Metabolists in Japan relished the grand aesthetic possibilities that tangle presented, seeking to build on the knot rather than unravel it. On the polemical side, the US architects Robert Venturi, Denise Scott-Brown and Steven Izenour learnt from Las Vegas, setting out the revolutionary idea that the Strip, with its semiotic chaos of signs and honky-tonk energy, might not be the hell it was generally held out to be. The US architectural historian Vincent Scully, writing in 1969, demurred, insisting on the ‘horrid viciousness’ of the Strip: ‘Some of the growth is cancerous, perhaps deadly. Frontier hedonism has always teetered on the edge of murder.’ But even he was forced to acknowledge the astonishing spectacle that was rising around him:

Yet it is at the vaster scale of the throughway and connector that the most devastating achievements of modern industrial life now grandly and savagely unfold. The rush of the movement itself, only partly under control and wild with danger, is the right and appropriate way to perceive the new urban scene, with its promise of smashed bodies, its mechanical roar, and its rubbery scream, just as the laggard footpace accompanied by the sound of human voices was the proper way to ascend the Spanish Stairs. That norm was peace; ours is apparently mechanised war. One now dives rattling past the towers of the city, lighted in rising tiers, and flashes through its heart, high above the little houses, released from confinement as through one reckless act of destruction and escape. It is the jailbreak of the urban masses, one-handed with a beery tattoo on the roof at 70 miles an hour in two tons of old iron. Baseball-bat-basher, beer-belly, breaker of cities, comes into his own in ours…

European observers might try to romanticise this distinctively American urban scene, but it was Americans who had to cope with the ‘violent and ungenerous social attitudes’ that it engendered. Even so, Scully granted, ‘in perceiving works of art there comes a moment when we must pass beyond preconceptions to an empirical recognition of what exists. Otherwise we can never learn anything’:

And all this is here; some of our greatest painters have seen it. It is our most complete work of art, multiple and tremendous. Its uniqueness, its inconceivable brutality, even only its visual magnificence, can never be denied.

Unique, brutal and magnificent. We are talking, of course, about a kind of urban sublime, one that rests upon sublimity’s ‘delicate equipoise of conflicting emotions’, to quote the MIT historian Rosalind Williams’s Notes on the Underground (1990). ‘It is connected with pain and fear, but not too closely; it is defined by nervous tension, but not too much; it depends on danger, but only theoretical danger.’

Only theoretical? But our sublime city is ‘violent and ungenerous’, a state of mechanised war. Could it still thrill us if it didn’t also threaten us?

In 1974, when mainstream discourse held the metropolis to be a behavioural sink fit only to be turned over to the waiting Droogs, the British author Jonathan Raban’s treatise Soft City made a case for city dwelling. Writing chiefly about London and New York, Raban acknowledged that they could be unfriendly and often violent (far more so than today), but added that they offered hugely attractive social and psychological possibilities. Where Mumford finds the teeming metropolis alienating and false, Raban sees it as a splendid masquerade, a place where one is free to try on and slough off identities multiple times in a day, a stage for exploring and defining the self. The difficulty and danger become perverse sources of pride and belonging, distinguishing the seasoned urbanite from the greenhorn. The city’s anonymity and restless flux are in fact its strength: its plasticity, its softness, where it awaits ‘the imprint of an identity’.

Raban’s free and expressive city life requires fluency in certain conventions. One has to learn the city’s codes, its languages. ‘So much takes place in the head, so little is known or fixed,’ he writes. ‘Signals, styles, systems of rapid, highly conventionalised communication, are the lifeblood of the big city. It is when these systems break down – when we lose our grasp on the grammar of modern life – that the Envies [a violent gang] take over.’

These ‘systems’ constitute an invisible social infrastructure; one, indeed, in which we might see a certain resemblance to the city’s tangle of service mechanisms. Both are always multiple, improvised, threatening to break under the weight of the city’s sheer extremity:

Language codifies an order, a hierarchy, a stable view of the world, which is grossly exceeded by the reality of the modern city: and the arrival of the immigrant propels him into abstractions and the contemplation of his own internal state of mind. It is a source of transformations and distortions of scale which can only be received with dumb wonder. In American fiction and autobiography, the townscape of New York is turned into a giant perceptual conundrum, as if it has been deliberately designed as a monstrous challenge to eyes and ears accustomed to the human proportions of village and small-town architecture. … If the city is normal, then he is a dwarf; if he is normal, then the city must be some kind of concrete optical illusion, for what perspective or grammar exists in which such breathtaking heights and breadths are possible? Shifting in size, at once dwarfed and elevated by these amazing confusions of scale, the greenhorn lurches forward into his myopic destiny. … City architecture is an eloquent proclamation of the absolute strangeness of city life, a reminder that here you abandon hope of holding on to your old values, your old language.

The city is a machine for teaching people to be city-dwellers. But first it necessitates submission, which it extracts by means of its scale, but also its complexity, its confusions – its tangle.

violence is not the secret ingredient of tangle. Violence, after all, has dwindled, yet tangle remains sublime

And, perhaps, its savagery. For Scully and Raban alike, the city’s sublime awesomeness comes sinisterly entwined with intimations of social breakdown. Just as 18th-century aesthetes sought their thrills in gothic ruins, the writers and thinkers of the 20th century often seem to get a frisson from contemplating society’s rubble – the Envies and the Droogs in the underpass, the black rubber running off the edge of the expressway. Then again, Raban and Scully were writing in rougher times, when urban violence was (it appeared) unstoppably ascendant. They could not see one without the other.

In fact, violence is not the secret ingredient of tangle. Violence, after all, has dwindled, yet tangle remains sublime. And its sublimity rests in its inhuman scale: its apparently overwhelming complexity, its rejection of the ‘laggard footpace’, its brutal practicality, its resistance to adornment. These things are magnificent in themselves.

But they are not fashionable. De-fanged by the sickly nostalgia of the new urbanists and the seductive sales patter of the postmodernist forces of ‘regeneration’, our urban discourse has settled into a cosy agreement that all we can and should do with the city is ameliorate and prettify. The consensus disdains intensity and drama, and obsesses about detail and paving and planting. Heavy infrastructure must be ignored, screened or removed. The human scale is venerated above all else.

I think we’ve got this backwards. The improved safety of the city gives us an unprecedented opportunity to celebrate, even foster, challenging, exhilarating urban spaces. Don’t remove the Westway or sheath it in greenery – delight in it as it is. Miami shows the way, with the Swiss architects Herzog and De Meuron’s parking structure at 1111 Lincoln Road: a concrete multi-storey car park designed and built without the slightest embarrassment or deference. The British writer and J G Ballard scholar Chris Hall has called this ‘post-Ballardian’ architecture: architecture that shrugs off the stigma that has adhered to the concrete city of towers and tangle since the 1970s.

Meanwhile, a fresh breed of utopians have a new weapon in the ancient fight against tangle: data. The 19th century had sewers and rail, the 20th century had electricity and highways, and the 21st century brings sensors and cameras. Visible and invisible infrastructure will be joined by immaterial infrastructure, the ‘smart city’. This is the old modernist highway planners’ dream of uninterrupted flow, only this time the flow is of bytes, not cars. Every blinking lamppost will report back to HQ, every pothole will have a social media account – for every physical glitch, a virtual snitch.

Awash with data, the smart city will liberate its citizens from the illegibilities and frustrations of the dumb city. Taxis will come to us, taxes will fall as services become more cost-effective, traffic jams can be bypassed, restaurants will always have tables, and the anonymous mass around us will become an exciting playground of consequence-free sexual encounters.

Beware. Tangle – the crunching intersection of different heavy-duty infrastructures – might not be pretty, but it serves an important, if unintended, civic role. It is our daily reminder that living in the city means occupying part of a hugely complicated machine comprising dozens of competing systems. The technical compromises and trade-offs that make up the modern city are made visible – no one wanted them to be visible, but that’s just another compromise. We are shown the sheer difficulty that underpins the human hive.

The urban mind needs its regular confrontations with tangle, too, a bracing shock that places the world in perspective

By contrast, the rhetoric around smart cities, like network-evangelist rhetoric in general, is all invisibility, intangibility, seamlessness and ease. The whole thing depends on massive physical infrastructure, of course – ever-growing server farms and fibre-optic grids – but this material truth has proven itself easier to hide than earlier systems. In place is talk of wirelessness and clouds, which might sound unblinkingly modern but are in fact a kind of genteel evasion. The invisible air gap between our smart phone and the nearest router or transceiver resembles a sheet of William Morris wallpaper: a thin layer of decoration to hide the actual machinery of the world around us.

There are dangers in full invisibility. ‘Unlike a rusting highway bridge, digital infrastructure does not betray the effects of age,’ wrote Rusty Foster in a recent New Yorker article on the Heartbleed bug, which exposed the potentially catastrophic vulnerability in the security systems of the web. We can see the filth building up on the cables running along the Underground tunnel, see the wear on the steps of the overpass. The need to continually watch and maintain these systems is always apparent. Less obvious is the unpatched, expired software that leaves critical infrastructure open to attack.

The city needs places of solace, calm, order and beauty – even prettiness. But prettiness and concealment are anaesthetic. The urban mind needs its regular confrontations with tangle, too, a bracing shock that places the world in perspective and informs us, without either warmth or rancour, that our lives are enmeshed in a vital mechanism. The city is a machine for teaching people to be city-dwellers: one made up of crushing cogs and steel.