Architecture, the most public of endeavours, is practised by people who inhabit a smugly hermetic milieu which is cultish. If this sounds far-fetched just consider the way initiates of this cult describe outsiders as the lay public, lay writers and so on: it's the language of the priesthood. And like all cults its primary interest is its own interests, that is to say its survival, and the triumph of its values – which means building. Architects, architectural critics, architectural theorists, the architectural press (which is little more than a deferential PR machine) – the entire quasi-cult is cosily conjoined by mutual dependence and by an ingrown, verruca-like jargon which derives from the more dubious end of American academe.

From early in its history, photography was adopted by architects as a means of idealising their buildings. As beautiful and heroic, as tokens of their ingenuity and mankind's progress, etc. This debased tradition continues to thrive. At its core lies the imperative to show the building out of context, as a monument, separate from streetscape, from awkward neighbours, from untidiness. A vast institutional lie is being told in architectural magazines the world over, in the pages of newspapers and in countless television films. It's also being told on the web – which is significant, and depressing, for it demonstrates how thoroughly the convention has seeped into the collective.

The mediation of buildings can never be neutral. As long ago as the 1930s, Harry Goodhart-Rendel observed: "The modern architectural drawing is interesting, the photograph is magnificent, the building is an unfortunate but necessary stage between the two." Goodhart-Rendel was an architect who belonged to no school, and is thus regarded as peripheral. He was also a writer: a rare combination of talents. John Vanbrugh was a good playwright who became a great – if not the greatest – of British architects. Sergei Eisenstein trained as an architect but gave it up. Thomas Hardy, to judge from Max Gate, the house he designed for himself in Dorchester, made the right choice when he elected to abandon architecture in favour of writing.

Architecture talks about architecture as though it is disconnected from all other endeavours, an autonomous discipline which is an end in itself. Now, it would be acceptable to discuss opera or sawmill technology or athletics or the refinement of lard in such a way. They can be justifiably isolated, for they don't impinge on anyone outside, say, the lard community – the notoriously factional lard community. To isolate architecture is blindness, and an abjuration of responsibility.

If we want to understand the physical environment we should not ask architects about it. After all, if we want to understand charcuterie we don't seek the opinion of pigs. Architects make the error of confusing a physical environment with what they impose on it. Wrong. What is going on around us is the product of innumerable forces. Accidents – some happy, some not – clashes of scale and material, municipal idiocies and corporate boasts – these are some of the more salient determinants of our urban and suburban and extra-urban environments. Buildings are, of course, the major component of these environments. Some of those buildings will be the work of architects. But with the exception of those places where they have been granted the licence to do what they yearn to do – to start from zero – architects have less influence than they believe.

The places where those accidents don't occur are salutary. The places where architects indeed had the opportunity to start from zero. We think of Bath's crescents and circuses, of the successive Edinburgh new towns, of the exiled Polish court's rebuilding of Nancy. At a higher level, Ledoux's Arc-et-Senans and Le Corbusier's l'Unité d'Habitation both instruct us in what genius is. The roof of l'Unité is a transcendent work: it is as though Odysseus is beside you. In a few gestures, it summons the entirety of the Mediterranean's mythic history. It is exhilarating and humbling, it occasions aesthetic bliss. It demonstrates the beatific power of great art, great architecture.

Artifice ... Seaside, Florida. Photograph: Nik Wheeler/ Nik Wheeler/CORBIS

L'Unité is absolutely atypical. So are the other places I mention. They are the exceptions to the rule that planned towns, tied towns, new towns, garden cities, garden villages, communist utopias, national socialist utopias, socialist utopias, one-nation utopias, comprehensive developments and wholesale regenerations that lurch between the mediocre and the disastrous. From Letchworth to Marne-la-Vallée, from New Lanark to Welwyn – the first provincial town in Britain, incidentally, to develop a serious smack habit – from the Aylesbury estate in south London to Seaside, Florida, from cuteness to high modernism, from beaux arts to new urbanism. It doesn't matter what idiom is essayed, it is the business of attempting to create places that defeats architects. Architects cannot devise analogues for what has developed over centuries, for generation upon generation of amendments. They cannot understand the appeal of untidiness and randomness, and even if they could they wouldn't know how to replicate it.

New buildings are simple: imagination and engineering. New places are not. It seems impossible to achieve by artifice the parts with no name, the pavement's warts and the avenue's lesions, the physical consequences of changed uses, the waste ground, the apparently purposeless plots.

It shouldn't be impossible. One cause of this failure is architects' lack of empathy, their failure to cast themselves as non-architects: architect Yona Friedman long ago observed that architecture entirely forgets those who use its products. Another cause of failure is their bent towards aesthetic totalitarianism – a trait Nikolaus Pevsner approved of, incidentally. There was no work he admired more than St Catherine's College, Oxford: a perfect piece of architecture. And it is indeed impressive in an understated way. But it is equally an example of nothing less than micro-level totalitarianism. Arne Jacobson designed not only the building, but every piece of furniture and every item of cutlery.

At macro-level, a so-called master planner will attend to the details of streets, avenues, drop-in centres, houses, offices, bridges. The master planner is almost certainly an architect, even though planning and architecture are contrasting disciplines. There are countless differences between a suburb and, say, a shopping mall in that suburb. We are all familiar with the hubristic pomp that often results when actors direct themselves. Appointing architects to conceive places is like appointing foxes to advise on chicken security.

The human ideal is to revel in urbanistic richness, in layers of imperfection. I got sick of Rome when I worked there: too much perfection, too constant a diet of masterpieces – the lumbering, sod-you-ness of Basil Spence's British Embassy was peculiarly attractive. The only town in the Cotswolds that attracts me is Stroud, where the tyranny of oolitic limestone is ruptured by brick and slate.

The overlooked can only survive so long as authority is lax. When authority goes looking for the overlooked, the game is up – as it is today in the Lea Valley in east London. The entirely despicable, entirely pointless 2012 Olympics – a festival of energy-squandering architectural bling worthy of a vain, third-world dictatorship, a payday for the construction industry – occupies a site far more valuable as it was. It was probably the most extensive terrain vague of any European capital city. The English word "wasteland" is pejorative, lazy and more or less states that the place has no merit – so why not cover it in expressions of vanity?

In praise of decay ... a view of the River Lea, east London, before its regeneration for the 2012 Olympics Photograph: Dan Chung for the Guardian

The development has, regrettably, proved to be an almost caricatural illustration of the chasmic gulf that exists between the needs of writers and the aspirations of architects. A writer, at least this writer – and I am hardly alone – sees entropic beauty, roads to nowhere whose gravel aggregate is that of ad hoc second world war fighter runways, decrepit Victorian oriental pumping stations, rats, supermarket trolleys in toxic canals, rotting foxes, used condoms, pitta bread with green mould, polythene bags caught on branches and billowing like windsocks, greasy carpet tiles, countless gauges of wire, flaking private/keep-out signs that have been ignored since the day they were erected, goose grass, shacks built out of doors and car panels, skeins of torn tights in milky puddles, burnt-out cars, burnt-out houses, abandoned chemical drums, abandoned cooking oil drums, abandoned washing machine drums, squashed feathers, tidal mud, an embanked former railway line, a shoe, vestigial lanes lined with may bushes, a hawser, soggy burlap sacks, ground elder, a wheelless buggy, perished underlay, buddleia, a pavement blocked by a container, cracked plastic pipes, a ceramic rheostat, a car battery warehouse constellated with CCTV cameras, a couple of scraggy horses on a patch of mud, the Germolene-pink premises of a salmon smoker, bricked-up windows, travellers' caravans and washing lines, a ravine filled with worn car tyres, jackdaws, herons, jays, a petrol pump pitted and crisp as an overcooked biscuit, a bridge made of railway sleepers across duckweed, an oasis of scrupulously tended allotments.

That's what I see: layers of urban archaeology. It's what painters such as Carel Weight and Edward Burra would have seen, what George Shaw and Julian Perry still see. A site of richness and multiple textures which feeds curiosity. It is obviously decaying. But decay, as anyone who has watched meat rot knows, possesses a vitality of its own. Such vitality is infinitely preferable to sterility and stadia.

What an architect sees, blindly and banally, is not richness and severality. But, rather, something that is crudely classified as a brownfield site, that is tantamount to being classified as having no intrinsic worth. It is a non-place where derivative architecture can gloriously propagate itself with impunity. A brownfield site is a job opportunity, a place where the world can be physically improved. The architectural urge doesn't acknowledge the fact that it'll all turn to dust.

 This is an edited extract from Museum without Walls by Jonathan Meades, published 28 September by Unbound at £18.99.