In January 1914, Aston Webb, architect of the Victoria and Albert Museum, the facade of Buckingham Palace and parts of Birmingham University, had a dream. He told the London Society, a group of architects and others concerned with the improvement of the capital, how he was transported a century forward to the year 2014, where he saw “a beautiful sylvan line practically all around London”, with “a certain amount of open spaces, pleasure grounds” set aside by “town planning schemes”. He called it a “green belt”.

As futuristic fantasies go, it was prescient. London is indeed surrounded by a green belt, as are 13 other urban areas in England and 10 in Scotland, zones that forbid new development, except, as the policy wording has it, in “very special circumstances”. They occupy 13% the total land area of England, compared with the 10% that is urbanised.

Almost anyone you talk to on the subject agrees that the green belt is one of the great successes of planning, anywhere in the world. It has prevented the interminable exurbia, the light smearing of development over landscape that you get in the United States and many other countries. But it also has costs. It stops cities expanding, which had previously done so for centuries. It contributes to the scarcity and cost of decent homes in large parts of the country. It encourages bizarre and wasteful patterns of commuting. It often fails in its original aim of providing accessible recreational space for city dwellers. It is enforced with a rigidity that makes little sense, except as a sign of mistrust.

It would be naive to think that the country’s housing problems would solved just by handing tracts of the green belt to developers. Something more is required, which is the ability to plan positively, to create new places as well as protect old ones, a skill this country had until relatively recently. It is not a small thing to rediscover this art, but then neither is the green belt, nor the current issues of housing. The green belt arouses strong passions, but the debates around it are about something still larger, which is the ability of a country to act together in a shared endeavour, or to subdivide into competing interests.

The fact that it is named in the singular, although there are many green belts, indicates its status as an idea, even an ideal, as well as a place. It is part of English, if not British, national identity, protected by the shade of William Blake. It is brandished at party conferences and makes tabloid headlines, with frequent references to the “concreting over” of a green and pleasant land. It is used to justify many things; when, for example, towers are approved along the banks of the Thames, it is partly on the (specious) grounds that doing so protects the green belt. It has the power to strangle the careers of conservative politicians who question it. As one campaigner tells me: “It is the only universally popular part of the planning system.”

Yet it is being questioned. The libertarian thinktank the Adam Smith Institute has argued that, if a strip merely half a mile wide were shaved off the London green belt, 800,000 new homes could be built. Defenders of the green belts, such as the Campaign to Protect Rural England (CPRE), are accused of ignoring “those who are marginalised and condemned to poor housing”. The London Society, the group to which Aston Webb told his dream and which was instrumental in creating the green belt, has commissioned research from the commercial real estate company Colliers International, which says that we “need to move away from the idea that the countryside is a sacrosanct patchwork of medieval hedgerows” and towards the recognition of “housing as a need to be met in locations with appropriate environmental capacity”.

A particularly vocal critic, Professor Paul Cheshire of the London School of Economics, has written that “the unstoppable damage they do to societal fairness, housing affordability, the economic efficiency of our cities, even the environment, is devastating”. He argues that new houses in Britain are 40% more expensive than the Netherlands, which is more densely populated, and that the green belt is at least partly to blame. Business secretary Vince Cable recently tossed away the nation’s golfing vote by calling for homes to be built on golf courses, apparently inspired by the recent discovery that Surrey, the most green belty of counties, has more space dedicated to hitting small white balls with sticks than it has for homes. Cable’s was a bold statement, reminiscent of Fidel Castro’s and Che Guevara’s 1961 decision to build national art schools on the luxuriant links of the Havana country club. He also argued that most people would rather have greenery “outside their own house than sitting 20 miles away where they have no access to it”.

England’s green belts. Graphic by Pete Guest for the Observer New Review

It is said that green belts are not fulfilling their original aims of providing recreation for city dwellers. Dwellers of Toxteth or Hackney are not especially likely to make the journey to their nearest green belts, rather than Sefton or Victoria Park, or to countryside further afield. Rather, says Cheshire, the idea has been “used as a form of exclusionary zoning by the more affluent residents who had got there already”. It is a device, in this view, for rich dwellers of green belts, who enjoy proximity to both cities and countryside, to stop anyone else sharing their idyll. He argues that green belts force commuters to live outside them, and jump over them daily, some travelling from as far away as Norwich to work in London.

It is also claimed that much of the green belt is not all that green, but includes such things as scrap yards and gravel pits. Cheshire questions the environmental quality of the intensive agriculture that is the largest single use of green belt land, and cites the National Ecosystem Assessment of 2011 as saying that such land “has no environmental benefits at all”. Suburban gardens contribute much more biodiversity. It is argued, finally, that other designations, such as areas of outstanding natural beauty, national parks and sites of special scientific interest do some of the job of protecting what is really important about the countryside, which makes green belts less vital.

We can only guess how different the debate would be if the green belt were still known by its original name, the green girdle. It was coined in relation to London by Conservative members of the London county council, Lord Meath and William Bull, in 1901, and was proposed in various versions by several others in coming years and decades. A key supporter was Patrick, later Lord, Abercrombie, one of the founders of the CPRE and creator of the Abercrombie plan, which would guide the development of London and its surroundings after the second world war. From the late 1930s to the 1950s, legislation and policy gave local authorities progressively more power to declare what were by then called green belts, first around London and then in the rest of the country.

A consistent theme was that a green belt should be “a healthful zone of pleasure, civic interest and enlightenment”, or “a great communal estate – secure for all time to the use and enjoyment of the people of London”, or should “protect its inhabitants from disease, by providing fresh air, fresh fruit and vegetables, space for recreation and contact with and knowledge of nature”. After the Great War, proponents cited an occasion when 540 out of 1,000 potential army recruits were found to be too unhealthy to serve: the recreational opportunities of a green belt would make the urban worker more fit to fight. After the last war, with its demand to dig for victory, the agricultural importance of the green belt was stressed.

Jonathan Manns, of Colliers International, also describes how in changing from girdle to belt it increased in size. Its earliest advocates were thinking of a strip a quarter of a mile wide, which it was later suggested might thicken in places to a mile or two. Duncan Sandys, the Conservative minister who promoted green belts in the 1950s, envisaged a width of seven to 10 miles. In the event, local authorities liked the idea so much that it can reach 35 miles. Manns argues that nimbyism was a powerful motivation from early on: in 1960, it was observed that most of Surrey was being designated as green belt “with the object of ensuring that if London’s population overleaps the green belt, as it’s clearly doing, the emigrants shall alight, say, in Hampshire or Sussex, rather than in Surrey”.

According to these narratives, in short, a place conceived for the benefit of environmentally deprived city dwellers has been appropriated by the well-off property owners who live within it, who want everyone else, as a Daily Telegraph campaign on the subject almost puts it, to “get off our land”. It is also questioned whether a part-haphazard construct should be sacrosanct, as if it were a Magna Carta in geographical form.

In all this debate, and despite its celebrity, the exact nature of the green belt – or belts – is elusive. If you live in or near one, you probably know well that you do; if you don’t, you might be vague as to where they are. The inner boundaries of green belts are often clear – knife-sharp boundaries where buildings simply stop – but the outer edges can be invisible. From one field to another, the designation ends.

As might be expected of something so large and widely distributed, it is many things. It includes places of great beauty such as Box Hill in Surrey, and parts of Lancashire and Yorkshire, but also straggly farms near Heathrow and various industrial uses. Students at the Royal College of Art, led by the architects Charles Holland, David Knight and Finn Williams, have produced an impressive series of maps of the metropolitan green belt around London that show such things as the distribution of landfill sites, Travellers’ sites, branches of Waitrose, crime (mostly low), income (mostly high) and political allegiance (quite conservative). “One of the main characteristics of the green belt,” says Holland, “is that it has no main characteristics.” Its nature is sufficiently vague as to serve whatever political rhetoric is projected on to it.

Take Amersham, Bucks, a hybrid of idyll and unease. It is an historic settlement miraculously accessible, as the green belt’s early proponents might have hoped, from the London underground, from where you can easily walk through its not-sprawling periphery into fields and woods and an 18th-century park landscaped by Humphry Repton. Some of this countryside is delightful; some of it so highly processed that it is like walking through the sliced bread shelves at Tesco, which is what presumably it will soon become. A ceaseless thrum of bypass traffic pervades and the threat hovers of the High Speed 2 rail line, which is due to charge through the area, it being a paradox of green belt policy that it is easier to build major infrastructure in it than houses.

Or go to Knutsford, outside Manchester, Elizabeth Gaskell’s Cranford, now in the constituency of George Osborne, where the green belt has largely done its job of keeping the shape of a fine old town. One walk out of it takes you through postwar housing that seems to have been permitted before the green belt was decided, past uses important to the town such as sports clubs and allotments, towards dullish fields along a road hostile to pedestrians. A more rewarding route is into Tatton Park, a magnificent landscape attached to a stately home, which is preserved by the National Trust more than by green belt laws.

Then there is Grenoble Road, Oxford, currently a front line in struggles over the green belt, as Oxford city council and Magdalen College want to build on some fields they own, in the face of fierce opposition from the area’s CPRE and other local groups. The road is wide and fast, with aggressive traffic. On one side is the stadium of Oxford United, its car parks and leisure complex, plus, behind high hedges, the two-storey houses of the Blackbird Leys estate, which from time to time attracts the attention of the national press for its joyriding, keying of cars and other crimes.

On the other side are fields, hedges, clumps of trees as in a landscape painting, but also electricity pylons, with a sewage station and an electrical substation nearby. Thanks to the road, access from homes is not pleasant. This is the sort of niceish-but-not-wonderful landscape that leads many to ask what harm could come from nibbling off a bit of it for housing. It is not, however, the territory of Jag-driving nimbies of legend, anxious to preserve their property values and exclude people different from themselves.

Nor does this description fit the representatives of the CPRE I meet here and in other locations in the Oxford green belt – Kidlington, a charming old village extended at one end by less charming postwar estates, where there is talk of further expansion, and Cumnor, where a plan for 200 homes on what looks like a particularly inappropriate site has been, for now, beaten back. They include the former headmaster of a secondary school, a retired civil servant and a retired business startup adviser.

Urbed’s snowflake-shaped concept for a garden city which it applied to Oxford. Photograph: PR

One of them, Alan Lodwick, says he feels “ disenfranchised and disempowered” in the face of the real money, “the barristers, property developers, planning and property consultants and the landowners they represent, who can throw vast resources into making sure they make even more money out of the [green belt] land that they own.” Another, Michael Tyce, says that the fields off Grenoble Road “are doing exactly what the green belt is supposed to do – it is serving its purpose as a green lung”. City-dwellers do use it for recreation, he says, and, indeed, some walk by as he speaks.

Tyce and his colleagues contest the figures of the strategic housing market assessment, or SHMA, the device by which local authorities try to determine the number of new homes they need. They say that the case for development around Oxford is based on a false premise, which is that it is an area for growth of jobs and business as well as homes. There is virtually full employment in Oxford, so why not attract work to places in the north that need it, which, says Tyce, used to be government policy? “Now the policy seems to be to bring people to the south, but it has never been put to the test. There has been no consultation.”

Together with Helen Marshall, director of Oxfordshire CPRE, he argues that the green belt is “actual countryside that is well-loved and well-used by local people”, and that the danger is of making the county “into one big sprawl”. “I don’t think you’ll find anyone who wants that. I think most people would rather it continue as a rural county, rather than become Wolverhampton.” Alan Lodwick, in Kidlington, makes a further point, which is that the green belt exists also to protect the character of towns and cities and that the character of Oxford, with its compact medieval centre, is based on its being the size it is now.

A recurring theme of conversation is the congestion now in and around Oxford – waits of over an hour on rush-hour roads, a bus system with nowhere to expand. More homes, they say, will only make it worse. I stand in another ho-hum field outside Kidlington, the premises of a molecular genetic company off to one side, and ask again the question that keeps coming up in this debate – would it be so terrible to build on it? Lodwick explains that the issue is not just about this field, but the cumulative effect of many fields being developed, such that Oxford and its surroundings become completely different places.

Oxford has particular significance in the arguments about green belt. Danny Dorling, professor of geography at Oxford University, has said that the scarcity and cost of housing could easily cause the university “to lose its top position… the growth of knowledge-based industries would also suffer if the area is unable to provide junior staff with homes they can afford, and services such as education for their children. The area could ossify and end up like Santa Barbara, a place where only the rich can live.” Like the losing cox in a boat race, Oxford looks nervously at Cambridge, which is finding it easier to expand and to provide homes for university staff and sites for technology companies attracted by the university.

Oxford also features in the study that recently won the £250,000 Wolfson economics prize, which asked: “How would you deliver a new garden city which is visionary, economically viable and popular?” The winning entry, by the planning and urban design practice Urbed, argued that the best way to create successful new communities was to graft them on to the “rootstock” of existing cities, to take advantage of their infrastructure and establishe a sense of place. This would require taking “confident bites” out of the green belt. Most of Urbed’s essay concerned an imaginary town called Uxcester – a modified version of York – but it chose Oxford as real-life test case. David Rudlin, of Urbed, endorses the importance of green belts, but says that in this city “it has become a tourniquet”.

Urbed propose a “snowflake” plan, with new sub-centres radiating from the old centre, leaving green spaces between them. The new neighbourhoods would have exemplary tram systems, would be designed to encourage walking, and for every acre given to development, another would be given to publicly accessible open space – Tatton Parks, if one were to be really optimistic, rather than those dull fields outside Knutsford.

Urbed’s inspirations are both English garden cities of the early 20th century and much admired recent housing in Hammarby Sjöstad in Stockholm and in Vauban, near Freiberg, Germany. Essential to their idea is the huge difference in value between agricultural land and land with permission to build. At present, the difference turns into a lottery-sized jackpot for landowners, once they get permission, and profit for builders, who then have a strong motivation to build as cheaply as possible. Urbed wants some of this differential to pay for higher quality homes, infrastructure and open spaces.

The idea of garden cities, or new towns, has always been closely linked to that of the green belt, with both sharing the belief that it is good for people to live near nature. The first campaigners for a green girdle, or belt, were inspired by Ebenezer Howard, who in 1898 first proposed the idea of garden cities. The same Lord Abercrombie who helped found the CPRE, and successfully advocated the green belt, was also behind the new towns planned in Britain following the second world war. Although often sneered at, they successfully fulfilled their social aims while also achieving a financial surplus.

In Abercrombie’s world, constriction went with creation and prevention with provision. Building was stopped in some places so that it could be done well in others. Since Margaret Thatcher’s government the positive half of this formula been abandoned, leaving only the power to negate: she wound down the new towns’ development corporations and took their proceeds. She also set the scene for a battle between halves of conservatism that is still raging, between free marketeers and those who think their party is about conserving something, between the Adam Smith Institute, which sees green-belt policy as a restriction to be challenged like any other, and the Daily Telegraph, which wants builders’ Hands Off Our Land. The people who lose out in this row are citizens who want a decent home.

In Thatchers’s time, her environment secretary, Nicholas Ridley, proposed relaxing the green belt, but was shut up. More recently, the planning minister, Nick Boles, didn’t seem quite sound on the subject, which may be why he is now the ex-planning minister. His successor, Brandon Lewis, was so quick to slap down Urbed’s proposals that he can barely have had enough time to read them. The government didn’t want urban sprawl, he said, and anyway was supporting a planned garden city of “up to” 15,000 homes in Ebbsfleet, Kent.

Which is depressing. Urbed’s work is a serious attempt to address an ever-growing and scandalous problem, which is the low quantity and abysmal quality of new homes, to which Ebbsfleet alone will make a minuscule difference. Nor is Urbed proposing “sprawl”. It deserves a fair hearing from a government that cannot offer anything more than a slow seepage of overpriced developers’ housing. Urbed’s snowflakes cannot address the nation’s housing issues by themselves – such things as the urban developments championed by Richard Rogers are also important – but the scale of the problem is such that every good tool available must be considered.

Box Hill in Surrey, in green belt land. Photograph: Alamy

Until abolished by the current government, there was a laudable target of building 60% of new housing on “brownfield sites” in cities. This, of course, left the not-small proportion of 40% to be built in the countryside. There is a large chunk of countryside not designated as green belt, which would, if plans such as Urbed’s are to be ignored, take the brunt of the 40%. Which makes little sense, as much of this countryside has a greater environmental value than green belt land. Development there, being further from cities, encourages longer journeys by car.

Standing in Grenoble Road, it’s possible to see how Urbed’s thinking could help a place like this. In every respect except the preservation of green space this spot is a planning disaster – the dominating road, the cut-off housing estate, the shoddy buildings around the stadium. If, at the cost of losing some of the nearby fields, these problems could be addressed, while also allowing more people to live close to nature, might that not be a good thing? “Rubbish,” says Michael Tyce of the CPRE. Why? Because “Urbed’s scheme relies on destroying existing protections”, which would set a precedent. There would be no stopping further encroachment and the gaps between the dendrites of the snowflake would be filled. “Even though it’s not what Urbed wants, that’s what will happen.” In other words, the boundaries of the green belt, set partly arbitrarily 50 or 60 years ago, can never be altered, ever, no matter how the world changes, because we can’t trust politicians and developers not to rush into any breach in the rules.

The trouble is, on the question of trust, Tyce is right. There is pitifully little evidence, on recent form, that new development will be anything like the well-considered settlements Urbed proposes. When the property consultancy Colliers International seem to disparage medieval hedgerows, they don’t inspire great confidence in whatever alternative they might propose – nor do they make convincing Jacobins, or champions of the common man. There need to be more examples of what could be, which will require a greater mind than Brandon Lewis’s to make it happen.

Yet this gap in trust has to be bridged. The costs of not doing so include housing that is more scarce and expensive than it need be, of worse quality, and more badly located, such that people have to make long commutes across green belts to their nearest cities. The current reality, as Urbed’s Rudlin puts it, “is that development is dribbling out in all the wrong places”. Meanwhile, pressure on cities that try to grow within existing boundaries will eventually become intolerable. Consider, for example, that London used to be admired as a city of houses and gardens, yet in new developments gardens have almost disappeared, which is directly attributable to the city’s green belt. Other popular cities will face the same problem. For all these reasons, it is no longer good enough to insist that green belts must, at all costs, never change.