Walking around the stuccoed streets of Kensington and Chelsea today, you are in for a surreal sight. The marching rows of doric columns, pedimented porticoes and dentilled cornices that define these imposing ranks of wedding-cake mansions have been joined by an unlikely addition to the classical architectural vocabulary.

Poking up at regular intervals, thrusting outwards from their moulded openings as if performing a salute to passers-by, are lines of angled conveyor belts. Slowly rumbling away, they reach high above the trees, pouring a continuous stream of rubble into the cradles of awaiting skips. You would be forgiven for thinking that the residents of the royal borough have established a kind of coal-mining cottage industry. Or maybe they're digging for gold?

"It is an absolute disgrace," says one elderly resident, out walking her freshly coiffed miniature poodle in between the rows of hoarded-off skips. "It feels like they've turned Kensington into a war zone."

The reason for all this quarrying is not the discovery of a coal-rich seam beneath the Wrenaissance streets, but the local enthusiasm for subterranean development. Over the past four years, this local authority alone has granted planning applications for more than 800 basement extensions, refused 90, and has a further 20 outstanding. It is the most densely populated borough in the country, with no room to build outwards, and no permission to build upwards – so the only way is down.

But this desire for digging isn't to everyone's liking. Last week a furore erupted when plans were released for a four-storey basement beneath a 19th-century schoolhouse in Knightsbridge, for Canadian former TV mogul David Graham.

Tripling the size of the property, this gargantuan pleasure cave would house a ballroom and swimming pool, with hot tub, sauna and massage room, as well as 15 bedrooms, seven bathrooms and 20 toilets, plunging deeper into the earth than the height of neighbouring homes.

"These plans are absolutely monstrous and unnecessary," said one neighbour, the Duchess of St Albans. "It's just absolute greed. No one needs that much space. Quite apart from that, the commotion is going to be dreadful."

"This is totally out of keeping with the relatively small size of other houses in the area," agreed a spokesman for the nearby Milner Street Area Residents Association. "Why should we all suffer just so one man can indulge his fantasy?"

Such fantasies are not restricted to this one man alone. The past five years have seen sprawling underground leisure lairs excavated across west London, from Knightsbridge to Belgravia, Fulham to Notting Hill. They contain playrooms and cinemas, bowling alleys and spas, wine cellars and gun rooms – and even a two-storey climbing wall. It is leading to a kind of iceberg architecture, a humble mansion on the surface just the visible peak of a gargantuan underworld, with subterranean possibilities only limited by the client's imagination.

"Houses in this part of London are trophy asset purchases," says Peter Preedy, associate director of residential property at Jones Lang LaSalle. "People don't want to move out, so they have to find a way of bringing everything they want into their homes. These mega basements are not about increasing the value of property – they are very personal things, which might in fact prove difficult to sell on."

One of the most personal plans – which itself set the precedent for this recent burrowing frenzy – dates back to 2008, when Foxtons founder Jon Hunt received permission for the most audacious basement of all, a colossal cellar to trump even the swankiest Beverley Hills crib.

Having bought a palatial villa on Kensington Palace Gardens, the most expensive street in the country, he was not to be outdone by his neighbours. Lakshmi Mittal's nearby "Taj Mittal" already featured an underground complex of Turkish baths and a pool lined with marble from the same quarry as the Taj Mahal. A few doors down, property mogul Leonard Blavatnik had bought up three former Russian Embassy buildings and since added an underground swimming pool, gym, private cinema and extensive garaging.

In a surreal competition of keeping up with the Joneses, billionaire-style, Hunt went one step further into the realms of fantasy. He proposed to dig a 22m-deep hole beneath his garden to house a tennis court, pool and gym, as well as a private museum for his collection of vintage Ferraris. The cavernous chamber would be illuminated from above, through the glass floor of a glistening rooftop infinity pool.

"We were really taking the piss," laughs Ademir Volic, of Volume 3 Architects, who dreamed up the proposal for Hunt, referring to the fact that such a large-scale extension was at that time, before the Royal Borough of Kensington and Chelsea issued supplemental planning guidance on subterranean developments, a simple matter of making a £150 householder application fee [see footnote].

"We analysed the planning laws and realised that they cover everything about the surface of the ground, but nothing beneath it. There was nothing whatsoever that could stop us from drilling all the way down to the south pole."

"Of course, the council hated it," he recalls. "But we had a barrister, and the policy would not cover their reasons for refusal, so they had to grant permission – and it has triggered a whole avalanche of stuff ever since."

Hunt's project has since been mired in legal battles with the neighbours – not exactly surprising, with the Indian High Commissioner on one side and the French Ambassador on the other – and has yet to break ground. Prince William and Kate Middleton, soon to be neighbours over the gilded fence in Kensington Palace, will be breathing a sigh of relief: the project was of such a scale that it would have required a specialist in horizontal digging. And what kind of builder was lined up for the job? None other than the same contractor that built the channel tunnel.

While Hunt's cavernous car museum might be at the extreme end of the scale, and never likely to see the light of day, Volic argues that digging down is often the only answer to improving existing properties, given the restrictive planning system above ground in the UK.

"We live on a little tiny island, where we all like the idea of having our own houses with a front and back garden," he says. "But our Georgian and Victorian stock is so inflexible, frozen in time. We're selling this city as a forward-looking metropolis, yet we can't change a single window in a conservation area. Everything has to be hidden underground."

He also argues that these basement developments serve a useful structural function: "We are dealing with 200-year-old buildings, with all sorts of structural damage in them already. By adding a basement, we are underpinning the whole building, giving these things a proper foundation for the first time."

But the planning authorities are not quite so enthusiastic. "There may be some truth in saying that inserting a large concrete box beneath a house might stabilize it," says Derek Taylor, head of development control at the Royal Borough of Kensington and Chelsea (RBKC), and the man being faced with the barrage of ever-bolder basement dreams. "But there is a counter argument, that if the neighboring buildings aren't also stabilised, then they might well part company from it."

This is precisely what happened, to residents' dismay, in Kensington's Palace Gardens Terrace this summer, when Goldman Sachs boss Christoph Stanger started digging beneath his £7m property to create a playroom for his children.

After initial excavations, the house began to subside, pulling its neighbours down with it. Cracks appeared within the adjacent basements, and the facades sunk to such an extent that door frames shifted and people were trapped inside.

"We had to call the contractors out six times in three weeks to open our own front door," said a retired neighbour. "One lady who lives in my block was stuck in her flat because she couldn't get the door open from the inside. I had to come and barge it loose."

Subterranean pools are a favourite feature of luxury bunkers. Photograph: Sarel Jansen/Licorice Design

Nor did Stanger's 160-year-old house escape unscathed. Ten tonnes of roofing material was removed from the four-storey property by emergency building teams, after a pillar holding up its portico cracked through.

On top of neighbours' worries over structural damage, and the noise and inconvenience caused by the building process, concerns have also been raised over how all this emerging underground city will affect the capital's ability to cope with heavy rain. Vast swathes of impermeable concrete beneath the surface can prevent rainwater soaking away and increase the risk of flash flooding.

In response to the unbound torrent of applications for basement extensions, RBKC commissioned a report from Arup's geotechnical engineering group in 2009, which led to the first ever Supplementary Planning Document on Subterranean Development. Arup's research showed there was little to worry about in terms of drainage, the main problem being the disruptive effect of building sites on neighbours.

"Planning law is entirely based on the use and appearance of land," says Derek Taylor. "But basements are hidden, and they don't change the land use. So it all comes down to one issue: the nuisance of construction work."

The borough is currently refining its regulations, in discussion with neighbouring Hammersmith & Fulham and Westminster, which will likely see the addition of dust and noise-reduction requirements, as well as a possible levy on basement construction.

So is it all worth it? Property experts are beginning to sound sceptical.

"Many people who have done such extensions assume that the value of adding this space is proportionate to the rest of the house," says Oliver Hooper, director at high-end buying advisors Huntly Hooper. "But due to its limited use, the value diminishes the further you go down, while the costs of doing so increase."

And the problem remains that spaces below ground are always going to be dingy and confined to particular functions. As Tom Tangley, partner at Knight Frank, says: "If your typical London terraced house is 20ft wide, and you dig out a double basement, all you are ending up with is a deep, narrow corridor. It's very difficult to ventilate and light – so what's the use?"

Still, he does admit a soft spot for a property on their books behind Kensington Square, currently on the market for £15.95m, which features a swimming pool whose floor rises up to become a ballroom.

But the tide is beginning to turn, according to Lindsay Cuthill, a director at Savills estate agents: "When this trend began five years ago, there were all sorts of amazing media rooms with surround sound and banked seating, designed for friends to come round," he says. "But people are realising they're not using them like that. Friends aren't coming round. They're probably great if you're 13 years old, but it seems they have limited appeal beyond that."

It's a description that can't help but conjure visions of lonely oligarchs and forlorn sheikhs, sitting alone in their secure basements, multimillion-pound entertainment laid on for an audience of one. Perhaps there is something about the geotropic, burrowing urge that betrays a kind of deep-seated introspection – a desire to dig, to escape further from reality, to withdraw into a private fantasy world. A look at one of history's more eccentric characters would certainly suggest so.

William Cavendish-Scott-Bentinck, the fifth Duke of Portland, took over the estate of Welbeck Abbey in north Nottinghamshire when his father died in 1854, and began an eccentric reign of excavation. He dug an extensive web of tunnels beneath the grounds, connecting a series of underground chambers and stretching up to 24km – including a passage through which he could ride his carriage all the way to the station, hidden from public gaze.

Like today's residents of Kensington Palace Gardens, the Duke also had a subterranean ballroom constructed, 50m long and 20m wide, with a ceiling painted like a giant sunset. He even installed a hydraulic lift, capable of bringing 20 guests at a time down into the hall from the worldly realms above.

Yet not once did he invite people into his home. Never was the ballroom to host a dance. He would not even speak to his staff of servants, communicating only in written form through letterboxes installed in his rooms.

As Kensington's tycoons dream of digging ever deeper, retreating behind ever more layers of private facilities, they might pause to think how empty those echoing chambers will feel.

• This footnote was added on 21 November 2012. Following publication we were contacted by Ademir Volic, of the architecture firm Volume 3, who clarified that when he said, "We were really taking the piss" he was making an informal and off-the-cuff remark about a technicality over the type of planning application made, rather than referring to the design. Ademir Volic was referring specifically to the fact that at that time, which pre-dated the Royal Borough of Kensington and Chelsea's supplemental planning guidance on subterranean developments, planning permission for a large-scale extension could be granted following a £150 householder application fee. We are happy to accept this clarification and apologise for any confusion caused. In addition, Ademir Volic asks us to clarify that Volume 3 carries out its work in basement construction with the utmost concern for public safety, and that it has had a long and positive working relationship with the Royal Borough of Kensington and Chelsea, which has always sought to ensure the stability of neighbouring properties.