For the ordinary resident, there is much to fear and loathe in London: the scourge of 4X4s too wide for a normal parking space; the fit-to-burst property bubble; the Tesco Expresses that used to be pubs. Presently there is no greater totem of resentment than the humble basement.

Or in some cases, not so humble. The media is fond of stories about the "iceberg homes" of tycoons and oligarchs: already enormous houses under which three– and four–storey basements have been dug to house swimming pools and car collections. Even modest basement conversions under suburban terraces are a cause for alarm, if only because of their sudden proliferation. In certain neighbourhoods it's not unusual to see four or five being dug in the same street – boxy hoardings squatting on front gardens with conveyor belts carrying spoil over the pavement into a skip. In 2001 the borough of Kensington & Chelsea received 46 planning applications for basements; last year it received 450.

You don't have to travel far to hear stories of endless noise, dust and disruption. Even celebrities have begun to complain. Joan Collins told a Belgravia residents' magazine she found it shocking that "people are digging down to put in swimming pools and bowling alleys when they only live here for two or three months of the year". On his blog, Queen guitarist Brian May described his basement-digging neighbours as "selfish and brutish" and the piling rig they were using as an instrument of torture.

The problems don't stop at noise. When Andy King moved into his North Kensington home, permission had already been granted for next door's one-and-a-half storey basement excavation. "We inherited a party wall agreement," he says, referring to the standard written agreement aimed at protecting adjoining property owners in the event of damage. He was suspicious of the works from the moment digging began: "They were banging in piles, the street was sinking. You know something's wrong."

He didn't know how wrong until he returned from holiday to a load of missed calls. "The person staying in the house was trapped," he says. "They couldn't open any of the doors." His foundations had subsided – in places the walls had dropped a foot – requiring costly underpinning. He and his family had to move out for six months. "That was the beginning of three years of discovering what a waste of time that party wall agreement was."

Basement conversion under way … most complaints relate to the disruptive, noisy work rather than the finished product

There are more generalised fears about what will happen once whole swaths of London have been hollowed out below ground. The mega-basements of the super-rich have even spawned their own urban myth: that many of the JCBs employed in their digging could not practically be retrieved from the bottom of the giant hole, and were simply buried. It is said that newer, deeper excavations have begun to run across them on the way down.

"We've never buried one," says Stephen Merritt, managing director of London Basement, which has completed about 1200 basement conversions since 1994. "We don't actually use the mini-excavators that often. Most of our digging is done by hand."

Ian Hogarth, an architect whose own west London basement was featured in an episode of Grand Designs, also finds the story ridiculous, although he admits that an auger from one of the piling rigs that sank his foundations is still buried under the house. "It got stuck there," he says.

Some of the more elaborate mega-basements display an almost childlike frivolity. The Tory donor Edmund Lazarus recently applied for planning permission for a 16,000 sq ft mega-basement that would have included a 25-metre swimming pool, a cinema, a games room, a cigar room, a two-level gym, a catering kitchen and a yoga studio. It makes it sound as if he doesn't want to live in his house as much as wait out the apocalypse in it.

Welcome to our basement … Ian Hogarth and Claire Farrow. Photograph: Sarah Lee for the Guardian

Hogarth agrees that a certain impracticality can take hold. Some of his clients spend more time worrying about wine storage than living space. "They're very interested in the wine cellar climate," he says, "and then they put the nanny in this lightless fucking box."

He believes, however, that a lot of the anger about so-called iceberg homes is down to plain old envy. "I think there's a kind of 'what does he want that for? It's vulgar.' Why would anyone want to have a collection of 67 Rolls-Royces? I don't know. Rich people are weird." I remind him that his own basement contains a lighted dancefloor.

"It's a multipurpose playroom that happens to have an illuminated floor," he says, smiling. On the day I visit, there is an enormous pram sitting in the middle of it.

Whatever you think of the dancefloor, the Hogarth family home is a pretty cool building. Built on a tiny scrap of land at the end of a mews and backing on to a railway line, the above-ground portion of the property is a small and sympathetic extension of the mews row. Fully two-thirds of its square footage lies beneath the house and garden. But watching that old episode of Grand Designs, it's obvious the house took its toll on Hogarth and his wife, Claire Farrow: planning permission took eight months instead of the usual two; the project went £100,000 over budget; a giant piling rig had to be driven down the narrow cobbled mews where their already anxious – and in some cases angry – neighbours lived.

According to urban myth, many diggers, too awkward to retrieve, have simply been left, buried underground. Photograph: Alamy

One feature of the dig-down extension is that most of the planning objections are about the works, rather than the outcome. A completed basement doesn't look like anything from outside, but the digging of one is far more disruptive – for longer – than a comparable above-ground extension. "Much, much worse," says Hogarth. "There are no two ways about it. I've seen some that take two or three years. In some cases I don't know why – it may be that the neighbours have made it so difficult they've slowed it down. But most are doable in a year."

He doesn't accept, however, that there are major, unforeseen consequences to building so many basements, and insists that in any case, there aren't that many being built. "If you look at a map of London, and put the basements on it, it's not even a spotty dog," he says. "It's a man with a couple of pimples."

On the wall of the conference room of London Basement's HQ, located in a light industrial area in the western reaches of the capital, there is just such a map, covered in coloured stickers representing the company's completed projects. They resemble, if not a spotty dog, a man with severe acne scarring. "The orange ones are the older ones," says Merritt. "The green are more recently completed ones, and the blue ones are the wet ones."

Ah, the wet ones. Anyone digging down in London stands a fair chance of coming into contact with what is known as the upper aquifer: perched groundwater that sits on top of a deep layer of impermeable clay and flows through channels of sand and gravel. Hogarth's house is hard by the old bed of a "lost" Thames tributary called Counters Creek. It's not surprising that at one point his builders encountered a great deal of water. What's surprising is that when it comes to excavating basements, the presence of a great deal of water isn't a reason to give up.

"It takes a bit longer," says Merritt. "Looking at one in Sheen we did recently, that took four weeks longer than it would have if it were dry, because our conveyor belt gets clogged up. Digging wet clay out is a lot more awkward and time-consuming than nice dry sand." And the hole left behind, as you might imagine, tends to fill up with water. "The one in Sheen had three pumps going 24/7." How do all these wet basements stay dry? "Basements 15 years ago were all about blocking water – tanking," says Merritt. "And tanking would invariably fail at some point. These days we manage the water." Modern systems allow water to drain down a cavity membrane inside the foundation and into a sump, where it is pumped out. Hogarth's house has two pumps going, with a back-up for each in case of failure.

Part of Ian Hogarth and Claire Farrow's west London basement conversion, which featured in Grand Designs. Photograph: Sarah Lee for the Guardian

Neither Hogarth nor Merritt believe there are any truly insurmountable engineering obstacles to digging a basement. Merritt cites the Crossrail station in Canary Wharf – excavated to 26 metres in the middle of a dock. "If they can do that in London," he says, "they can do anything." The only real stumbling blocks are financial ones. Indeed, the main factors driving London's basement boom are economic.

"If you look in all those areas," says Merritt, pointing at his map, "you'll see the prices of houses are at least £1,000 a square foot. A basement, all in, is probably about £500 a square foot." A basement conversion may be dirty, disruptive, stressful and expensive (even a modest one will set you back £250K), but if you live in the right neighbourhood it's still cheaper than moving house.

"The focus, courtesy of the popular press, is on the six-car basement," says Hogarth, "but actually now, very modest little worker's two-up two-downs in places like Fulham are being done. There's just no housing stock."

A below-ground conversion by London Basement

Merritt concurs. Of the 30 or so excavations London Basement are undertaking this year, none goes down farther than a single storey, and most are dug to provide functional living space. "If you look on the website you'll see a few swimming pools," he says. "They look great on a website, but in reality they represent a very small percentage of the projects we've done." He's not in the business of creating "mega-basement icebergs"; he says there aren't that many on the go in London anyway. "As soon as somebody puts in for planning permission for a three-level basement it gets in the newspapers. Whether or not it gets built is another question."

If one were optimistic, one might argue that the basement-boom is self-limiting. It only makes sense in particular parts of London, where the prices are ridiculous, houses are small, and planning restrictions prohibit above-ground expansion. The economics don't really favour the iceberg home, where the third storey down will probably cost more to dig than it could ever be worth. If one were less optimistic, however, one might conclude that the financial imperatives of basement-building can only encourage those who would seek to cut corners on safety, structural integrity and the minimisation of disruption.

looks as if regulation is finally starting to catch up with the basement boom. Kensington & Chelsea recently denied permission for Lazarus's mega-basement out of concerns for the wellbeing of two trees in his garden. The borough is in the process of introducing a new planning policy that would limit excavation to a single storey in most circumstances, and restrict the footprint to 50% of the existing garden. Already it is illegal to dig under a listed building in Kensington & Chelsea, and a traffic management plan – an approved scheme for transporting spoil and equipment to and from the site – is now a condition in most boroughs.

The immediate effect of digging basements under London is still more basements. When one project in a street receives planning permission, several neighbours follow suit. "It's become an expectation, and normal," says Hogarth. "It's actually – with the conveyor belt and the hoardings and all that – it's not so bad." And when whole London neighbourhoods are thus undermined and underpinned, what then? Where will the groundwater go? How will it flow? A decade ago Kensington & Chelsea commissioned a study of the issue from engineers Arup, who realised that west London's unprecedented basementisation actually had a precedent: the City of London, where deep, near-contiguous basements have been in place for decades. They found the localised damming from all the subterranean construction presented no significant problems. The water just finds another way round.

Of course all the underground water in London is, to some extent, managed. Old rivers have long been imprisoned in sewers. The rising water table has been artificially depressed by a large scale "dewatering" programme since the 1990s. To stay dry all these new basements will require pumps, and back-up pumps, indefinitely. What happens if, at some point in the future, all this precious square footage loses its value, and these properties are no longer maintained?

"I think one thing is for certain," says Hogarth. "Grass will one day grow through our motorways. We're just passing through, and it will all become like a Mayan ruin."

• This article was amended on 20 August 2014 to correct the spelling of auger.