Urban exploration: a guide for the uninitiated. Urban exploration, urbex or UE is recreational trespass in the built environment. Among the requirements for participation are claustrophilia, lack of vertigo, a taste for decay, a fascination with infrastructure, a readiness to jump fences and lift manhole covers, and a familiarity with the laws of access in whatever jurisdiction you're undertaking your explorations. Archive and web skills are useful too, for acquiring the schematics and blueprints that will inspire and orient you. Among the sites in your sights are disused factories and hospitals, former military installations, bunkers, bridges and storm-drain networks. You should be content on the counterweight of a crane 400 feet above the street, or skanking along a sewer 10 yards under the asphalt.

The cultural origins of urbex would include, to my mind, Tarkovsky's Stalker, the fiction of JG Ballard, old-school mountaineering and caving, blasts of steampunk (there is a love of girders, rivets and brickwork), console culture (Bioshock), apocalypse dreams (from Planet of the Apes to The Road), the Mission Impossible films and (inevitably) Guy Debord and his situationist dérive – the randomly motivated walk designed to disrupt habitual movement through the cityscape. It's quite some gumbo. If urban explorers didn't exist, China Miéville would have had to invent them.

The scene has its subscenes. Just as certain climbers prefer granite to gritstone, and certain cavers prefer wet systems to dry ones, the explorers have their specialisms: the bunkerologists, the asylum seekers, the skywalkers, the builderers, the track-runners, the drainers. Most people start out in ruins, though: these tend to be the easiest sites to access, and the aesthetic payoffs – the pathos of abandonment, the material residue of inscrutable histories – are rapid. Ruinistas dig "derp" (UE slang for "derelict and ruined places"). Detroit was the world mecca for derp, until it became a city-sized version of Don DeLillo's "most photographed barn in America", and it was impossible to see it except through a haze of ruin-porn imagery: HDR stills of dusty ballrooms and atria, with artfully scattered detritus (detroitus) in the foreground.

The hero shot … Robert Macfarlane at an underground reservoir. Photograph: Scott Cadman

Along from the ruinistas come the adventurers, who are mostly out for the kicks. Photography is important to the adventurers too, they specialise in the "hero shot": the lone explorer seen from behind on the rim of a building or bridge, or heavily backlit (partly to preserve anonymity) and framed in a storm-drain or archway. Such images unmistakably have their origin in Caspar David Friedrich's icon of Romanticism, Wanderer Above the Sea of Fog (1818): the dark frock-coated traveller atop his peak, with the mists of unknowing spread out beneath him. Every modern-day mountain summit shot owes a debt to Friedrich's painting, and UE has absorbed and adapted the same image.

Then there are the self-styled "guerilla preservationists", deep into heritage theory, and genuinely committed to creating a coherent photographic and textual record of buildings that would otherwise crumble unnoticed until a developer arrived to raze all trace of them. Their archives are carefully curated on websites, their identities disguised with pseudonyms and firebreaks.

Up at the avant-garde of urbex are the infiltrators, the "real" explorers, who tend to be more stimulated by systems and networks than by single sites, and who cherish the challenge involved in accessing super-secure locations. Like climbers, infiltrators experience what Al Alvarez called, in his classic essay on climbing, "feeding the rat". The rat lives inside you, and itfeeds on fear. The more you feed the rat, the larger it grows, the greater its appetite – and therefore the more fear you must experience in order to sate it. Infiltrators run tracks in the brief gaps between trains, they take dinghies down storm-drains, they lift-surf, and occasionally they die – in ways that may strike you either as noble, or as liable for a Darwin Award, depending on your attitude to urbex.

The culture of urbex is mostly but not overwhelmingly male. Its politics are hard to simplify: libertarian in the main, fringed here and there with a Fight-Clubby anarchism, and in certain people aimed at resisting the rise of surveillance and the privatisation of urban space. Like all subcultures, it thrives on acronyms and slang. Security guards are "seccas". "The Fresh" is sewage. Manhole covers are "lids", and you "pop" them. Sleeping overnight in a site is "going pro-hobo". Certain terms have been imported from urban design: "Sloap" is Space Left Over After Planning. "Toads" are Temporary, Obsolete, Abandoned or Derelict Spaces.

Lightning strike. Photograph: Bradley Garrett

Urban exploration is international, with groups around the world, but it is too various in its motives and methods to constitute anything like a community. A code of honour is broadly adhered to: no criminal damage, no sueing anyone if anything bad happens to you. In the white sandstone under Minneapolis, digging teams work in shifts to open routes into sealed caves. In Toronto an explorer has bolted a pitch and abseiled into the vast tailrace pipe under the Niagara Falls. This year, Russian explorers are on fire, taking the practice to places – Dubai, Hong Kong – it's never been before.

Urbex is not for everyone. Let me put that differently: urbex is hardly for anyone. Participation is high in profile but small in number (perhaps 20,000 globally), and the thrills are niche. Not for urbexers the sturm und drang of mountains or the arid elegance of desert exploration. Their epiphanies are mucky, their metaphysics mephitic. The short-term risks are grim: drowning in sewage, falling from girders, gralloched by razor-wire, skewered on scaffolding. Longer-term dangers include respiratory problems from exposure to dusts and gases. I know, I know: why would you? Who would? It is a hugely strange scene, and – occasional claustrophiliac with an intermittent taste for decay that I am – I find myself rather gripped by it.

❦

I met Bradley Garrett at London Bridge early one afternoon. He said he had a great story for me, and he did.

"The bridge is hollow," he said, tapping his foot on a utility hatch two‑thirds of the way along. "There's a control room at the north end; if you get into that, you can cross the Thames inside the bridge. Come – I'll show you."

We took the stairs by the north end. Partway down, Garrett hopped over the stair-rail and began edging along a narrow skirt of masonry that stuck out from the bridge's side, 10 feet above a Sloap of concrete, ventilation hatches and aerials. He had his hands flat against the vertical brickwork, and perhaps half a foot's purchase on the skirt.

"Are you happy coming out along this with me?"

I wasn't. It had been raining, the masonry skirt was wet and angled, and I needed to be able to pick my children up from school without crutches.

"No matter," said Garrett, and hopped back over the rail. "We'll see it another way." We followed the steps until we were under the bridge. There was a steel door, secured with a hunky padlock. Garrett pulled a ring of keys out of his pocket, chose one, had the lock off in about a second, ushered me inside, and closed the door with a soft clang behind us.

"That's some bunch of keys you have there," I said. I flicked on a headtorch. We were in a control room. Zinc venting, ducts and technicoloured wiring lashed with cable ties. Two wall-mounted dashboards with switches and dials.

"So – if you follow this ducting south out of here, then you're inside London Bridge," Garrett said. "Keep going all the way over the river, and you reach a much bigger control room at the south end. Hit the exit bar on the emergency door there from the inside, and you can let in who you want. When we made a film about UE a few years ago, called Crack the Surface, that's where we held the premiere. We had 86 people, a generator, a screen, a projector, and a lot of beer. It was a great party!"

We slipped out and Garrett locked up. Two men in suits gave us puzzled looks but didn't break stride.

Victorian drain, south London. Photograph: Bradley Garrett

From London Bridge, Garrett took me on a haphazard walk through the City. He had climbed pretty much every major building we passed. He and other explorers have topped out the Shard four times, the Cheesegrater twice, the Lloyd's Building once ("many CCTV cameras, no response") and the Walkie Talkie building multiple times. The Gherkin went up before Garrett arrived in London, to his enduring regret. On the whole, he prefers mid-level structures to skyscrapers: "Something like the Shard has no relationship to the city. From its summit, you look down and London resembles a giant circuit-board. It all seems chilly and lifeless from up there."

Certainly, Garrett perceives the city like no one else I know. Seen through his eyes, it is newly porous, full of "vanishing points", "imperfect joinings" and portals – service hatches, padlocked doorways – that you wouldn't usually notice. The usual constraints on urban motion, whether enforced by physical barriers or legal convention, don't restrict him. The city's accessible space extends far down into the earth (sewers, bunkers, tunnels) and far up into the air (skyscrapers, cranes), with the street level only serving as a median altitude.

We stopped at the foot of the Walkie Talkie building, aka 20 Fenchurch Street. "Look at that," Garrett said. "There's a big wheelie bin pushed right up under the scaffolding. If you were staying over tonight, we'd come back here later on. Up on the bin, on to the scaffolding, drop down, and we'd be into the site. From there it's just a case of getting across no man's land, and into the stairwells and the inner core. Then we pelt up 34 flights of stairs to the summit."

The purple hoardings around the site carried creepy corporate feel-good mottos: "Rise To The Top Faster", "The Building With More Up Top" and "The View Belongs To Everyone". "Except the view doesn't belong to everyone," said Garrett. "It costs £25 per person to go up the Shard. This'll be the same. The least they could do is make it free. But they don't – so we take it for free."

Zeche Hugo, Gelsenkirchen, Germany. Photograph: Bradley Garrett

❦

Garrett (urbexer, academic geographer, blogger) is an extremely interesting man. He is also generous, unpredictable – a lot of fun to be around. I think he might be among the few genuinely fearless people of my acquaintance. Things inconvenience him (security guards, flesh-wounds, court cases), but as far as I can tell, nothing much scares him. He has thick‑rimmed black glasses, a goatee and moustache, and chin-length dark brown hair that gets banded back into a ponytail when action beckons. His speech mixes West Coast dude-isms with the gnarlier syntax of culturaltheory.

Garrett grew up in California. In 2001, aged 19, he co-founded a skateboard shop in the city of Riverside. He sold out to his partner two years later, and used the money to study maritime archaeology in Australia, then to start a "cultural resource management firm" in Hawaii. In search of some "seriously empty space", he moved back to northern California and began work for the US Bureau of Land Management, specialising in the archaeological heritage of Native American groups. He felt uneasy at the politics involved – and decided to become an academic geographer instead. He ended up in Britain with a studentship at Royal Holloway to study three marginal groups: neo-druids, mudlarkers and urbexers. The druids and the mudlarkers fell away (for the good, I feel) and UE became Garrett's exclusive ethnographic focus.

His research method was extreme and immersive. He spent four years embedded with a group of London-based explorers – "the scribe of the tribe" – as they enjoyed what he now describes as a "golden age" of UE. He took part in "more than 300 trespass events in eight countries with over 100 explorers". Among the results of his research were a doctorate from the University of London; a gallery of remarkable photographs; arrest by British Transport Police (BTP); the battering-down of his front door and confiscation of his computers, phone and passport; a court battle; a post-doc at Oxford; and now the book of his PhD, published as Explore Everything: Place-Hacking the City.

The book's style is volatile and its stories are extraordinary. It narrates "the rise and fall of the London Consolidation Crew (LCC), the UK's most notorious place-hackers", and Garrett's years with them. It might be imagined as a gonzo road trip rewritten by a committee comprising Margaret Mead, Edward Abbey and Dizzee Rascal. I wouldn't be surprised if film rights have already been optioned. The narrative of the book follows Garrett from noob (uninitiated) explorer to cutting-edge infiltrator – though he is careful throughout never to style himself as either champion or leader. Intercut with the helter skelter storytelling is heavy duty analysis of, among other subjects, the politics of UE, the affective role of photography and video, and the phenomenology of urban flow. Studding the text are dozens of Garrett's startling photographs. This combination of anecdote, image and exegesis gives the book a distinctive triple-tone that will not be to everyone's taste.

Saint-Sulpice church, VIe arrondissement, Paris, France.

After a dramatic prologue describing his detention by BTP (hauled from a plane at Heathrow while up in first class – Gordon Brown fumed at the delay), Garrett examines the emergence of urbex in the late-1970s, and details his own early forays into the scene. He earns the trust of the explorers who will become his key companions – only ever identified by aliases ("Gary", "Patch", "Winch", "Marc Explo"), with whom he learns the ropes and ticks off the London classics: Battersea Power Station, Millennium Mills.

Many adventures follow. Rumours are investigated. Tip-offs are pursued. Garrett and a female explorer called "Rouge" hear about a derelict Soviet submarine floating in the Thames near Rochester: a U475 Black Widow. They buy a kid's dinghy and paddle out after dark to the submarine. The dinghy nearly sinks, they're almost swept away by the current, then once aboard Rouge is almost knocked out by the sealing wheel of a falling hatch. When they do get off the sub it's low tide, and they have to mud-wade to safety.

One weekend, "Moses" proposes traversing the Forth Road Bridge from north to south: "The plan was mental and everyone loved it." They drive to North Queensferry, find an open hatch in one of the pylons, and climb an internal ladder to the upper girders. This was, as Garrett puts it, "serious edgework", especially when it begins to rain. "Start crawling really fast right fucking now!", yells one of the crew. Inexplicably, they all make it across.

The team head out across Europe, sleeping in derelict motels, scoping out site after site, getting "sleep-deprived, stinky and buzzing". Garrett hits America, climbing a Chicago skyscraper in a storm and gaining astonishing images of a city "bathed in black cloud and blue light … with lightning strikes crawling down from the clouds into Lake Michigan". In one jaw-dropping episode in the Mojave desert, he penetrates a "boneyard" of decommissioned aeroplanes, climbing over barbed wire, and then hiding in the landing gear of 747s and military cargo-carriers while security patrols pass by. "It was," notes Garrett drily, "a vast playground and a long night." In the sewers of Minneapolis, he and Marc Explo "charge headlong into a tiny stoop filled with raw black sewage like molasses, a den of faeces packed with cobwebs and little white subterranean spiders, which we fended off with nothing more than a stick and a bottle of André champagne until the fumes almost took us down" (I re-emphasise: urbex is not for everyone).

After two years, Garrett's group merged its efforts with another team to form the London Consolidation Crew, which soon became known for its audacity and ambition. The intensity of their activity increased ("dusk was another dawn"), and the rats inside them grew: "Our thirst for the adrenaline rush of getting away with things became insatiable." The crew settled on a "holy grail": to reach all the "ghost stations" of the London Underground, and complete "a photographic survey of the disused parts of the LU". Consulting "pre-war Tube maps" and "new worker track maps" they confirm 14 stations as "ghosts": "the crown jewel of the system was Aldwych … the most difficult was going to be the British Museum". Stepping on to the tracks instantly raised the stakes. Trespass is not a criminal offence in the UK unless you prevent someone from going about their normal business. This is not the case on railways, however, where bylaws permit criminal prosecution (with a six-month statute of limitation).

Garrett's attempts to reach the ghost stations form the most controversial episodes of the book. These actions led eventually to his arrest and the forced dispersal of the LCC (bail conditions currently prevent them from communicating). His trial is ongoing. Transport for London, apparently fearing copycatism, have recently threatened his publisher, Verso Books, with legal action over the publication of "illegally obtained information" in Explore Everything.

Forth Rail Bridge, Edinburgh. Photograph: Bradley Garrett

❦

As will be obvious, urbex is not without its critics. Detractors style it variously as naive, fetishistic, self‑heroising and, well, criminal. Its brand of subvertionist play can easily resemble Scooby-Doo-ish japery (don't get stuck in the vent-shaft, Shaggy!) or wilful trouble-making. And there is rich possibility for insensitivity to those people who are compelled to live their lives in a context of dereliction: the thousands of homeless who inhabit the storm-drain network beneath Las Vegas, for instance.

Garrett is familiar with these lines of attack and – as a good ethnographer must – gives them due consideration in his book. Much depends on the motives you ascribe to UE, but the explorers themselves are mostly poor at self-analysis, preferring to fall back on T-shirt catchphrases: "Live on the edge", or "Do epic shit" (an imperative that can be read two ways).

Garrett acknowledges the difficulty of generalising a motive for urbex (or, as he puts it, "reifying a co-ordinated explorer ethos"), but he personally celebrates it as a form of activism, which "recodes people's normalised relationships to city space", and creates temporary "regions of misrule". Or – as Foucault in a militant mood might have said – they aim "to disrupt dominant hegemonic spatial control through tactical urban infiltration". His book ends with a manifesto-climax that readers will find either rousing or riling: "Wherever doors are closed, we will find a way through. Wherever history is buried, we will uncover it. Wherever architecture is exclusionary, we will liberate it."

Perhaps. It's still unclear to me exactly how urbex will roll back privatisation or resist surveillance culture. It may even do the opposite (more cameras, more "seccas"). Successful access campaigns have tended to be large-scale movements rather than lone wolves, the most famous instance being the Kinder Scout Trespass of 1932, though I suppose a counter-example might be the fascinating optical trespasses of the contemporary American photographer, Trevor Paglen, whose ultra-long-lens cameras peer into the black-ops sites and classified landscapes of the American security complex, making visible what the state keenly wishes to keep unseen.

❦

Late that night, I met Garrett again at Blackfriars Bridge, at low tide. Two of his friends joined us: Scott and Alex. Our plan was to lift a manhole cover and drop into the Victorian sewer tunnels through which flows the Fleet, one of London's "lost rivers". Garrett wanted to show me the Fleet Chamber, a vast Bazalgettian structure near the outfall into the Thames. We had waders and headtorches ready to go. Garrett was mildly concerned about flow levels in the Fleet, due to the day's rain.

"We'll get in there and have a look. If it's running too high, we'll just turn around and come out."

"I need to make the half-midnight train from King's Cross," I said. "We'll get you there," replied Garrett. "In fact, if you want we'll walk you north up the tunnels, and pop you out of a manhole just by the station." I liked the thought of taking the tube rather than the Tube back to King's Cross. But I pitied whoever sat next to me on the way home.

Garrett and I had already tried and failed to get under London earlier in the day. We'd accessed the network of steam-tunnels that runs beneath the Barbican, pushing through a door in an underground parking lot, but had been seen almost immediately and left at speed (Garrett jangling his magic keys coolly, me sweating through my scalp with nerves). And our Fleet Chamber adventure was also to be frustrated: dozens of workmen in hi-vis jackets were swarming around the Fleet manholes, putting in a late-night maintenance shift on the tunnels.

"Fuck! I have a positive relationship with Thames Water," Garrett said. "We've told them where dozens of leaks are. But it's not good enough to share the Fleet with their workers. Plan B it is, then."

Plan B was a huge 19th-century subterranean reservoir, buried under a north London park and now drained of its water. Garrett knew of its existence but had never been there; Alex had scoped it out once. He declared its awesomeness as a space.

We caught the Tube north. The other three reminisced about a big urbex meet in Antwerp. "It was mad! People were running everywhere," said Scott. "The sewers were full. The train tunnels were hopping." "I fell into a hole and gashed my leg to the bone," said Garrett. He rolled up his trouser leg to show me the scar. It was long, wide and shiny. "But I was totally twisted on dope, and didn't notice. We got out and went to a bar for something to eat and drink. I started feeling dizzy, and realised my boot was full of blood."

In the park, in the dark, we got kitted up in silence. There was a bank to climb up, and some fencing to roll under. Alex and Garrett located the lid, and used two drain keys to pop it and pull it away with a screech. One by one we climbed down a utility ladder into an antechamber, from which a rickety staircase led into the belly of the reservoir proper. We descended the staircase, headtorch beams probing the blackness, whistling and hooting at what we saw.

It was indeed an awesome space, possessing the extreme functional elegance of major Victorian infrastructure, and as beautiful in its way as the Roman cisterns at Micenum and Constantinople. A cast-iron overspill sluice curved down near the entrance, its cup perhaps 12 feet in circumference. Dozens of brick archways extended in series away from us, with the runnels between them holding wide rungs of still water. The iterated forms of the arches and the reflections of the water created the illusion of infinite regress.

We walked the reservoir end to end and side to side, our voices and the splashes of our passage echoing. Above us in the shadows hung the vaults of the ceiling itself, hundreds of thousands of yellow-brown bricks. Fine white silt clouded in the water at our footfalls. At the far end we sat down, took stock, smoked. Garrett set some music going; a drum and bass track called "Stresstest". That seemed right. Then he and the others set up a photograph.

"All noobs need a hero shot. You be the model, Rob." They backlit the arches with LED panels, and set my headtorch to red-beam.

We got out just before midnight. There were scattered clouds, underlit pink and orange by the city's glow, with stars visible between them. Three men moved through the trees to our east, scanning the grass with golden torch beams, looking for something.

"Macfarlane Hero Shot.jpg" came through by email early the next morning from Garrett. I was in thigh-waders rather than trousers, and a hoodie rather than a frock-coat, but the homage was unmistakable. There it was: Caspar David Friedrich Redux – Traveller Below A Sea of Bricks (2013).

• Robert Macfarlane's The Old Ways: A Journey on Foot is out from Penguin. Explore Everything: Place‑Hacking the City by Bradley L Garrett is published by Verso next month.