The London Olympics 2012 project grows vaster every day, spreading itself across the eastern metropolis like flowing lava, hardening into colossal new structures: Olympic Park, Olympic Stadium, Olympic Village. Is it a good thing? How can you even ask that? The Games brochure is sleekly triumphant about the value of their work: "The London 2012 Games are the catalyst for transforming 2.5sq km of land in east London," it reads. "What was once industrial, contaminated land has been rapidly transformed over the past three years."

Yes, but there's an enemy at the Park gates. On the edge of the Olympics site lurks its most sworn enemy, dedicated to its failure. Remember Hardy's poem, "The Convergence of the Twain", in which, while the Titanic is being built, the iceberg also grows in bulk out in the Atlantic? For most of the Olympic Park's gestation period, Iain Sinclair has been researching and writing Ghost Milk, a scorching 400-page diatribe against this and other "grand projects" that have inflicted metallic follies on the British landscape, only to see them fail and decay.

"The invasion philosophy of the Olympic Park strikes me as just like the invasion philosophy behind going into Iraq," he says, "or anywhere else that you blast into, put up the fence, establish the Green Zone, explain everything, put it all into this lovely eco-terminology..."

Sinclair has lived in east London since the early 1960s, and knows it backwards. He walks through it every day, studies its history, talks to its human relics and finds secret, mystical connections between its streets. He comes across as a living embodiment of pages 52-53 of the London A-Z, a crazily knowledgeable local historian with a shaman's grasp of strange energies, unseen ley lines, urban esoterica. And he's written his findings in books that hover between non-fiction and mythology: especially London Orbital, Lights Out for the Territory and Edge of the Orison. When he looks at Stratford and the Lower Lea Valley, he doesn't see an "industrial, contaminated land", he sees William Blake's Albion.

He's been hailed the nation's top "psychogeographer", though Sinclair is uneasy with the term. "Like everything, psychogeography has become a brand. The Situationists invented the word in the 1960s; it meant challenging the city and its energies, doing things like walking through Paris with a map of Venice, an absurdist activity. Now it can be referred to anything from Peter Ackroyd's novels to Will Self's [former] Independent column, neither of which have anything to do with the conceptual philosophy."

We meet in Victoria Park, which will play a big part in the Olympics; according to the website, it'll be "a Live Site with free-access big screens". Today, it looks a mess. The water has been drained while the Olympics work is going on. Wires and fences dominate the view. Three expropriated ducks stand glumly beside the Pavilion Café where I discover Sinclair, a tall, professorial guy in his sixties, thinking about the past.

"I remember interviewing an old barber around here," he says, "who told me the only leisure time barbers enjoyed was from 6-7am, when they'd come to this lake, take a boat and row for an hour, before going back to cut hair all day. When we moved here in the 1960s, houses were dirt-cheap and you could get the boats and the swimming pool across the road."

This is not just old-geezer nostalgia. Sinclair believes that the energies of the past – whether it's the spirit of Israel Zangwill, the Victorian Jewish novelist whose blue plaque is on a wall near us, or the rowing hairdressers on the lake or the Swedenborgians who once preached on the Park's central lawn – live on in the present. His own patch of Hackney, Stratford East, and Lea Valley is, to him, full of resonant ghosts and peculiar beauty, and he's watched with horror its invasion by Lord Coe's myrmidons. He and his wife walk round Victoria Park every morning, "and one morning there was this guy with a clipboard, asking for people's opinions about the Olympics. Two or three months later, the fences started appearing. Huge tranches of land were cut off. There was one little gate that led onto the Regent's Canal – it's been turned into a huge tarmac swoop, to make a better world for the biking banker on his way down to Canary Wharf."

Sinclair's first awareness that the Grand Project was about to invade his home turf was on 6 July 1995, when the success of the Olympic bid was celebrated by Londoners. "I was watching TV in a desultory way when everyone started cheering. I never thought they were going to get it, but I didn't realise there'd been a deal with Berlusconi – it was like a moment from The Godfather, when Tony Blair flew off to ask him the Big Favour. My heart sank with horror when I heard the news, because I had an inkling of what was coming, after our experience with the Millennium Dome."

OMG, the Dome. Remember the horror of 1999, as the biggest white elephant in a generation slowly rose in the east, the cost spiralled to £800m, nobody could agree what should go inside it, top brass resigned or were fired, and the opening night found hundreds of dignitaries stranded for hours on Stratford station? Sinclair does. He wrote a short, pungent philippic called "Sorry Meniscus", warning that it would be a permanent, useless blight on the Greenwich landscape. The Dome's failure to attract more than half the expected audience must have filled him with delight. And of course, from 2000 to June 2007, it lay dark and unattended, until it was reborn as the 02 Arena with a Bon Jovi concert. Wouldn't he agree, though, that it's now turned out quite successfully? Sinclair's eyebrows shoot up like McDonald's arches.

"Has it? You think?"

Well, I falter, it's a very popular music venue.

"Public hangings were popular once."

Public hangings and Beyoncé shaking her booty, I say, are hardly in the same moral universe.

"David Haye won the world cruiserweight championship in the 02 Arena and then couldn't get out of the car park," he claims. "He ended up having to walk home. Transport-wise, it's one of the world's worst bottlenecks. If the Jubilee Line isn't working, you're fucked. It's because they didn't put a bridge across from the Dome to Beckton, which could have eased things. Instead they've now got this proposed chair-lift. Everything now aspires to the condition of Blackpool funfair. Look at the Thames riverbank – shark tanks, Ferris wheel, fake sand."

Ghost Milk – the title means "an ectoplasmic substance that surrounds us here, made up of the memory of old films and photographs, through which I'm trying to access the past" – is full of conspiracy theories: that the Olympics razzle-dazzle is merely a pretext for ravaging the Lea Valley ("I noticed bankers and financial jackals sniffing about here a long time back. All this has been lined up for a generation"); that the whole project is a joint initiative of Government and major retailers ("At the end there'll be just this huge Westfield, and the Stadium will be renamed Westfield Stadium"); that the residential towers will be used, post-Olympics, as holding pens for asylum seekers; he even warns that the Park will be contaminated with radioactive thorium. You can try to argue back, or challenge Sinclair to produce evidence, but he bats away any scepticism and surges on with more evidence for the prosecution, more rubbish "projects".

"I came across this eco-park in Doncaster, a fantastic thing. If you went there by car, you were surcharged £5 a head because you were burning up fossil fuels, so everyone parked in the town, which became gridlocked. Nobody went to the site, they put on archery for kids on Saturdays, then abandoned it altogether. One security guard was posted by the perimeter fence. Grass and weeds started coming up. Now it really is ecological because it's become a wilderness. But millions of pounds have gone down the chute.

"You can see the same thing happening across the north of England, these millennial projects – how utterly futile and disastrous they were, a series of monstrous abandoned buildings. The National Centre for Popular Music in Sheffield. Huge museums without content, which soaked up money before they were closed.

"In Manchester, there's a place called Urbis, which looks like a huge glacier. People said, 'What is it?'. The council said, 'Well – it's not a museum, but it captures the spirit of the city'. In other words, it's nothing. They put on little bits and pieces of exhibitions, and then it folded. Now it's going to house the Football Museum that's already failed in Preston."

You sound, I say, positively gleeful about things going wrong.

Iain Sinclair was born in 1943 in Maesteg, south Wales, to a GP whose patients were local miners on the edge of the valleys, and a mother who had been his father's chemist, "a mixer of potions" – an appropriate calling for the mother of such a modern alchemist. He studied at Trinity College, where he edited the student magazine Icarus, and did postgraduate work at the Courtauld Institute, before joining the London School of Film Technique (now the London Film School). Influenced by American underground movies, he made 8mm documentaries, "in Brixton, in Spain, about Pamplona. Observational stuff, very pure." He wrote scripts for the original Conan the Barbarian, and (from sublime to ridiculous) Cliff Richard. He sold a 1967 documentary about Allen Ginsberg to Jonathan Miller's BBC arts show, Monitor. But spending time in the cutting-room didn't excite him ("it was such a civil-service bureaucracy") and he left to make his own films and "just do labouring stuff for a living for 10 years or so".

This led to an extended epiphany. Among all the anger in Ghost Milk, there's a little memory of place and people to which he constantly returns. It's called Chobham Farm, in Angel Lane near Stratford station, and was where Sinclair, then 28, worked in 1971 as a casual labourer. He describes the unpromising landscape around Stratford station as if it were a magic land. "It was very romantic working on these railway yards," he says, of the six months he spent transferring crates of talcum powder from Tilbury lorry to warehouse to smaller delivery vans, "though I didn't realise at the time I was part of an attempt to break the docks and the unions by bringing the containerisation business inland." It must have been a mortifying discovery for a left-wing stalwart like Sinclair – but it doesn't stop him mythologising about the squad of itinerant labourers that came and went, as if they were visionaries like himself, with his wool beanie hat and a copy of Dante's Inferno in his back pocket.

"But the landscape was magical," he says defensively, "because it still had old apple orchards that had run completely wild and abandoned forests, industrial ruins like the Watch Dial factory, lots of straggly-foresty fringe stuff you get in the lower Lea Valley, some great plants that fed on the toxic waste and mutated into strange, Triffid-like forms. We'd wander through there in our spare time at weekends." It's what he now calls Edgeland, "on the edge of the intensity and density of London as I knew it," a place about to be swallowed up by developers.

He's a most unusual anti-development activist. Rather than protest at City Hall, write to his MP, or picket planning departments, Sinclair accreted information from a thousand sources and shaped it into a long literary polemic, full of dazzling phrases (London's MI6 building is "the Aztec jukebox") and angry denunciation.