Let me make one thing clear from the start: this is not going to be one of those hard-hitting Lynn Barber-type profiles. I am a huge William Gibson guy, and I have been since I was 13 years old. Earlier this year I took my devotion to the extreme lengths – the excessive lengths, let’s all admit – of publishing what was essentially a William Gibson tribute novel, like a small-town Black Sabbath playing an evening of wonky covers at the pub. So when I tell you that Gibson picked me up in his car from my hotel in Vancouver, so I didn’t even have to get a taxi – and indeed that he was impeccably hospitable and generous over the ensuing 24 hours – you might think: “How awfully convenient that this lifelong hero of Beauman’s, whom he is clearly desperate to be friends with, should happen to be a really nice guy.” Yes, I agree, it is awfully convenient, but sometimes things just turn out that way.

The effect that Gibson’s debut novel Neuromancer had on me when I read it as an adolescent – comparable to a defibrillator applied directly to my forebrain – was the same effect that it had on speculative fiction in general when it was first published in 1984. People who wouldn’t normally be seen reading a book about a hacker in the future who sneaks on to a space station to help a computer turn into a god – and that’s a lot of people, perhaps including you – they made an exception for Neuromancer, because it was just too brilliant to ignore. Neuromancer originated a subgenre of science fiction called cyberpunk, which later withered away, redundant, because cyberpunk had become the condition of the real world. Gibson could therefore be credited with anticipating the information age, with the result that for the last 30 years he has regularly been pressed into service as a prophet. And yet you’re not giving his fiction its due if you just score it on the accuracy of its predictions like a six-horse accumulator. Call me biased, but I think the most interesting thing about a novelist is the inherent quality of his or her novels. And Gibson’s – 11 so far – are some of the best and most singular novels by anyone writing in English.

Gibson picked me up that day so that we could drive about 20 minutes northeast to Dusty Greenwell Park, a margin of grass overlooking Vancouver Harbour. Dusty Greenwell Park is not the most arcadian of retreats. Down the slope, past a snarl of blackberry bushes, is Canada’s largest container grain-loading facility, where trainloads of malt are disgorged into containers and then trucked off to the shipping terminals. Trains thunder past for minutes at a time, almost drowning out conversation, and a grain elevator, discoloured by rust and moss and pigeon shit, blocks part of the view of the North Shore Mountains across the bay. We visited because this was the setting of one of the climactic scenes of Gibson’s 2007 novel Spook Country, in which “the narrow park leap[s] into shivering, seemingly shadowless incandescence” as a spy flees a patrolling helicopter.

Spook Country is not the book that Gibson, 66, is currently promoting. That is his superb new novel The Peripheral. But The Peripheral is set partly in the American south in the 2030s, and partly in London in the 2100s, so it would have been difficult to find a directly relevant excursion within the Vancouver city limits.

In the book, it’s a sort of transtemporal Skype running on a mysterious Chinese server that accounts for those two eras becoming entangled in a single story. By bringing them face to face, Gibson was addressing the tendency of any given generation to assume that “the inhabitants of the past are hicks and rubes, and the inhabitants of the future are effete, overcomplicated beings with big brains and weak figures. We always think of ourselves as the cream of creation.” Although he generally defers to his unconscious when asked about his creative process, Gibson allowed that this book may have had its origin in a news story he read about a midwestern Christian militia called Hutaree whose members were arrested in 2010 for an alleged conspiracy to kill local police officers.

“The thing that struck me was that several very young teenagers had been left to fend for themselves when all of the adults had been taken away to jail,” he said. “I started imagining being one of those kids and how I would understand the world.” The setting also fed on Gibson’s own upbringing in small-town South Carolina, the film Winter’s Bone and the HBO series Deadwood, about a lawless town during the Gold Rush. “I wanted the equivalent of the city slickers, from a very different world, turning up in Deadwood. Initially I thought the other world would be New York or Los Angeles. But at some point I realised that could be the future.”

By now we had found a bench and we were both sipping from little cans of Boss Coffee that we’d picked up at Fujiya Japanese Foods on the way to the park. This made the outing rather Gibsonian in ways that neither of us realised at the time. Boss Coffee is manufactured by Suntory, and I’ve since learned that a lot of the malt in the containers before us would have been bound for Suntory’s beer and whisky breweries in Japan. Huge Japanese conglomerates with tentacles in the most unlikely places; the hypercomplex networks of globalised commerce; really good espresso – these have been quintessential Gibson tropes ever since Neuromancer.

By and large, I like Gibson now for the same reasons I liked Neuromancer when I first read it: his books are really cool – and I don’t mean “cool” as in “hip” or “chic”, I mean “cool” as in “awesome” or “rad”. That sounds like faint praise, because 13-year-olds have no taste and “cool” is not a respectable term of critical approbation. But what I mean by “cool” is that Gibson presents you with something new – a technology, a garment, a building, a scheme, an expertise, a power structure – and this new thing is burnished with so much imagination and lyricism and attention to detail, and so much of the noir and the gothic and the postmodern all at once, that it’s electrifyingly exciting just to contemplate. He does this several times on every page, and intersperses some old junk that he did not invent, and then connects all this stuff up so unpredictably that the connections are themselves exciting. And before long the connections are dense enough that he has a world, and he lets you shadow a small cast of reprobates as they pinball through every echelon of that world.

The amazing thing is, Gibson is cool in the other sense, too. Imagine a science-fiction novelist in his mid-50s, with grown-up children, living in a suburban house in Vancouver, sitting down and saying to himself: “I am going to create a female character 20 years younger than me, and she’s going to be one of the coolest characters in modern fiction, so cool that a decade later people on fashion message boards will still be discussing the personal style of this imaginary person, so cool that a Japanese company, Buzz Rickson’s, will do a brisk trade in £400 replicas of her bomber jacket.” (That monologue is a dramatic reconstruction, not a direct quote.) In almost any other hands, it would have been laughable and embarrassing, but Gibson came up with Cayce Pollard, the formidable protagonist of his 2003 masterpiece Pattern Recognition (who works as a professional “cool hunter”).

‘Sometimes, I’m delighted to be alive in the actual 21st century that I was trying to imagine back in 1985’: William Gibson with writer Ned Beauman, a fan since the age of 13. Photograph: Hubert Kang

Of course, when Sonic Youth have already written songs based on your novels [“The Sprawl” on Daydream Nation], it must give you a bit of confidence in that respect. Also, he told me: “I came originally from a place that had a serious cool deficit, so I really became a nerdish tabulator of cool. As a writer, I wanted to know how it worked because I judged it to be one of the only engines of global commerce that I had any talent for deconstructing.” Today, Gibson is one of those rare figures, like Patti Smith or David Lynch who, as they enter their late 60s, do not resign from cool but rather merge with it and become part of its specifications. You wouldn’t immediately know it to look at him: although he knows more about men’s clothes than any other novelist I’ve ever met, he seems to feel that at his stage of life it would be unbecoming to draw too much attention to his appearance. So no Lynchian pompadour.

What I didn’t appreciate about Gibson’s work when I was 13 was how his instinct for tabulating and deconstructing also functions on a scale so grand that, in his own words, it “takes the temperature of the present”. Here, futurology is a form of satire, as of course it always has been. Like Don DeLillo or David Foster Wallace, Gibson can identify the fluxes and fevers and fissures that animate an era, and in his new book he does this with London.

He once wrote that the enigma of London “has troubled me quietly and constantly, and I have returned there more repeatedly, and more determinedly, than to any other world city”. The Peripheral is the fourth of his novels to be set at least partly in the capital, and by far the most trenchant in that respect. In the early 22nd century, London is eerily deserted, and most of the characters we meet are either Russian oligarchs or working for oligarchs.

“One of London’s roles today seems to be to be the natural home of a sometimes slightly dodgy flight capital,” Gibson observed. “It’s where you go if you successfully rip off your Third World nation. And there’s a huge industry of people there who are just there to facilitate that move, if you’ve got the readies. I can’t help but think that that will eventually come back to bite somebody’s ass, although it may well be your grandchildren’s.” Gibson told me that when he visits London, he’s struck by the extent to which overseas money has warped the fabric of the city, but even more so by “the denial of my lifelong Londoner friends. Some people just don’t seem to see that there’s anything happening to it, even though it seems to me to be such a radical change. It amazes me when people argue: ‘Oh, it’s only happening in that neighbourhood, and if that’s no longer fun we’ll just move.’ I thought that was what the developers wanted you to do so you can gentrify the next bit.”

A further quality of The Peripheral’s future London which cannot have represented a crippling expenditure of imaginative resources is that the city finds itself captive to an unchecked financial establishment. “I was talking with Nick Harkaway [author of Tigerman] in his back garden a couple of years ago and he started explaining how the guilds of the City work.” In particular, Harkaway told him about the unelected offices of the Lord Mayor and the City Remembrancer, who can be mistaken for figureheads but, in fact, have extensive powers. “To this day I still don’t know if he was pulling my leg, totally making it up, or giving me a very accurate description of it.” (“Well, basically it’s true,” Harkaway clarified by email. “I’d just come off a weird and wonderful trip that the City of London PR office organises every year for writers.” If they ever read The Peripheral to the end, that office may have second thoughts about the image benefits of getting novelists interested in this kind of thing.)

Then there’s the other half of the book. Gibson described The Peripheral to me as “a sort of two-headed dystopia in which it’s impossible to decide who’s got the worst deal”. Just like the London of the 2100s, the rural American south of the 2030s is thrown off balance by the recently landed – but here it isn’t foreign oligarchs, it’s veterans returning from yet another American war, scorched and futureless, with nothing much to do but play around with surveillance drones and attack drones that are as plentiful as chickens.

It has been a constant of Gibson’s work that we live in a sort of military surplus society, where military technologies and philosophies are forever leaking into civilian life. When he watched the coverage of the Michael Brown protests in Ferguson, Missouri, where police in full camouflage drove armoured vehicles through the streets of a small town, it felt almost like an adaptation of his own work. “Ferguson,” he told me, “is very much the implicit rural America of Virtual Light,” his fifth novel, which was published in 1993 and set in around 2006. “I haven’t found anybody who’s been able to convince me that it’s necessary to arm those small-town police forces with Iraq-grade equipment. So it’s got to be good for somebody. It’s definitely good for the contractors who are paid handsomely to build those vehicles. There are chains of intergovernmental commerce there that I know nothing about, but given human nature I’m a bit suspicious of.”

Gibson does not, in fact, “know nothing about” such contractors, because they were among the villains of Zero History. Regardless, he was able to counterpose a more benign but equally plausible (and Gibsonian) conspiracy theory about American police forces buying armoured vehicles from the army. “I’ve heard that in some small towns in the United States, buying that sort of equipment was a very effective way to address the potential dangers of militia movements,” such as the aforementioned Hutaree. “Those movements can be trouble – they can turn to crime. But if you start an official county militia and you get a tank, they’ll come and join your militia, and then you start training them. ”

As the sun began to set over Vancouver Island, we adjourned for dinner at a strip-mall izakaya. The following morning I went over to Gibson’s house in the West End, where he lives with his wife Deborah, so that we could talk more about The Peripheral. At one point, Netherton (the protagonist in the far future) is forced to explain to Flynne (the protagonist in the near future) exactly how 22nd-century London came to be so depopulated. The scene is an enactment of something science fiction cannot be expected to do, which is sit you down and tell you what’s going to happen in the future. For as long as Gibson has been a writer, he has had to remind people not to regard him as a clairvoyant. When he wrote his early books, he told me, “I was conscious almost to the point of embarrassment of the extent to which this effect of futurity was completely dependent on collaging the present. But eventually I decided that that’s what all predictive science fiction does.”

Indeed, at one point, Flynne is seen eating a cronut, the chimerical pastry invented last year by New York chef Dominique Ansel. The cronut has only existed for about 18 months, so anyone who took their own visionary powers too seriously would not risk discredit by betting that it will endure another 20 years. But Gibson feels no sheepishness that the characters in his early books, which are set in the mid-21st century, communicate by fax machine and telephone booth, and by including the cronut in The Peripheral he seems almost to be mocking the idea that anyone would look to his fiction for a meticulous report on the future. “Either the cronut has legs, or it’s so ephemeral that readers who are fully unfamiliar with the word will just think it’s part of my overheated imagination. One of the pleasures of writing imaginary futures for me is my consciousness of the vagaries of how they might age. The only thing we can really be sure of is that they’re obsoleting for us on the page. Sometimes, I’m delighted to be alive in the actual 21st century that I was trying to imagine back in 1985.”

And yet not all of The Peripheral is the present dressed up as the future. Some of it is the future wearing no disguise. Explaining the reasons for London’s sepulchral quiet, Netherton tells Flynne that a lot of apocalyptic dominoes fell, but it began, of course, with the climate: “People in the past, clueless as to how that worked, had fucked it all up, then not been able to get it together to do anything about it, even after they knew, and now it was too late.” This is not futurology as satire, any more than a long-range weather forecast is satire. James Lovelock, after all, has predicted that 80% of humans will perish by AD2100, and the same figure can be found in the novel. If Gibson gives us catastrophe, it’s because right now he finds catastrophe plausible. “Since I turned in the book, every time I open my news feed I see something and wince and think: ‘I didn’t even factor that in.’ It doesn’t look good. I’m reluctant to say it, because people my age have always said we’re going to hell in a handbasket. I’ve heard it all my life from some writers who were then the age I am now. So I think: ‘Am I due yet? Have I started to do that?’ But it really doesn’t look good…”

The doorbell rang. A photographer had arrived to interrupt our casting of the runes, and until I took a taxi to the airport we talked mostly about specialised jackets.

I understood perfectly well before reading The Peripheral that our planet is beginning to roll down a very steep slope. And yet there was something terrifying about finding it here, perhaps because I’ve been absorbed in Gibson for most of my lifetime. His previous dystopias, with their overflowing slums, feel jolly by comparison; here, even Gibson’s hackers and mercenaries and television personalities are bereaved by climate change. It’s as if Gibson’s work is a city, and I have lived in that city since I was a teenager, and now that city is being drowned before my eyes. Before I left, Gibson admitted that his own story “surprised me with its grim matter-of-factness. I wasn’t fully aware of the implications, at the start. Though in my personal model of writing fiction, one never really is.”

All the same, this shouldn’t discourage anyone from reading The Peripheral, which is not, in fact, a remotely grim book. First of all, books that are as frantic with imagination as Gibson’s books, as frantic with the appetite to see what happens to us next, cannot be grim; second, Gibson’s famous weakness for happy endings has not been entirely suppressed here; and third, the details of our fate are mostly confined to that one scene towards the end, when Netherton, our descendant, explains what happened before he was born. As we read it, we ought to be like Flynne, who sits under the oak in her front yard and listens without hysterics as she hears the story of a world in which “everything, however deeply fucked in general, was lit increasingly by the new.”

To order The Peripheral by William Gibson (£18.99) for £14.99, go to bookshop.the guardian.com. He is speaking on Neuromancer: 30 Years On at the Guardian Book Club on 24 November. Tickets £15 (£12 members). See membership.theguardian.com