One peculiar satisfaction of being a reader is seeing an author you have followed for a long time finally break into the big time. VanderMeer has been a favourite among aficionados of New Weird fiction for more than a decade, exploring his fascinations with fungi, subterranean spaces and decay across half a dozen books. But with his Southern Reach trilogy – Annihilation, Authority and Acceptance, all released in 2014 – he has finally hit the bestseller lists. And with good reason. This trilogy is a modern mycological masterpiece.

In the first volume, a team of four scientists, identified by occupation rather than name (biologist, anthropologist, psychologist and surveyor), are sent to explore the mysterious "Area X", a wilderness in the American deep south. Eleven previous expeditions have failed, with mysterious influences driving the explorers to suicide, madness and cancer. This 12th trip is narrated by the group's biologist. She descends a spiral staircase into a subterranean structure she insists on calling a "tower", and finds a mysterious message written on the walls in letters made of some sort of fungoid vegetable growth. Expedition members appear to have been hypnotically conditioned by the Southern Reach, the organisation that sent them. But this does them no good, assuming it was ever supposed to. They go bonkers, or start glowing yellow, and are killed off one by one. Remnants of previous expeditions are found, including the journal kept by the narrator's now-deceased husband. There's a weird beast in the wilderness, and a strange being called the Crawler seems to be stalking the group. But what makes this book so remarkable is less what happens in it, and more its tense, eerie and unsettling vibe. Creating such an atmosphere is a balancing act: on the one hand, the writer must not destroy the mood with too much brute explanation; and on the other, he must not alienate the reader by being too annoyingly oblique. VanderMeer hits exactly the right balance, like a gymnast on a beam – albeit a creepy gymnast who's been infected with occult fungal spores and is starting to glow yellow.

The second volume, Authority, takes us inside the Southern Reach organisation itself. Its leader, Control, is struggling with the repeated failure of its missions into the slowly expanding Area X. This is a novel built around interviews with baffled returning explorers, unhelpful video evidence and the paranoid internal politics of the organisation. Of the three novels in the trilogy, it's the least successful. It may be that you find hordes of strange rabbits or phones apparently creeping across a roof to be the stuff of nightmares; but for me it had too much exposition and too little genuine creepiness.

But where Authority drags a little, Acceptance is much more on the button. (Or, we might say, the button mushroom.) The biologist from Annihilation and Control plus two others stage a new expedition into Area X, and their adventure recaptures the eerie momentum of the first volume. We get answers – but not to everything, and the ones we do get are not facile. If I'm sounding evasive, I can't help that. Giving too much away would spoil not just this novel but its two predecessors, and that would be a shame. Your best bet is to ignore the publisher's slightly gimmicky decision to release three novels in one year, and treat the trilogy as a single piece of work. If you start reading it, you will read right through in short order. You may also have nightmares.

Science fiction is sometimes seen as obsessed with shiny, futuristic technological equipment; but in fact nowadays the genre tends much more towards VanderMeer's Poe-like unease with a pulsing, microsporidic natural world. Mind you, VanderMeer writes much better prose than Poe ever did; and Acceptance in particular is full of beautiful descriptions of the natural environment. Structurally it hops a little distractingly between its various points of view, and the shortness of the chapters sometimes creates a choppiness that is at odds with the book's longer-burn Lovecraftian spookiness. But this choppiness is accidental: one of the things this novel is doing is playing non-linear games with time, wrongfooting any preconceptions we have been foolish enough to build up from the first volumes. Finding a way satisfactorily to pay off so much mysteriously tense apprehension is no small challenge for a writer – and VanderMeer manages to avoid banality and opacity both, and generates some real emotional charge while he's about it.

In interviews, VanderMeer has claimed the landscapes of the Southern Reach novels are autobiographical, based on hikes he has taken along the coast of northern Florida. All I can say is: if that's true, remind me never to book a holiday there. This is genuinely potent and dream-haunting writing. VanderMeer has arrived.

• Adam Roberts' novels include Yellow Blue Tibia. To order Acceptance for £10.39 with free UK p&p call Guardian book service on 0330 333 6846 or go to guardianbookshop.co.uk.