When you were a child, did you ever repeat some random word until it went strange in your mouth? Do you recall it growing heavy, as if by repetition it was acquiring the power of a spell?

"The biologist" (we're not told her name) has spent her life staring into puddles, into rock-pools, until "I had a sense that I knew nothing at all – about nature, about ecosystems." Now she is staring into the kind, bland eyes of her husband. He is newly returned from fabled Area X, unharmed, intact and utterly scraped out.

Area X is an abandoned and apparently unspoilt stretch of US coastline, held under strict quarantine by a mysterious government agency called the Southern Reach.

Into this place come the biologist and her colleagues: a surveyor, a linguist, and a psychologist. They are all women. And that is all. Sensitive readers will already have begun to feel their fingers prised loose from the edge of the swimming pool, when it turns out these explorers are unable to divulge their names. "Names belonged to where we had come from, not to who we were while embedded in Area X."

In Annihilation, the first part of an imaginatively marketed and beautifully produced trilogy (the other parts are out in May and September), the novelist and publishing entrepreneur Jeff VanderMeer sets out to create a lasting monument to the uncanny by revisiting – without embellishment, and with a pitiless focus on physical and psychological detail – some very old ground. An alien invasion site. Assimilative spores. An unfurling of promiscuous alien biology.

On the first page we are told that the women's enterprise is doomed. Their equipment is either nonsensical, or inadequate, or antiquated. Their training and instructions are sometimes vague, sometimes misleading. They cannot recall the moment they crossed into Area X, and they have no clear idea how they will leave. They cannot agree about what they are seeing (a shaft? a tower? a throat?) and three of them are all the while half-aware of being hypnotically manipulated by their team leader.

You enter Area X with them, thinking the uncanny must lurk in some particular spot. The lighthouse? The reed beds? The "tower"? Very quickly you spot your mistake, as a subtle, well-engineered wrongness turns up in every character, every deed, every observation until, at last, you find yourself afraid to turn the page.

The uncanny, by VanderMeer's measure, is not, and never was, a thing. It is, and has always been, the actual state of the world. Familiarity is a fiction we perpetuate through psychological necessity. The closer the nameless biologist comes to this realisation, the more she falls back on her scientific training – not in any petulant, pedantic way, but rather as a means of limiting the kinds of questions she needs to ask the world, and of her rapidly transmogrifying self.

Infected early in the book, she wonders if she has "changed sides", become more "X" than human. She decides the question is meaningless. "A religious or superstitious person, someone who believed in angels or in demons, might see it differently. Almost anyone else might see it differently. But I am not those people, I am just the biologist; I don't require any of this to have a deeper meaning."

From this self-destructively objective vantage point, there can be no "us" or "them", no threshold to cross, no home to flee to when all's done. Science is there to handle the uncanny, and the biologist's declaration near the end of the book – "Our instruments are useless, our methodology broken, our motivations selfish" – is anything but an expression of doubt. It is as stirring in its admission of human frailty and ambition as Beckett's "You must go on. / I can't go on. / I'll go on."

Where this story will end I cannot begin to guess. We are less than 200 pages in to the Southern Reach Trilogy by the end of this first volume, and already home is a distant memory, and an unreliable one, too: for who's to say that home was not always X at heart?