Recently I set out to explore my lifelong relationship with science fiction, both as reader and as writer. I say "lifelong", for among the first things I wrote as a child might well merit the initials SF. Like a great many children before and since, I was an inventor of other worlds. Mine were rudimentary, as such worlds are when you're six or seven or eight, but they were emphatically not of this here-and-now Earth, which seems to be one of the salient features of SF. I wasn't much interested in Dick and Jane: the creepily ultra-normal characters did not convince me. Saturn was more my speed, and other realms even more outlandish. Our earliest loves, like revenants, have a way of coming back in other forms; or, to paraphrase Wordsworth, the child is mother to the woman. To date, I have written three full-length fictions that nobody would ever class as sociological realism: The Handmaid's Tale, Oryx and Crake and The Year of the Flood. Are these books "science fiction", I am often asked. Though sometimes I am not asked, but told: I am a silly nit or a snob or a genre traitor for dodging the term because these books are as much "science fiction" as Nineteen Eighty-Four is, whatever I might say. But is Nineteen Eighty-Four as much "science fiction" as The Martian Chronicles? I might reply. I would answer not, and therein lies the distinction.

My desire to explore my relationship with the SF world, or worlds, has a proximate cause. In 2009, I published The Year of the Flood, the second work of fiction in a series exploring another kind of "other world" – our own planet in a future. The Year of the Flood was reviewed, along with its sibling, Oryx and Crake, by one of the reigning monarchs of the SF and fantasy forms, Ursula K Le Guin. Her 2009 review in this paper began with a paragraph that has caused a certain amount of uproar in the skin-tight clothing and other-planetary communities – so much so that scarcely a Q&A session goes by at my public readings without someone asking, usually in injured tones, why I have forsworn the term science fiction. Here are Le Guin's uproar-causing sentences:

To my mind, The Handmaid's Tale, Oryx and Crake and now The Year of the Flood all exemplify one of the things science fiction does, which is to extrapolate imaginatively from current trends and events to a near-future that's half prediction, half satire. But Margaret Atwood doesn't want any of her books to be called science fiction. In her recent, brilliant essay collection, Moving Targets, she says that everything that happens in her novels is possible and may even have already happened, so they can't be science fiction, which is "fiction in which things happen that are not possible today". This arbitrarily restrictive definition seems designed to protect her novels from being relegated to a genre still shunned by hidebound readers, reviewers and prize-awarders. She doesn't want the literary bigots to shove her into the literary ghetto.

The motive imputed to me is not in fact my actual motive for requesting separate names. What I mean by "science fiction" is those books that descend from HG Wells's The War of the Worlds, which treats of an invasion by tentacled Martians shot to Earth in metal canisters – things that could not possibly happen – whereas, for me, "speculative fiction" means plots that descend from Jules Verne's books about submarines and balloon travel and such – things that really could happen but just hadn't completely happened when the authors wrote the books. I would place my own books in this second category: no Martians. Not because I don't like Martians, I hasten to add; they just don't fall within my skill set. Any seriously intended Martian by me would be a very clumsy Martian indeed.

In a public discussion with Le Guin in the fall of 2010, however, I found that what she means by "science fiction" is speculative fiction about things that really could happen, whereas things that really could not happen she classifies under "fantasy". Thus, for her – as for me – dragons would belong in fantasy, as would, I suppose, the film Star Wars and most of the TV series Star Trek. Mary Shelley's Frankenstein might squeeze into Le Guin's "science fiction" because its author had grounds for believing that electricity actually might be able to reanimate dead flesh. And The War of the Worlds? Since people thought at the time that intelligent beings might live on Mars, and since space travel was believed to be possible in the imaginable future, this book might have to be filed under Le Guin's "science fiction". Or parts of it might. In short, what Le Guin means by "science fiction" is what I mean by "speculative fiction", and what she means by "fantasy" would include some of what I mean by "science fiction". So that clears it all up, more or less. When it comes to genres, the borders are increasingly undefended, and things slip back and forth across them with insouciance.

Bendiness of terminology, literary gene-swapping, and inter-genre visiting has been going on in the SF world – loosely defined – for some time. For instance, in a 1989 essay called "Slipstream," the veteran SF author Bruce Sterling deplored the then-current state of science fiction and ticked off its writers and publishers for having turned it into a mere "category" – a "self-perpetuating commercial power-structure, which happens to be in possession of a traditional national territory: a portion of bookstore rack space"A "category", says Sterling, is distinct from a "genre", which is "a spectrum of work united by an inner identity, a coherent aesthetic, a set of conceptual guidelines, an ideology if you will".

Sterling defines his term slipstream – so named, I suppose, because it is seen as making use of the air currents created by science fiction proper – in this way: "I want to describe what seems to me to be a new, emergent 'genre', which has not yet become a 'category' … It is a contemporary kind of writing which has set its face against consensus reality. It is fantastic, surreal sometimes, speculative on occasion, but not rigorously so. It does not aim to provoke a 'sense of wonder' or to systematically extrapolate in the manner of classic science fiction. Instead, this is a kind of writing which simply makes you feel very strange; the way that living in the late 20th century makes you feel, if you are a person of a certain sensibility."

His proposed list of slipstream fictions covers an astonishing amount of ground, with works by a wide assortment of people, many of them considered to be "serious" authors – from Kathy Acker and Martin Amis to Salman Rushdie, José Saramago and Kurt Vonnegut. What they have in common is that the kinds of events they recount are unlikely to have actually taken place. In an earlier era, these "slipstream" books might all have been filed under the heading of "traveller's yarn" – Herodotus's accounts of monopods, for example, or medieval legends about unicorns, dragons and mermaids. Later they might have turned up in collections of the marvellous and uncanny, such as Des Knaben Wunderhorn, or – even later – the kind of you-won't-believe-this hair-raiser to be found in assortments by MR James or H P Lovecraft or, occasionally, RL Stevenson.

But surely all draw from the same deep well: those imagined other worlds located somewhere apart from our everyday one: in another time, in another dimension, through a doorway into the spirit world, or on the other side of the threshold that divides the known from the unknown. Science fiction, speculative fiction, sword-and-sorcery fantasy, and slipstream fiction: all of them might be placed under the same large "wonder tale" umbrella.

Ustopia is a world I made up by combining utopia and dystopia – the imagined perfect society and its opposite – because, in my view, each contains a latent version of the other. In addition to being, almost always, a mapped location, Ustopia is also a state of mind, as is every place in literature of whatever kind. As Mephistophilis tells us in Marlowe's Doctor Faustus, Hell is not only a physical space. "Why this is Hell, nor am I out of it," he says. "Hell hath no limits, nor is circumscrib'd / In one self place; but where we are is hell, / And where hell is, there must we ever be." Or, to cite a more positive version, from Milton's Paradise Lost: "then wilt thou not be loth / To leave this Paradise, but shalt possess / A Paradise within thee, happier far." In literature, every landscape is a state of mind, but every state of mind can also be portrayed by a landscape. And so it is with Ustopia.

How did I come to create my own Ustopias – these not-exactly places, which are anywhere but nowhere, and which are both mappable locations and states of mind? Why did I jump the tracks, as it were, from realistic novels to dystopias? Was I slumming, as some "literary" writers are accused of doing when they write science fiction or detective stories? The human heart is inscrutable, but let me try to remember what I thought I was up to at the time.

First, The Handmaid's Tale. What put it into my head to write such a book? I had never done anything like it before: my previous fiction had been realistic. Tackling a Ustopia was a risk. But it was also a challenge and a temptation, because if you've studied a form and read extensively in it, you often have a secret hankering to try it yourself. I began the book – after a few dry runs – in Berlin in the spring of 1984. I had a fellowship, in a programme run by West Berlin to encourage foreign artists to visit, as the city was at that time encircled by the Berlin Wall and its inhabitants felt understandably claustrophobic. During our stay we also visited East Berlin, as well as Poland and Czechoslovakia, and I thus had several first-hand experiences of the flavour of life in a totalitarian – but supposedly utopian – regime. I wrote more of the book once I was back in Toronto, and completed it in Tuscaloosa, Alabama, in the spring of 1985. Tuscaloosa provided another kind of flavour – that of a democracy, but one with quite a few constraining social customs and attitudes. ("Don't ride a bicycle," I was told. "They'll think you're a communist and run you off the road.")

The writing of The Handmaid's Tale gave me a strange feeling, like sliding on river ice – exhilarating but unbalancing. How thin is this ice? How far can I go? How much trouble am I in? What's down there if I fall? These were writerly questions, having to do with structure and execution and that biggest question of all, the one every writer asks him- or herself with every completed chapter: is anyone going to believe this? (I don't mean literal belief: fictions admit that they are invented, right on the cover. I mean, "find the story compelling and plausible enough to go along for the ride".)

These writerly questions were reflections of other, more general questions. How thin is the ice on which supposedly "liberated" modern western women stand? How far can they go? How much trouble are they in? What's down there if they fall? And further: if you were attempting a totalitarian takeover of the United States, how would you do it? What form would such a government assume, and what flag would it fly? How much social instability would it take before people renounced their hard-won civil liberties in a trade-off for "safety"? And, since most totalitarianisms we know about have attempted to control reproduction in one way or another – limiting births, demanding births, specifying who can marry whom and who owns the kids – how would that play out for women?

And what about the outfits? Ustopias are always interested in clothing – either less of it compared to what we wear now, or more of it. The clothing concerns usually centre on women: societies are always uncovering parts of women's bodies and then covering them up again. My rules for The Handmaid's Tale were simple: I would not put into this book anything that humankind had not already done, somewhere, sometime, or for which it did not already have the tools. Even the group hangings had precedents: there were group hangings in earlier England, and there are still group stonings in some countries. Looking further back, the Maenads, during their Dionysian celebrations, were said to go into frenzies during which they dismembered people with their hands. (If everyone participates, no one individual is responsible.) For a literary precedent, one need search no further than Emile Zola's Germinal, which contains an episode in which the town's coal-mining women, who have been sexually exploited by the shopkeeper, tear him apart and parade his genitalia through town on a pole. A less raw but still shocking precedent is Shirley Jackson's short story "The Lottery" (which I read as a teenager, shortly after it came out, and which made a chilling impression on me).

The coverups worn by the women in The Handmaid's Tale have been variously interpreted as Catholic (as in nuns) or Muslim (as in burqas). The truth is that these outfits are not aimed at any one religion. Their actual design was inspired by the figure on the Old Dutch Cleanser boxes of my childhood, but they are also simply old. Mid-Victorians, with their concealing bonnets and veils to keep strange men from leering at their faces, would not have found them so unusual.

I prefaced the novel with three quotations. The first is from the Bible, Genesis 30, the passage in which the two wives of Jacob use their female slaves as baby-producers for themselves. This ought to warn the reader against the dangers inherent in applying every word in that extremely varied document literally. The second is from Jonathan Swift's "A Modest Proposal": it alerts us to the fact that a straightfaced but satirical account – such as Swift's suggestion that the grinding Irish poverty of his times could be alleviated by selling and eating Irish babies – is not a recipe. The third – "In the desert there is no sign that says, 'thou shalt not eat stones'" – is a Sufi proverb stating a simple human truth: we don't prohibit things that nobody would ever want to do anyway, since all prohibitions are founded upon a denial of our desires.

The Handmaid's Tale was published in Canada in the fall of 1985, and in the US and the UK in the spring of 1986. In the UK, its first reviewers treated it as a yarn rather than a warning: Britain had already been through Oliver Cromwell and his Puritan republic and there seemed to be no fear of re-enacting that scenario. In Canada, people asked, in anxious Canadian fashion: "could it happen here?" In the US, Mary McCarthy, writing in the New York Times, gave the book a largely negative review on the grounds that it lacked imagination, and anyway it was unlikely ever to take place, at least not in the secular society she perceived as the American reality. But on the west coast, so attuned to earthquake tremors, switchboards on talk shows lit up like Las Vegas, and someone graffitied on the Venice Beach seawall: "The Handmaid's Tale is already here!"

It wasn't already here, not quite, not then. I thought for a while in the 1990s that maybe it never would be. But now I'm wondering again. In recent years, American society has moved much closer to the conditions necessary for a takeover of its own power structures by an anti-democratic and repressive government. Approximately five years after The Handmaid's Tale was published, the Soviet Union disintegrated, the west slapped itself on the back and went shopping, and pundits proclaimed the end of history. It looked as if, in the race between Nineteen Eighty-Four and Brave New World – control by terror versus control through conditioning and consumption – the latter had won, and the world of The Handmaid's Tale appeared to recede. But now we see a United States weakened by two draining wars and a financial meltdown, and America appears to be losing faith in the basic premises of liberal democracy. After 9/11, the Patriot Act passed with barely a cough, and in Britain citizens have accepted a degree of state supervision that would once have been unthinkable.

It's a truism that enemy states tend to mirror one another in organisation and methods. When colonies were the coming thing, everyone wanted one. Atom bombs in the United States created the desire for some in the USSR. The Soviet Union was a large, bureaucratic, centralised state, and so was the America of those times. What form will the United States assume now that it's opposed by unrelenting religious fanaticisms? Will it soon produce rule by the same kind of religious fanaticism, only of a different sect? Will the more repressive elements within it triumph, returning it to its origins as a Puritan theocracy and giving us The Handmaid's Tale in everything but the outfits?

I've said earlier that dystopia contains within itself a little utopia, and vice versa. What, then, is the little utopia concealed in the dystopic world of The Handmaid's Tale? There are two: one is in the past (the past that is our own present). The second is placed in a future beyond the main story by the afterword at the end of the book, which describes a future in which Gilead – the tyrannical republic of The Handmaid's Tale – has ended and has thus become a subject for conferences and academic papers. I suppose that's what happens to ustopian societies when they die: they don't go to Heaven, they become thesis topics.

After The Handmaid's Tale there was a period of approximately 18 years during which I did not write ustopian novels, but then came Oryx and Crake in 2003. Oryx and Crake is dystopic in that almost the entire human race is annihilated, before which it has split into two parts: a technocracy and an anarchy. And, true to form, there is a little attempt at utopia in it as well: a group of quasi-humans who have been genetically engineered so that they will never suffer from the ills that plague Homo sapiens sapiens. They are designer people. But anyone who engages in such design – as we are now doing – has to ask: how far can humans go in the alteration department before those altered cease to be human? Which of our features are at the core of our being? What a piece of work is man, and now that we ourselves can be the workmen, what pieces of this work shall we chop off?

The designer people have some accessories I wouldn't mind having myself: built-in insect repellant, automatic sunblock, and the ability, like rabbits, to digest leaves. They also have several traits that would indeed be improvements of a sort, though many of us wouldn't like them. For instance, mating is seasonal: in season, certain parts of the body turn blue, as with baboons, so there is no more romantic rejection or date rape. And they can't read, so a lot of harmful ideologies will never trouble them.

There are other genetically engineered creatures in the book as well: chickie nobs, for instance, which are chicken objects modified so they grow multiple legs, wings and breasts. They have no heads, just a nutriment orifice at the top, thus solving a problem for animal rights workers: as their creators say, "no Brain, no Pain". (Since Oryx and Crake was published, the chickie nob solution has made giant strides: lab-grown meat is now a reality, though it is probably not in your sausages yet.)

A sibling book, The Year of the Flood, was published in 2009. Its original title was God's Gardeners, but although this was perfectly acceptable to the British publisher, the American publisher and the Canadian publisher objected to it on the grounds that people would think the book was a far-right extremist tract, which goes to show how thoroughly the word "God" has been hijacked. Many other titles were proposed, including "Serpent Wisdom", which the Canadian publisher liked but the US felt suggested a new age cult, and "Edencliff," which the British thought sounded like "a retirement home in Bournemouth". Book titles are either immediately obvious, like The Edible Woman, or very hard to decide on, and The Year of the Flood was the second kind.

The Year of the Flood explores the world of Oryx and Crake from a different perspective. Jimmy/ Snowman, the protagonist of Oryx and Crake, has grown up within a privileged though barricaded enclave, but The Year of the Flood takes place in the space outside such enclaves, at the very bottom of the social heap. Its pre-disaster plot unfolds in neighbourhoods that the security forces – now melded with corporations – don't even bother to patrol, leaving them to criminal gangs and anarchic violence.

However, this book, too, has a utopia embedded within a dystopia; it's represented by the God's Gardeners, a small environmental religious cult dedicated to the sacred element in all creation. Its members grow vegetables on slum rooftops, sing sacred-nature hymns, and avoid hi-tech communications devices such as cellphones and computers on the grounds that they can be used to spy on you – which is entirely true.

Oryx and Crake and The Year of the Flood cover the same time period, and thus are not sequels or prequels; they are more like chapters of the same book. They have sometimes been described as "apocalyptic", but in a true apocalypse everything on Earth is destroyed, whereas in these two books the only element that's annihilated is the human race, or most of it. What survives after the cataclysmic event is not a "dystopia", because many more people would be required for that – enough to comprise a society. The surviving stragglers do, however, have mythic precedents: a number of myths tell of an annihilating flood survived by one man (Deucalion in Greek myth, Utnapishtim in the Gilgamesh epic) or a small group, such as Noah and his family. Do the surviving human beings in Oryx and Crake and The Year of the Flood represent a dystopic threat to the tiny utopia of genetically modified, peaceful and sexually harmonious new humans that is set to replace them?

People have asked, many times, about the "inspiration" for these two books and their world. Of course there are proximate causes for all novels – a family story, a newspaper clipping, an event in one's personal history – and for Oryx and Crake and The Year of the Flood there are such causes as well. Worries about the effects of climate change can be found as far back as 1972, when the Club of Rome accurately predicted what now appears to be happening, so those worries had long been with me, though they were not front-page stories in the spring of 2001 when I began Oryx and Crake. As with The Handmaid's Tale, I accumulated many file folders of research; and although in both there are some of what Huckleberry Finn would call "stretchers", there is nothing that's entirely without foundation.

So I could point to this or that scientific paper, this or that newspaper story, this or that actual event, but those kinds of things are not really what drive the storytelling impulse. I'm more inclined to think that it's unfinished business, of the kind represented by the questions people are increasingly asking themselves: how badly have we messed up the planet? Can we dig ourselves out? what would a species-wide self-rescue effort look like if played out in actuality? And also: where has utopian thinking gone? Because it never totally disappears: we're too hopeful a species for that. "Good", for us, may always have a "Bad" twin, but its other twin is "Better".

It's interesting to me that I situated the utopia-facilitating element in Oryx and Crake not in a new kind of social organisation or a mass brainwashing or soul-engineering programme but inside the human body. The Crakers are well behaved from the inside out not because of their legal system or their government or some form of intimidation but because they have been designed to be so. They can't choose otherwise. And this seems to be where Ustopia is moving in real life as well: through genetic engineering, we will be able to rid ourselves of inherited diseases, and ugliness, and mental illness, and ageing, and … who knows? The sky's the limit. Or so we are being told. What is the little dystopia concealed within such utopian visions of the perfected human body – and mind? Time will tell.

Historically, Ustopia has not been a happy story. High hopes have been dashed, time and time again. The best intentions have indeed paved many roads to Hell. Does that mean we should never try to rectify our mistakes, reverse our disaster-bent courses, clean up our cesspools or ameliorate the many miseries of many lives? Surely not: if we don't do maintenance work and minor improvements on whatever we actually have, things will go downhill very fast. So of course we should try to make things better, insofar as it lies within our power. But we should probably not try to make things perfect, especially not ourselves, for that path leads to mass graves.

We're stuck with us, imperfect as we are; but we should make the most of us. Which is about as far as I myself am prepared to go, in real life, along the road to Ustopia.