Go into a bookshop. You are surrounded by classifications. Crime fiction, romance, science fiction, fantasy … These are the genres; they specialise. Crime fiction is a puzzle. Science fiction addresses philosophical questions in the form of an adventure story. Romantic fiction is about love, but there are restrictions on what kind of love it is – otherwise the book belongs somewhere else.

There are limits and rules. Usually the book slots into its genre like a well-aimed dart. Sometimes there's a question. Maybe it's genre. Maybe it's "literary". "Literary" doesn't have a labelled shelf. There are reasons for this, one of which is that it's hard to define, but let's have a go. Literature is writing of high quality, sustained by intelligent structure and informed by original thought. It requires integration of all the elements into an intellectually and emotionally satisfying whole. Trickiest of all: it has to say something.

Now, if a book slots easily into its genre, it's because it's been designed that way by a writer who knows exactly what he or she is doing. That, I suggest, is an important difference between literary and genre fiction. Not that writers of literary fiction don't know what they're doing, but there is a difference in the level of planning. A genre novel is governed by limitations, and the whole of the writer's skill is directed towards creating the best possible novel within those limitations. A literary novel is governed by nothing – nothing I can think of, not even the requirement to be comprehensible – and the whole of the writer's skill is directed towards creating the best possible novel. This involves, at some point, a surrender to the unknown.

Something follows from this. We know to what, let's say, the author of a crime novel is aspiring. To what is the author of a literary novel aspiring? Well … it is extraordinarily difficult to say. The work may have excellent qualities, yet fail in its own terms. Because it is reaching beyond. To what? An epic canvas? A psychological depth? A vision of the human predicament? The truth?

This is why no one wants to talk about literary fiction. It's too embarrassing. Critics avoid the term because it sounds old-fashioned, writers dare not use it of their work because it sounds pretentious. There's a vogue for saying that literary fiction is just another genre. But it is trying to do something different, and there is need for a term that acknowledges that.

So what is the relationship between literary and genre fiction?

I propose the image of a wheel. It has a hub, spokes and a rim. The position of any point on any spoke is defined by its distance from the hub in one direction and the rim in the other. We can call the spokes crime fiction, science fiction, horror, what you will. The hub holds the spokes together, but their strength is in their separateness. And in the fact that they do what they do, and not something else.

What is in the hub? Clearly, because the spokes connect with it, it has to be a bit of everything. But in literary terms, that would be a mess. It has to be the possibility of everything. This is why the literary novel cannot be governed by rules. But note, the hub needs to be strong, or the spokes will pull it apart. And it isn't easier to write something that doesn't have rules; it's harder. There's nothing to start from.

It is my argument that each of these spokes is a continuum, that every book is somewhere on its spoke but might – given some difference in treatment – have been somewhere else. The nearer a book is to the hub, the more "literary" it is. Books close to the hub may be regarded as both literary and genre.

I'll give few examples of books that are somewhere on the science fiction spoke. JG Ballard's The Drowned World: a hallucinatory evocation of heat and decay, containing a fascinating idea about the regression of the human central nervous system. These are literary qualities. But it goes into a thrillerish escapade with a bunch of pirates which has nothing to do with anything and can only be explained by the need to invent a plot.

Ursula K Le Guin's The Left Hand of Darkness: original, but do we care whether the people of the dreadful planet Gethen join the Federation? No, and nor does Le Guin, because what she is interested in is the fact that they are hermaphrodites. But this aspect, well explored, is not completely integrated; you could have the story without it.

Brave New World: a classic. The writing is as smooth as silk and there's a brilliant premise: human beings are conditioned from the womb to be contented with their lot. Hence no conflict, or no conflict that stands a chance of getting anywhere. And no complexity of character: the premise excludes it.

In Iain M Banks's Consider Phlebas – hard SF – a solid world, an integrated whole, but, again, no character. Genre doesn't demand character: some writers are good at it – John le Carré, Raymond Chandler – but it isn't essential.

Genre fiction encourages a focus on one aspect at the expense of others. This can produce an obsessive, powerful work. But a literary novel has, somewhere, to open out into the universal.

What's close to the hub?

I choose Margaret Atwood's Oryx and Crake. It's apocalyptic. Humans have been wiped out by a virus. Genetically mutated animals roam. There is a new humanoid species, the Crakers, created in a laboratory; they are herbivorous, pacific and amiable. If there's a future for intelligent life, they are it. There's one human left, apparently: he's called Jimmy and he is supposed to be looking after the Crakers. We are taken in flashback through what preceded the disaster: giant corporations take over the world, order is enforced by the wonderfully named CorpSeCorps, cloud forests are cut down to grow HappiCuppa coffee beans.

Jimmy takes his eye off the Crakers and by the end of the book they have met three more humans. These are not good people. If Jimmy does not kill these three humans first, the Crakers will be massacred, and with them will go whatever hope there is. If he does kill them, he is annihilating what remains of the human race. What is he going to do?

Oryx and Crake is rooted in genre – Atwood calls it speculative fiction – but has all the qualities of a literary novel. The writing is spare. The structure is tight. The observation of the human condition is both profound and impish. Character is crucial. The issues are huge and we feel the weight of them. Finally, it leaves the reader on a cliff-edge the like of which I have never encountered elsewhere. It was nominated for the Man Booker. I think it should have won.

So: of course there is a difference between literary and genre fiction. Our experience as readers tells us so, commercial practice says so. But it is not the difference between two continents separated by ocean. It is the difference between the two ends of a continuum. Between those two points is an infinity of fruitful positions.

Anita Mason's The Illusionist was shortlisted for the Booker prize in 1983. Her most recent novel is The Right Hand of the Sun (John Murray). This is an edited version of a speech given at an Oxford Literary Festival debate against the motion "Genre fiction is no different from literary fiction".