Ian McEwan’s latest novel, Machines Like Me, is a fiction about science – specifically, artificial intelligence. It is set in an alternative reality where Alan Turing does not kill himself but invents the internet instead; where JFK is never assassinated and Margaret Thatcher’s premiership ends with the beginning of the Falklands war. The near future of the real world becomes the present of the novel, giving McEwan the space to explore prescient what-ifs: what if a robot could think like a human, or human intelligence could not tell the difference between itself and AI?

Machines Like Me is not, however, science fiction, at least according to its author. “There could be an opening of a mental space for novelists to explore this future,” McEwan said in a recent interview, “not in terms of travelling at 10 times the speed of light in anti-gravity boots, but in actually looking at the human dilemmas.” There is, as many readers noticed, a whiff of genre snobbery here, with McEwan drawing an impermeable boundary between literary fiction and science fiction, and placing himself firmly on the respectable side of the line.

On the face of it, it is as absurd for McEwan to claim he’s not writing sci-fi as it is for him to imply that sci-fi is incapable of approaching these themes interestingly: alternative history and non-human consciousness are established sci-fi motifs, thoroughly explored in defining genre works such as Philip K Dick’s The Man in the High Castle and Do Androids Dream of Electric Sheep? But genre is as much about what you keep out as what you let in, and science fiction – as well as being a label in its own right, suggestive of a certain tone and content – functions as a kind of insalubrious “other” against which literary authors can demonstrate their superiority.

In 1968, Vladimir Nabokov told the BBC: 'I loathe science fiction with its gals and goons, suspense and suspensories'

This is not a new development. In 1968, Vladimir Nabokov priggishly informed a BBC interviewer: “I loathe science fiction with its gals and goons, suspense and suspensories.” At the time, he was working on Ada, a sprawling incest saga set on a parallel Earth called Demonia. When it was published, critics were graciously careful to differentiate Ada from the “trashy space-operas” that it superficially resembled. But then again, SF is not a respectable genre. Its origins are brash and cheap, its richness often married to its slightly disreputable status.

Roger Luckhurst, editor of Science Fiction: A Literary Historyand professor in modern and contemporary literature at Birkbeck, University of London, believes that sniffy attitudes towards SF are rooted in the gothic boom of the 18th and 19th centuries. Envy played a part, he says: “Someone like Ann Radcliffe was selling thousands and thousands of copies and being paid enormous amounts of money, so consequently proper literary people said it was all ghastly and rubbish, and they were really annoyed that they weren’t earning that much.” In 1818, Mary Shelley’s Frankenstein married a gothic vision to cutting-edge medical theories about “galvanism”, which posited that life could be created outside the womb, and in so doing gave birth to a prototype for science fiction. But the term itself wouldn’t come into use until the pulp magazines of the 1920s helped to stratify genre conventions. Named for their cheap paper stock, these mass-market publications were segmented into “romance”, “detective”, “spicy”, “horror” and, of course, science fiction. SF magazine Amazing Stories published Jules Verne, HG Wells and John Wyndham, as well as authors now more associated with horror or fantasy such as HP Lovecraft and Edgar Allan Poe.

The pulp market faded out in the mid-20th century and the strict separation of genres to some extent faded with it. Consequently, so too has the stigma attached to SF. Colson Whitehead won both the Arthur C Clarke award and the Pulitzer prize for fiction in 2017 for The Underground Railroad, which imports a brilliant steampunk conceit into a re-creation of slavery-era America. In the same year, Naomi Alderman’s The Power (in which women first evolve the ability to deliver electric shocks, and then start using it against men) won the Baileys prize for women’s fiction. Jeanette Winterson’s soon-to-published Frankissstein revisits the Shelley original to explore sexbots, AI and biotechnology.

Facebook Twitter Pinterest Sci-fi? … The Handmaid’s Tale, based on Margaret Atwood’s novel. Photograph: Hulu

For many authors, genre is something their marketing team might worry about but not something they will sweat over themselves. Perennial SF refuseniks such as Margaret Atwood have softened. In 2003, she claimed that her near-future novel of gene manipulation and climate change Oryx and Crake was speculative fiction because it didn’t include “monsters and spaceships”; recently, she has begun to refer to The Handmaid’s Tale as science fiction. In this context, McEwan looks like something of a stubborn holdout – but there is evidence that, even as the silos start to collapse, readers remain highly attuned to genre conventions, and that writers can be punished for breaching them.

Science fiction retains a rather juvenile set of associations that can make readers embarrassed to enjoy it. Some decide, if they like a book, it’s too good to be SF

A 2017 study found that when a text included vocabulary such as “airlock” and “antigravity”, readers reported lower empathy with the characters. This is the problem Ursula K Le Guin grappled with in her 1976 essay Science Fiction and Mrs Brown. If the essence of the novel is character (epitomised by Virginia Woolf’s everywoman Mrs Brown), wondered Le Guin, is it even possible for the science fiction author to write a novel? “Have we any hope of catching Mrs Brown, or are we trapped for good inside our great, gleaming spaceships hurtling out across the galaxy?” she asked. Or was the problem that convincing novelistic psychology fatally undermined the world-building of sci-fi? “Could it be that Mrs Brown is actually, in some way, too large for the spaceship? That she is, you might say, too round for it – so that when she steps into it, somehow it all shrinks to a shiny tin gadget?” And whereas in the pulp era the masthead established the brand and authors could move between genres by moving between titles, for contemporary authors, their brand is their name. If you become known as a “sci-fi author”, then everything you do is likely to be judged in that light. (One solution is the pseudonym: Iain Banks would add an M between his names to signal that he was working in space mode.)

Michel Faber is one author who moves between the realist and the SF registers, and he has noticed how readers can be unsettled by their genre expectations: “I’m forever hearing people saying that they were surprised by how much they loved and were moved by The Book of Strange New Things or Under the Skin, because they ‘don’t like science fiction’ or ‘wouldn’t usually read a book like that’.” Science fiction retains a rather juvenile set of associations – “heroic captains in black and silver uniform ... mad scientists with nubile daughters,” as Le Guin put it – that can make readers embarrassed to enjoy it. One way to deal with that embarrassment is to decide that, if you liked it, it’s clearly too good to count as SF. “Under the Skin was discussed on BBC Radio 4’s Open Book recently and the three presenters tried their best to argue that it wasn’t really sci-fi because it was beautifully written and had such strong characterisation and profound themes,” recalls Faber. “On the one hand it’s lovely to be appreciated, but on the other you can see the institutionalised disrespect for the genre and understand why it drives sci-fi writers mad.”

Not everyone agrees that science fiction would be improved by being more respectable. As a student, the American writer Joanna Russ was taught that women lacked the universal perspective from which to create literature (one of those who taught her was Nabokov). Genre fiction gave her a way back to writing: “Convinced that I had no real experience of life, since my own obviously wasn’t part of Great Literature,” she wrote in 1983, “I decided consciously that I’d write of things nobody knew anything about. So I wrote realism disguised as fantasy, that is, science fiction.”

Facebook Twitter Pinterest Gothic vision … Bela Lugosi and Ilona Massey in Frankenstein Meets Wolf Man, inspired by Mary Shelley. Photograph: Allstar/UNIVERSAL

You can see the same attraction to science fiction’s outsider status in something like Alasdair Gray’s 1981 novel Lanark, which tells the story of one main character (introduced as Lanark, but also called Thaw) over four parts, with two SF/fantasy sections enfolding the central realist passages set in Glasgow. Gray’s “realism disguised as fantasy” isn’t speculative so much as satirical, holding up a mirror to the predations and frustrations of a class-bound capitalist society. When the child Thaw is shown a sci-fi magazine in one of the realist sections he protests: “I don’t like science fiction much. It’s pessimistic.” In Gray’s vision, it is pessimistic precisely because it offers the most faithful way of portraying uncomfortable truths that the narrow scope of literary fiction would exclude.

But science fiction is not inevitably bleak, or necessarily shameful. For Becky Chambers, author of the Wayfarers trilogy of books, which blend expansive adventure with warm curiosity about concepts such as artificial intelligence and body modification – science fiction is a source of pride and joy. “I just wanna stand up on my soapbox and be like: ‘I write science fiction, dammit!’” she says. At the same time, “there is a fear that your work is not going to be taken seriously. That you’re gonna be told to go sit at the science fiction table and you won’t get to mingle with other people.”

This exclusion can be “disheartening”, she says. “You don’t want to just be preaching to the choir. The whole point of science fiction is to get people to see the world differently, and that’s most effective if you’re reaching a lot of people.” But genre, as much as it’s potentially a trap, is also simply an essential way of navigating culture, adds Faber: “Labels are handy when people are faced with the mind-boggling oversupply of entertainment options offered up by turbocapitalism.”

Science fiction is a tool for writers as well as readers. In the Mrs Brown essay, Le Guin concludes that science fiction can be purposed to literary ends. “What is science fiction at its best,” she wrote, but a “new tool” of the kind Woolf set out to discover: “a crazy, protean, left-handed monkey wrench, which can be put to any use the craftsman has in mind”. If writers such as McEwan won’t acknowledge the monkey wrench in their hand, maybe it’s simply because they don’t understand its true power.