Devotees of fanfiction will sometimes tell you that it’s one of the oldest writing forms in the world. Seen with this generous eye, the art of writing stories using other people’s creations hails from long before our awareness of Twilight-fanfic-turned-BDSM romance Fifty Shades of Grey: perhaps Virgil, when he picked up where Homer left off with the story of Aeneas, or Shakespeare’s retelling of Arthur Brookes’s 1562 The Tragical History of Romeus and Juliet. What most of us would recognise as fanfiction began in the 1960s, when Star Trek fans started creating zines about Spock and Captain Kirk’s adventures. Thirty years later, the internet arrived, which made sharing stories set in other people’s worlds – be they Harry Potter, Spider-Man, or anything and everything in between – easier. Fanfiction has always been out there, if you knew where to look. Now, it’s almost impossible to miss.

In the last few years, fanfiction has enjoyed something of a rebrand. Big-name authors such as EL James, author of the Fifty Shades books, and Cassandra Clare, who has always been open about writing Harry Potter fanfiction before her bestselling Mortal Instruments series, have helped bring it into the mainstream. These days, it’s fairly common knowledge that some people just really like writing about Captain America and Bucky Barnes falling in love, or Doctor Who fighting demons with Buffy. The general image of fanfiction has brightened somewhat: less creepy, more sweetly nerdy.

But the divide between fanfiction and original writing holds strong. It’s assumed that if people write fanfiction, it’s because they can’t produce their own. At best, it functions as training wheels, preparing a writer to commit to a real book. When they don’t – as in the famous case of Fifty Shades, which one plagiarism checker found had an 89% similarity rate with James’s original Twilight fanfiction – they are ridiculed. A real author, the logic goes, having moved on to writing their own books, doesn’t look back.

‘Fanfiction is a great incubator for writers’ … Naomi Novik. Photograph: Beth Gwinn

“Here’s the thing,” Naomi Novik explains over the phone from New York. She is the bestselling author of the Temeraire books, a fantasy series that adds dragons to the Napoleonic Wars, and Spinning Silver, which riffs on Rumpelstiltskin. “I don’t actually draw any line between my fanfiction work and my professional work – except that I only write the fanfiction stuff for love.”

In between writing her novels – or indeed during, as she admits that fanfiction is one of her favourite procrastination techniques – Novik is an active member of the fanfiction community. She is a co-founder of the Archive of Our Own (AO3), one of the most popular hosting websites, and a prolific writer in the universes of Harry Potter, Game of Thrones, Merlin and many more.

And she’s not the only professional at work. Rainbow Rowell, the bestselling author of Eleanor and Park and other novels, once told the Bookseller that between two novels, she wrote a 30,000-word Harry Potter fanfiction. “It’s Harry and Draco as a couple who have been married for many years, and they’re raising Harry’s kids,” she said. “It’s them dealing with attachment parenting and step-parents and all these middle-aged issues.”

Marriage material...? Tom Felton as Draco, and Daniel Radcliffe in 2002’s Harry Potter and the Chamber of Secrets. Photograph: Allstar/WARNER BROS.

The divide between a fanfiction writer and an original fiction writer can look very arbitrary when looking at authors such as Michael Chabon, who once described his own novel Moonglow as “a Gravity’s Rainbow fanfic”. Or Madeline Miller, whose Orange-prize winning The Song of Achilles detailed the romantic relationship between Achilles and Patroclus, and whose latest novel Circe picks up on the witch who seduces Odysseus in the Odyssey. Miller said she was initially worried when one ex-boyfriend described her work as “Homeric fanfiction” but has since embraced her love of adapting and playing with Greek mythology. The tag could also be applied to classics such as Jean Rhys’s Wide Sargasso Sea, reworkings of Shakespeare by the likes of Margaret Atwood and Edward St Aubyn in the Hogarth series, and a spate of parodies: Pride and Prejudice and Zombies, or Android Karenina.

Neil Gaiman once wrote that the most important question an author can ask is: 'what if?' Fanfiction takes this to the next level

What separates these works from the Harry Potter fanfiction you find online may come down to snobbery. There is an undercurrent of misogyny in mainstream criticism of fanfiction, which is widely accepted to be dominated by women; one census of 10,500 AO3 users found that 80% of the users identified as female, with more users identified as genderqueer (6%) than male (4%). Novik has spent a good deal of time fighting against fanfiction’s stigma because she feels it is “an attack on women’s writing, specifically an attack on young women’s writing and the kind of stories that young women like to tell”. Which is not to say that young women only want to write about romance: “I think,” Novik says, “that [the popularity of fanfiction amongst women is] not unconnected to the lack of young women protagonists who are not romantic interests.”

Others may find it odd that published authors would bother writing fanfiction alongside or between their professional work. But it’s all too simple to draw lines between two forms of writing that, in their separate ways, can be both productive and joyful. Neil Gaiman once wrote that the most important question an author can ask is: “What if?” Fanfiction takes this to the next level. What if King Arthur was gay? What if Voldemort won? What if Ned Stark escaped?

“I believe that all art, if it’s any good, is in dialogue with other art,” Novik says. “Fanfiction feels to me like a more intimate conversation. It’s a conversation where you need the reader to really have a lot of detail at their fingertips.”

Fanfiction began in the 1960s, when Star Trek fans started creating zines about Spock and Captain Kirk’s adventures. Photograph: Allstar/Cinetext/PARAMOUNT

For writers still wobbling on training wheels, fanfiction offers benefits: the immediate gratification of sharing writing without navigating publishers; passionate readers who are already interested in the characters, and a collegial stream of feedback from fellow writers.

“There was an audience of people who wanted to read my writing,” says young adult author Sarah Rees Brennan, who wrote Harry Potter fanfiction in her teens and twenties before she published her own novels, the latest of which, In Other Lands, was a Hugo award finalist. “Here were all these people online who wanted stories about familiar characters. Audiences were pre-invested and waiting.”

For writers, whether already published or on the path to being published, this instantaneous readership functions as a writer’s workshop: Novik calls it a “community of your peers”. Spending hours thrashing out the details of Draco Malfoy’s inner life can’t help but function as a crash course in character motivation. And the limits and constraints of working within a pre-existing world, with its own characters and settings, is a unique challenge.

“Fanfiction is a great incubator for writers,” Novik says. “The more constraints you have on you at the beginning, the better. It’s why people do writing exercises, or play scales. That kind of constraint forces you to practice certain skills, and then at a certain point you have the control to bring out the whole toolbox.”

Once some writers get those tools, they never look back. Rees Brennan no longer writes fanfiction. “I had a friend say it’s like the difference between babysitting kids and having children of your own,” she says. “With a world you built yourself, and characters you built, there’s this sense of deep, overwhelming love.”

But Rees Brennan is still a fan of collaborative writing and shared universes, as in the short stories she writes with Cassandra Clare about characters from Clare’s Mortal Instruments universe. “It’s amazing to gather around a kitchen table and yell at each other excitedly about what’s going to happen to mutually beloved characters,” she says. “I want that for every creative person – a chance to find their imaginative family, wherever it may be.”

Novik scorns the idea that published authors should turn their back on fanfiction. She recalls being on a panel where one member said he couldn’t understand why someone would waste their time writing it over an original work: “I said, ‘Have you ever played an instrument?’ He was like, ‘Yeah, I play piano’. I said, ‘So, do you compose all your own music?’”

“When I was first published, I deliberately went to my editors and said, ‘Yes, I’ve been writing fanfiction for 10 years. I love it.’ It was non-negotiable for me. As soon as you do that, by the way, it turns out that like half of the publishing industry has read or been involved in fanfiction,” she laughs. “Shockingly! It’s amazing how all these women who like storytelling have some connection to the community.”

For Novik and many other writers, fanfiction is a fundamental a way of expressing oneself, of teasing out new ideas and finding a joyous way to engage with writing again after the hard slog of editing a novel. The journey to become a published writer isn’t a straight line; it’s a spiral, as we grow older and continue to explore the characters and tropes we love. There’s so many stories waiting to be told – perhaps one or two of them could involve getting Captain America laid. God knows he needs it.