It was a mild early summer’s day, with buttercups everywhere. I was map reading and my partner was driving. We were somewhere in West Sussex, on a country road because I couldn’t cope with the harsh rush of the motorway. I’d recently been ill, and felt strongly that I never wanted to re-enter the normal world, with its clattering trucks, fluorescent litter, hashtags and memes. For the last few weeks I had only been able to read PG Wodehouse and Dodie Smith. So I felt happy in the simple nostalgic calm of the English countryside, with hamlets called things like Old Wives Lees and roads called Frog Hole Lane.

And suddenly, just off the A272, was a place called Dragon’s Green.

“How brilliant,” I said to my partner. “If I ever write a children’s book I’m calling it ‘Dragon’s Green’. Can you imagine a more perfect name?”

“Would you ever write a children’s book?” he asked me, surprised.

“Of course not,” I said.

After all, I was a Serious Adult Novelist. Well, serious-ish. I’d written about massive talking mice and time travel and people flying, and my work had been getting more, well, playful. But I was about to publish my 10th novel and become a professor. That seemed pretty serious to me. Almost depressingly so.

On the A272 that day I was in a crisis. I’d been ill for months and I didn’t know why. The words “nervous breakdown” had been used. I’d been working on a memoir about trying to make it as a fortysomething tennis player, and although I’d got to No 6 in the country (mainly down to a computing quirk), the whole experience had thrown me off course. We were on the A272 because I was heading to see a psychologist whose book I’d read and who I hoped was going to help me.

Jacques Derrida for middle-graders? Why not? After all, no one was going to read it

He did. On the way home, I felt lighter. We stopped for tea in Dragon’s Green. And then the idea for my novel came in a big rush. It would have to have a dragon, for a start. What do dragons like? Princesses. Maybe I would have a princess rescued from a dragon, but by another girl. And perhaps it would turn out that the princesses were being farmed in a special school to make them appealing to dragons. I didn’t want this to be like the 1980s feminist primers I read as a child, though, and I wanted to appeal to boys as well. I would give my fictional children magical powers, but also limit them in some way, because magic can’t be easy. And I’d continue some of the investigations I’d begun in my adult fiction to do with the power of books, and how language and narrative create our world. Jacques Derrida for middle-graders? Why not? After all, no one was going to read it.

I mean, they couldn’t. Especially not my students. It would be too embarrassing. I’d actively banned them from writing children’s fiction for the last 12 years. We have a longstanding policy in my department to always give students fresh and complex experiences. We don’t encourage them to write dissertations on Harry Potter because university is time for new things. There is nowhere in the world other than university where you can sit in a group and read James Joyce’s story “The Dead” and discuss the concept of epiphany. And then, in my favourite session I teach, work out how to write one.

Another problem with using children’s fiction in creative writing classes is that good children’s fiction often does bad things. In the right hands these can become very good things – but not in a way that is easy to teach, or learn. There will be cliches, adjectives, adverbs, arch villains, wild exaggeration. People in Dragon’s Green talk in booming voices and sweep into rooms and cackle loudly. Instead of using the minimum amount of words necessary (a creative writing edict left over from the We-Must-All-Write-Like-Hemingway period), a good children’s voice can be a bit of a pile-up. Done well, this is free indirect style on acid. It is amazing. None of it will get you many marks when you are aiming for subtlety and sophistication, as our students are, unless you are a one-in-a-million genius. But my God it’s fun to write. Sorry, everyone.

Facebook Twitter Pinterest ‘I’ve never enjoyed writing anything more in my life’ … Scarlett Thomas. Photograph: Murdo MacLeod/The Guardian

Halfway through writing Dragon’s Green I recovered from my breakdown. I still found I had very little appetite for contemporary literary fiction, but I was able to go back to my usual favourite writers, among them Tolstoy, Chekhov and Katherine Mansfield. My novel suddenly acquired an underworld, full of people in black polo necks drinking strong coffee and talking about The Master and Margarita. And I made the ordinary world beautiful in the way I think our degraded world can be, with pink neon signs and strange things on the radio. I’d already abolished the internet on page eight, and good riddance. And our world had become 10% more magical. I have never enjoyed writing anything more in my life.

But it turns out that creating a fictional world is a very complex act. Who has power? How does magic work? Is everyone magical or are some people born muggles (an idea that, despite loving other aspects of Harry Potter, I have always disliked). As soon as you put your characters in large country houses with magical-looking turrets you have to decide how these houses are to be maintained. A feudal world looks pretty, but needs servants. What do you do if you don’t believe in servants? And what do you do about violence once you’ve given your children magical weapons? Solving these problems puts you in touch with the very core of your beliefs. You can lie to adults and call it “irony”. Lying to children is a different matter.

You can lie to adults and call it 'irony'. Lying to children is a different matter

Writing children’s fiction has made me understand how it can be worthy of study. It’s not that these are necessarily exemplary texts when it comes to putting sentences together. I don’t think children’s novels should replace James Joyce and George Eliot and Arundhati Roy on literature courses. But they do touch on the fundamental things about life. In their own way they too ask how the world should be, how it possibly could be, what is right, what is wrong, who can love whom. They are always political texts.

I still believe that there has to be a place where people are required to read “The Dead”. A contemporary reader must struggle to get to the epiphany at the end, but then one’s very soul is touched. It’s the closest thing to magic in this world, when marks on a page can do that. We owe it to university students to give them this experience. Real magic, as well as the fictional sort.

But perhaps it’s no coincidence that, in recent years, I have started teaching fiction with all different kinds of magic in it. George Saunders’ story “Sea Oak”, with its foul-mouthed zombie aunt, has long been a favourite. Kelly Link’s story “The Summer People” works brilliantly as an uncanny move into an adult way of looking at childhood things. And perhaps now I will add a few children’s books. Philip Pullman’s Northern Lights for its singular world-building, and Joan Aiken’s The Wolves of Willoughby Chase for its queer moments. In any case, I am abandoning my old classification of books. Instead of thinking there’s “literary fiction” and “everything else”, or even adult fiction and children’s fiction, I now believe that there are books with magic and without. And I much prefer them with, thank you very much.

• Dragon’s Green will be published by Canongate on 6 April. To order a copy for £11.04 (RRP £12.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.