When the novelist Maggie O’Farrell was 16, she was invited to a fancy-dress party and knew at once who to be. She put on a black shirt, with a ruffled paper collar, an inky cloak made out of a skirt, her Doc Martens and cheeky shorts over black leggings. To complete her ensemble, she borrowed a skull from her school’s biology lab. She had become obsessed with Hamlet: “He had got under my skin. I felt he was part of my DNA.” And while there is no mystery about Hamlet’s glamorous turbulence appealing to an adolescent, O’Farrell’s feeling was to be rekindled, as an adult, by her discovery of the play’s connection with Shakespeare’s son, Hamnet. There was, she was sure, a novel in it. Over the years, she repeatedly tried to write that novel and almost gave up. Yet it was a story that refused to abandon her.

And now, here it is: Hamnet – the novel of her career. And that is saying something because O’Farrell is the author of eight accomplished and hugely popular books. She won the Costa novel award, in 2010, for The Hand That First Held Mine, and shortlisted for both Instructions for a Heatwave (2013) and This Must Be the Place (2016). Her wildcard memoir, I am, I am, I am: Seventeen Brushes With Death (2017), about living close to the edge, was a bestseller. But Hamnet is a novel apart. And what distinguishes it from O’Farrell’s earlier work is that while it shares the page-turning verve of its predecessors, it pulls off what younger writers (she is now 47) seldom achieve: the power of letting a story appear to tell itself. It reads like a fairytale rooted in heartbreaking reality – there is no magic with which to save a child.

We’re told he did not want to be with Anne, that she trapped him. I started to feel anger about how domestic life is diminished

Hamnet is believed to have died of the plague, aged 11. His brief and precious life was, O’Farrell thinks, more significant than literary historians suppose. In Shakespeare’s time, Hamlet and Hamnet were, according to the critic Steven Greenblatt in the New York Review of Books, the same name. It is not that Hamnet has never featured on stage or screen: there was Kenneth Branagh’s 2018 film All Is True, written by Ben Elton; a short, one-boy show by Bush Moukarzel; and David Mitchell in The Upstart Crow (also by Elton), mentions Shakespeare’s bereavement. But no one has, until now, imaginatively investigated the connection between Hamnet and Hamlet – what is likely to be a clear link between Shakespeare’s life and his work. Claire Tomalin, Dominic Dromgoole and Kamila Shamsie are among those competing for superlatives to hail the novel, and everyone I know who has managed to get hold of an advance copy is in love with it. Of one thing I’m certain: anyone who reads it will find it impossible to see Hamlet again without thinking about the boy after whom the play was named.

It is 11 o’clock on a grey winter morning in Edinburgh and the plan is to meet O’Farrell in a local cafe and walk together to her house. I am waiting as she pushes open the cafe door in a fast-forward flurry. What is immediately likable is that she seems unaware of making a dramatic entrance and of her striking appearance. She is wearing a silver Puffa jacket – as though outer space might be her next stop – and has piercingly blue eyes and, a mass of auburn curls and is yanked into the cafe by a small lurcher. “Luna,” she explains, “my sister’s dog.”

We arrive at a double-fronted stone house and she leads the way into the sunny kitchen. The room’s occupants are mainly cats (it is a school day) and there is a sketch of Voldemort in fancy lace-up boots left behind by her daughter on the kitchen table. A wall of glass gives on to the garden and the vibe is bohemian in a good way, with plenty of evidence of a family life: she has three children with her novelist husband, William Sutcliffe – a son of 16 and daughters aged 10 and seven.

We sit down at the kitchen table, and O’Farrell explains the book’s beginnings when she was at Cambridge University studying English (having got in from a comprehensive school in North Berwick): “At that time, studying English was frustrating because it was all about post-Marxist readings – you were several removes from the text.” But it was through reading biographies of Shakespeare that she learned of Hamnet’s existence. “Whenever they talked about his death, it would be followed by several paragraphs about infant mortality in the late 16th century.” The authors would explain that infant mortality was commonplace and imply that parents barely reacted when their children died. “I found this an extraordinary assumption,” she says. “Hamnet was 11...”

After leaving university, she would ask friends if they knew the names of Shakespeare’s children – had they heard of Hamnet, the twin brother to Shakespeare’s second daughter, Judith? When she told them about Hamnet, they thought she was making him up. “They said: ‘No! Are you sure?’ And I said: ‘Yes, I’m absolutely positive.’ And I thought: if they don’t know, then maybe quite a lot of people don’t know.”

Facebook Twitter Pinterest ‘For all of Shakespeare’s incredible output, he left a very scant paper trail’... Maggie O’Farrell. Photograph: Murdo MacLeod/The Guardian

One of the stumbling blocks to writing the novel was the feeling of presumption about characterising a genius (she describes herself as an Anthony Burgess fan but thinks his novel about Shakespeare, Nothing Like the Sun, does not come off). I notice she never uses Shakespeare’s name in the book. “I couldn’t. When you’re sitting at your computer, immersed in the world you’ve created, and have to write: ‘William Shakespeare had his breakfast…’ it’s impossible not to think: I’m an eejit. Even calling him William seems colossally presumptuous,” and she gives in to a gale of laughter.

But an indignation was growing in O’Farrell that would prove stronger than doubt or mirth – not about Shakespeare but about his wife, Anne Hathaway (also known as Agnes). Agnes is also dismissed by biographers: “From scholarly texts to popular culture, we’re told Shakespeare’s wife was an older woman, a peasant. We’re told he did not want to be with her, that she trapped him into marriage. We’re drip-fed this image. I started to feel anger about how domestic life is diminished: people want to believe Shakespeare appeared in London fully formed, that he did not have a domestic life.”

The assumption he did not grieve is outrageous. It’s not nothing to call a play and a tragic hero after your son

Germaine Greer’s book, Shakespeare’s Wife, which O’Farrell found “inspirational”, recommends that people stop asking: why did Shakespeare marry her? and ask instead: why did she marry him? “People think of Agnes as a cradle-snatching yokel who married a younger man but her family was wealthy, they had a successful sheep farm. I kept thinking: why would a 26-year-old woman, with a good dowry and a secure home want to marry this fellow? He had no trade and was only 18.

“There is no evidence he hated his wife or his domestic life. And the assumption he did not grieve for Hamnet is outrageous. It’s not nothing to call a play and a tragic hero after your son – it speaks volumes. We may not quite know what the volumes are – but it’s a huge act.” She adds that Shakespeare cannot have been indifferent to his family as he sent money made in London back to Stratford and, when he retired, returned to live there with his wife. O’Farrell went to Stratford-upon-Avon to research the book: “What’s astonishing at Shakespeare’s birthplace in Henley Street is that, for all his incredible output, he has left a very scant paper trail, so to be able to walk into that house is astonishing. It’s incredibly moving. It seems so unlikely it has survived, that we can walk through it and into the room in which he was born, the room in which he ate. I went round it several times and talked to several guides and asked millions of questions. They were very patient – they exude such knowledge and love and enthusiasm.”

Facebook Twitter Pinterest William Shakespeare’s birthplace in Henley Street, Stratford-upon-Avon. Photograph: PC Turner/Getty Images/iStockphoto

And yet because so little is known about Shakespeare, O’Farrell had a free rein. This is a novel that, above all, like her memoir, addresses mortality. And, as readers of her memoir know, O’Farrell’s own survival has been against the odds. To mention just three incidents described in I am, I am, I am: she jumped off a harbour wall – a 15-metre drop into black water – and lived to tell the tale. She was held up by a machete-wielding man in Chile. She gave the slip to a man who later murdered a young woman. In Hamnet, she writes: “What is given may be taken away, at any time.” As a writer, she is most herself when on the brink, when thinking about what it is for a life to hang in the balance. And for her, balance has particular significance. She has written about encephalitis, an inflammation of the brain, which exiled her from school aged eight, turned her into a reader and tilted her world (she still sometimes sees things upside down). It is an “impairment to a number of neurological functions, one of which is the ability to sense where things are or should be and my place among them”. It is a condition from which she might have died.

Strangely, given how devastatingly she writes about bereavement, she has never lost anyone close to her: “What informed the novel is the peril we live in with my daughter.” Her middle daughter suffers from anaphylaxis, a potentially lethal allergic reaction, and can go into shock just sitting next to someone eating peanuts. “It’s every parent’s worst and most visceral fear that you will lose your child. That – and the idea you couldn’t save them or weren’t able to safeguard them. I cannot imagine the agony of having to bury a child. It must be unlike anything else.”

But is the situation with her daughter under control? “No. She is 10 and in great danger all the time from naturally occurring things in her environment. We can’t control it at all. I realised when I wrote my memoir, a memoir – the indefinite article is better – it’s only when you finish a book that you understand why you’ve written it. Doing the final copy edit, I realised that trying to pin down in words what she goes through was my way of trying to feel in control but that control was illusory.”

Living vigilantly – being constantly on red alert – is key to understanding O’Farrell and her writing. And although she is not superstitious, a further stumbling block to finishing Hamnet was the fear that it was playing with fire to imagine the death of an 11-year-old boy when she had a son who, at the time, was under 11 himself. “If I was doing this, I’d have to put myself inside the skin of a woman sitting beside her son, laying him out for burial. I thought, I cannot do it – imagine! My son used to say [teasingly picking up on his mother’s anxiety]: ‘On my 12th birthday you probably won’t give me a birthday party.’ That was not true and he’s almost 17 now...”

Facebook Twitter Pinterest Maggie O’Farrell with her cats Malachy, right, and Moses, in 2007. Photograph: Courtesy of Maggie O'Farrell

When I ask whether taking risks is still part of who she is, she says she has played safer since having children. But mightn’t writing be seen as a form of risk-taking? As brave? Exposing? She laughs in disbelief: “Writing is not brave, it’s the opposite – it’s just something I’ve always wanted to do. Surgeons are brave. If there’s a hierarchy of bravery, writers are somewhere near the bottom.”

O’Farrell is an interesting mixture – flighty and earthed. Open and judiciously closed. She never swanks. She talks freely about difficult subjects but swerves from even tame questions that close in on her character. She will not say whether she craves solitude or discuss her intuitions and ducks giving an account of her working day although, to be fair, this might be because she does not have one. “Routine” and “organisation” are alien to her. She gestures at her kitchen and insists tidiness is beyond her, absently fielding a dropped toy from the floor to confirm the point.

Ironically, she has much more to say on the subject of her stammer. Speech therapy – five years ago – has helped although radio interviews, she says, remain a challenge. “Every stammerer has a collection of sounds they can’t start off on and one of mine was ‘M’.” And that is not easy when your name is Maggie. But writing has always been a reprieve. “There’s indescribable joy in being able to write when nothing stops you.”

Might hearing the criticisms of a novelist husband sometimes stop her? “Will and I have known each other a long time. We met in the first year of university but weren’t together for another 10 years. I trust his judgment and know he will tell me what he thinks. When he read the first draft of The Vanishing Act of Esme Lennox (2006) he said: ‘You need to rewrite half of it.’ And I did feel annoyed – we had a slightly frosty dinner. But the next day, I thought: dammit, he’s right!”

Alas, poor Hamnet: spotlight falls on Shakespeare’s tragic only son Read more

O’Farrell loves living in Edinburgh but says she has to look after herself carefully when not writing: “My body is not my strong point – there are lots of things I have to do to keep it ticking over.” She is a yoga devotee and in summer swims in a loch outside the city. It sounds like a good life. Yet when I ask if she dreams of living elsewhere, she replies, “Always…” Perhaps imagining elsewhere is a novelist’s default position.

O’Farrell was born in Coleraine, Northern Ireland, and her family moved to Britain in 1974. The daughter of an economist, she grew up with her mother and two sisters in Wales and Scotland. As an adult, she has lived in Hong Kong (a year working for a computer magazine) and London (as an arts journalist). But Ireland never loosens its hold: “My passport is Irish and, since Brexit, I’m clinging proudly to it. I’m wary of claiming Irish heritage but I often mistily think: I shall move back to Ireland.”

She once wrote: “In any fairytale, getting what you want comes at a cost.” What does she want herself? “I want Brexit not to have happened. I worry about the next generation. We’ve messed everything up. Not only have our kids got to face the climate emergency but what’s going to happen to Britain? They’ll have huge strains on them and no immunity to digital onslaught and destruction. How can we equip them to cope?” About her own future, she volunteers nothing. Instead, she tells a story. One day, while she was writing Hamnet, her daughter looked over her shoulder and started to read. After a while, she remarked to her mother: “I don’t like this story – it’s too sad. Please write a happy story next.” And now, as Hamnet finds its audience, O’Farrell is settling down to a children’s book, to be published at the end of the year. “And it’s going to be happy – I’m doing it for her.”

• Hamnet is published by Tinder Press on 31 March (£20). To order a copy for £16 go to guardianbookshop.com or call 020-3176 3837