Ian McEwan was in a reflective, autumnal mood when we met recently to talk about his new novel, The Children Act. It's a mood that echoes the dominant note of this short book. McEwan's prose has always been pared down but the full foliage of, say, Atonement, has now become the thinning leaf of The Children Act, whose title refers to the landmark legislation of 1989 governing the welfare of minors.

This sombre tale of barely 55,000 words is a study in grey, a characteristically haunting examination of unarticulated love, of an old marriage in crisis, and a mature woman at the end of her tether. "I wanted," says McEwan, in his forensic, slightly nasal but commanding drawl, "to get inside the mind of a woman who is professionally totally engaged, but with turmoil in her life, who comes into a situation in which all kinds of buried feelings are stirred up…"

Fiona Maye is a high court judge who presides in the family division. She's flinty, cerebral and childless. Her legal judgments exhibit a crisp and even-handed prose that's "almost ironic, almost warm". Like McEwan, she loves classical music, especially Bach. Unlike him, she can also perform it as an amateur, especially the second Partita. This makes her sound horribly over-accomplished but McEwan says he did not want her to be "a goody two-shoes". Still, with her "godly distance", she may be colder to the reader than he realises.

Anyway, as the novel opens, Jack, her husband of 30 years, is telling her that he wants to have "one big passionate affair", a last hurrah, the Saga generation's sexual fantasy of "blacking out with the thrill of it". Jack and Fiona are on the cusp of 60. Old age is creeping up. "Not the full withering," writes McEwan with a kind of steely relish, "but its early promise was shining through."

He got the seed of the idea for The Children Act from his friend, the former appeal court judge Sir Alan Ward, coincidentally the husband of the lawyer Helen Ward, who handled McEwan's acrimonious divorce in 1995. "It's an aspect of getting older," he remarks, "that I find in my social circle a handful of judges, some of them retired. I've become very intrigued by their company and conversation." He has also become intrigued by the literary dimensions of the judicial process, especially its finely balanced judgments.

Ageing is the context but not the first subject of The Children Act. Simultaneously with Jack's calculated request for "one last go", Fiona finds herself called to try the case of a teenage Jehovah's Witness, a brilliant and beautiful boy named Adam Henry, who is refusing the blood transfusions that might cure his leukaemia. This basic plot, says McEwan, "crops up [in the west] about half a dozen times a year". What happens next, however, is much darker and stranger. On a whim, in flight from "the subdued drama of her half-life with Jack", Lady Justice Maye decides to assess the boy's resolve by visiting him in hospital. This encounter, an edgy mixture of the ecstatic and the mundane, typical of McEwan, sets up a denouement replete with terror, death and reconciliation. So far, you might say, so very McEwan.

First Love, Last Rites: Natasha Gregson Wagner and Giovanni Ribisi, in the 1997 film adaptation of McEwan's first book, published in 1978.

Ever since the publication of First Love, Last Rites, and its successors, notably The Cement Garden, The Child in Time, Enduring Love, Atonement, Saturday and On Chesil Beach, many critics have noted the quasi-gothic interplay of family and trauma in McEwan's work. But The Children Act is a more subtle, more reflective reconsideration of old themes. Although, as usual, it is compulsively readable, behind the top line of the narrative there are darker intimations of sorrow. In the last few years, McEwan the novelist has endured the harrowing experience of death very close to home. First there was the drawn-out and painful exit of his old friend and comrade-in-arms, the writer Christopher Hitchens. This, says McEwan, has left "one of the biggest holes imaginable". More shockingly, this was followed in April this year by the sudden death, from a massive heart attack of his much-loved literary agent of long standing, Deborah Rogers. When that call came, he says, "I could understand the grammar of the sentence – 'Deborah has just died' – but I couldn't absorb it, couldn't believe it. I said: 'Say that again.'"

Generally, McEwan, once a prince of darkness and artist of the danse macabre, is sanguine, optimistic and robust in his faithlessness, a convinced atheist. Lately, however, life and art have begun to elide. Mortality has reverberated through his life, as Auden says, like "the sound of distant thunder at a picnic". He concedes that now he finds "death does press in on the writing, and it does become the subject, even if it's not foregrounded". Perhaps warming to this theme, a bit later he confides that the closing pages of The Children Act contain an explicit homage to James Joyce's greatest short story, The Dead.

The tone of this conversation might be sere and autumnal, but the writer remains very much himself, vigorous, resolute, engaged, with plenty of private merriment. He divides his time between a house in the country and a pied-a-terre in Bloomsbury, sustaining an energetic and quite demanding schedule. "I'm only 66," he protests. "My notebook is full of ideas."

McEwan has always conducted himself as if there were many other worthwhile things to do besides writing fiction. "Don't believe any novelist who tells you it's all agony," he observes with sardonic glee. "It's actually one of the most delightful, privileged existences you can imagine – assuming you can live by it." As a coda, he adds, "one of the great luxuries of civilisation is solitude, as opposed to loneliness."

Although comfortable with solitude, McEwan admits: "I couldn't write all the time. As a writer, you have to come out into the world. I don't have a Salinger or a Pynchon impulse. There are so many things to do that are interesting." Unlike many of his literary associates he looks forward to appearing at festivals. "I do half a dozen literary festivals a year. The pleasure of festivals is meeting up with other authors and friends." Strikingly generous in a milieu noted for back-biting, he stresses the pleasure he gets from "seeing the pals of my generation", especially Julian Barnes, Martin Amis and James Fenton among many, and warmly celebrates the "fellow feeling" of this fraternity.

McEwan can also afford to be comfortable among friends and family. For many years he was ever-so-slightly marginal, the outsider in a group of Oxbridge wits, more reticent and apparently less combative. Their prose had the flash of rapiers; his was a cut-throat razor. Bright swords inevitably rust. Now, with the passage of time, McEwan's prose keeps its cutting edge and his books are the ones the reading public still crave. The surge in his popularity possibly places him nearer Graham Greene than James Joyce on the Richter scale of literary vibration, but McEwan's back-list is a staple of the school syllabus. There's now a generation raised on the transgressive menace of his imagination, a world that invites the reader to trespass on the dark side of human experience, and play truant from normality.

McEwan himself likes to keep on the move. In the spirit of the Romantics, his complement, or possibly antidote, to literary life is walking. The Children Act is dedicated to one of his most constant walking partners, the neuroscientist and head of the Wellcome Institute, Ray Dolan. In the past several years, often with Dolan, he has tramped thousands of miles: in the Cotswolds and Chilterns, the Lake District, the Dolomites, the Himalayas and, most recently, the Scottish Highlands. "I love walking in the northwest of Scotland," he says.

McEwan with his wife, the journalist Annalena McFee. Photograph: Richard Gardner/Rex

How Scottish is he? McEwan drops into a summary: "My father was Scots from Glasgow. Working class. Lied about his age to enlist at 17. Worked his way up to major. Hardly ever went back."

Major McEwan's son, Ian Russell McEwan, born in 1948, was a late-empire child in the dying days of British colonialism. Growing up – an only child – he was always on the move: in Singapore and the Far East, then Germany and Libya, looking back at England from overseas, and finally stranded as a teenager in an all-boys boarding school. This curriculum vitae now looks like an ideal apprenticeship for a writer. In 2002, in a twist of fate that could animate any McEwan storyline, he discovered he had an English half-brother, an Oxfordshire bricklayer named David Sharp.

On the Scottish question, he observes that his second wife, former Guardian journalist Annaleena McAfee, is "passionate about Scotland" but that "psychologically and temperamentally I'm an English novelist. Annaleena is pro [independence]. I'm a pathetic Don't Know." He adds carefully: "I hope it works out if they say Yes, but I worry that it won't."

After this cautious political foray, he moves on to safer ground. "The Act of Union never extended to the imagination. There are many English, Scots, Welsh and Irish poets and novelists. But there is no British novel, just as there's no British football. In the arts, there's nothing British." In a slightly sarcastic coda, he notes that "When we invade Iraq, it's all about Britain. But when we get turned out of the world cup, it's England."

McEwan's buried family ties to Scotland are evocative of a complex union and, in another breath, he freely identifies "a strong Scots-English thing" in the comparisons made between his work and the writings of two great Scots storytellers, connoisseurs of suspense, John Buchan and Robert Louis Stevenson. The latter, whose literary mantra was "The only art is to omit", might be another covert influence on The Children Act. Like Stevenson, McEwan is comfortable placing lonely boys in extremis at the heart of his fiction. Adam Henry may be a Jehovah's Witness but he harks back to David Balfour in Kidnapped and even Jim Hawkins in Treasure Island.

At the same time, contrariwise, McEwan also wants to locate his novel in what he calls "the family division" of English prose, the "morally centred fiction of George Eliot, Conrad, James and even Austen". He has a point. McEwan certainly echoes Stevenson's mobile Scottish brevity but, in the best English tradition, he likes to explore an enclosed world as refined as ivory. Where Saturday was located in Fitzrovia and the neurological community around Queen Square, The Children Act takes place amid the booming chambers of Gray's Inn and the shadowy precincts of the justice system. Where once McEwan sat in on brain surgery, here he has immersed himself in the rhetorical interstices of the law. "I love professions," he exclaims with sudden animation. "I've always liked research, and I love people's expertise. That's why I like film sets. People love nothing better than to talk about their work."

He becomes momentarily didactic. "Work got written out of the modern novel. In late-20th-century fiction there are too many figures whose work is never quite clear. Yet work is where most of us will spend most of our lives." It is work that gives McEwan the ideal opportunity to look in as an outsider, a spectator and literary voyeur.

Ian McEwan in 2004 with old friends Martin Amis, right, and Christopher Hitchens, whose death in 2011 left 'one of the biggest holes imaginable'.

In The Children Act he describes Fiona Maye trying to "place an accent" in a fellow barrister, a familiar and deeply English habit: "A touch of cockney, a fainter trace of west country – the confident voice of a man who took his own competence for granted, well used to giving orders. Certain British jazzmen spoke this way, a tennis coach she knew, and non-commissioned officers, a senior policeman, paramedics, an oil-rig foreman, who had once come before her. Not men who ran the world, but who made it run."

"Non-commissioned" is the tell-tale here: this is the voice of McEwan, the empire child, looking back from afar, in fascination, at the resilient peculiarity of English life, and the officer class that administers it. Those years following the flag as the only son of a British army major have left their faint but unmistakable, traces. He remains an outside-insider. When the conversation reverts, as it inevitably seems to do, to the subject of death, McEwan conjures a memory from this peripatetic childhood.

"I was eight or nine, and we lived in a six-storey block of flats in Tripoli. On those lovely warm north African evenings I used to play in the dust on my bike with the other children, mainly American service kids. Then, as it began to get dark, my father – the sort of man I dared not disobey – would come out onto the balcony, and call – Iaaannnn! – and I would have to go indoors. I imagine death is a bit like that. After the end of the day, and the fun, you get to spend a lot of time indoors…"

McEwan has no appetite for such restrictions. "Life is still too interesting. I really want to know what's going to happen in the 21st century." Time, however, ticks on without remorse, carrying him with it. Passing another cairn on the long climb up Parnassus, McEwan has just sold his manuscripts to the Harry Ransom Center in Austin, Texas. "Bundling up my papers," he says, "has been another ageing thing." The library, conventionally a sanctuary, has become a sobering transit-lounge. At once dry and droll, he describes it as "the antechamber to death".

Out of the blue he remembers interviewing the late John Updike in his final years. "We talked about all this," he recalls. "He told me: 'The older you get the less frightening death becomes.'" He frowns in puzzlement. "I'm not sure whether to believe him."

So does he believe him?

"No." A beat. "Do you believe those obituaries that say, 'Died peacefully in his sleep'?" (McEwan was at Christopher Hitchens's bedside shortly before he died.) "Still, wouldn't be a bad way to go."

What, then, is the promise of old age? Apparently, for a writer known for paring his material to the bone, he will go on exploring the dynamics of brevity. "I'm interested in novellas," he says. "Originally I saw The Children Act as a novella, max 40,000 words." The conversation meanders into a review of classic short fiction: Heart of Darkness, The Turn of the Screw and Death in Venice. "I don't know," he goes on, "if there's a difference between a short novel and a novella. In the end, The Children Act came out at 55,000 words.

"I think," he continues, arriving at a provisional conclusion, "I will write, in my 70s, more novellas. I love the idea of sitting down to read something in three hours – about the length of an opera, or a long movie, or a play where all of its structure can be held in the mind. A novella is a great length, and it's a demanding genre in which things have to be settled quickly."

It's often said – even McEwan's friends have said this – that he is, au fond, a brilliant story writer, a sprinter not a marathon runner. According to the New Yorker, when Martin Amis was asked to name McEwan's greatest achievement, he replied: "The first 200 pages of Atonement." In the same piece, philosopher Galen Strawson characterised McEwan's books as "stories pushed into a novel". How does he respond to that?

"I haven't written a short story since 1976," he says evenly. "Someone once said that about Enduring Love." Memorably, this novel opens with a standalone episode involving a runaway hot-air balloon that has all the stomach-churning dread of his earliest work. "Ever since Enduring Love people have said, 'Your novels open with these amazing short stories.' Oh, really?" he challenges. "Do they? Does Atonement? Does On Chesil Beach? That line has become a kind of on dit. I really don't think it makes any sense to say that any more," he concludes, still a touch defensive, "but perhaps I might return to the short story. It might be the perfect form for old age."

So, no stopping? "It's hard to stop. What would you do all day? Answer emails? Go for walks?" McEwan picks Philip Roth and his celebrated end-run of titles (The Plot Against America, I Married A Communist, American Pastoral, The Human Stain) as the benchmark for late flowerings, adding with approval, "He was 80 before he threw in the towel. Now, I believe, he micro-manages his biography."

Does McEwan have a pocket Boswell sharpening his pencil in the wings?

"No."

What about posterity?

"The flipside of death?" A thin smile. "Oblivion? Well, I am quite interested in the way that when writers die their reputation goes into free fall – and then comes back, if you're lucky." He reflects on writers such as Philip Larkin who've made that comeback, and others such as Angus Wilson, his former mentor and tutor, whose posthumous career has been a prolonged rallentando "with endless efforts to revive him", and also the fate of his near-contemporary Bruce Chatwin. "Bruce doesn't come up much now," observes McEwan with finality. "Generally it's not a good idea to die."

Meanwhile, in the posterity stakes, he'll take his chances. "I don't expect to be watching from afar," he jokes.

And if he had to choose between his books and his family? There's no hesitation. "Family. I adored having children. Work and fatherhood have kept me sane. The impulse to work is like a survival instinct."

McEwan at home projects an accomplished aura of self-fulfilment but his native detachment invites a question about alternative strategies. If not novels, what else? "After my first stories were published," he answers, "I wrote a television play, then a movie, and then a novel, and then [the film] The Ploughman's Lunch, and I thought, 'This is the perfect life.' I still have this yearning for an alternative life in parallel. The busy-ness, the urgency, and the extended panic. I used to say I wanted to be the bass player in a Grateful Dead tribute band."

He slips from teasing into seriousness. "What I really would have liked to have done would have been to make movies – as a writer-director." There's another long pause, as if his imaginary career in films is spooling out in front of us. Finally he smiles: this conversation has run its course. "How to be happy? Big question. I still think Freud got it right," he instructs. "Good health. Interesting work. Satisfying personal relationships. It's worth checking every now and then to see how you score on all three."

• This article was amended on 1 September 2014 to attribute a quote to Galen Strawson, the philosopher.

The Children Act is published by Penguin at £16.99. Click here to buy it for £13.59 with free UK p&p