“I was recognised in Prague, once,” says Philip Pullman. The author, who turns 73 next week, is remarkably unassuming for one of the best-selling British novelists this century, though his fame has at times caught up with him. “We had to move house,” he admits as we sit in his dog-eared parlour in Oxford, coffee perched on a great island of books. “Somehow people had got the address and kept turning up at the house.” On one occasion, a Benedictine monk from Missouri appeared at his door, like an apparition from his own pages. “He’d climbed the hill on a blazing hot day in full black robes, so we took mercy on the poor guy and gave him a drink.” I ask whether the monk had a bone to pick: “Not at all! He wanted me to sign his books.”

All things considered, Pullman is on good terms with the church. His Dark Materials, his three-part epic of the ’90s, told the story of two children – Lyra Belacqua and Will Parry – and their battle against “the Magisterium”, a global church of terrifying power and cruelty. Over the course of Northern Lights, The Subtle Knife and The Amber Spyglass, Pullman skewered Christian dogma so sharply the books were deemed “worthy of the bonfire” by a Catholic newspaper. “People occasionally write books saying I’m going to hell – you take no notice of that – but I’ve met nothing but courtesy and goodwill from clergy-people,” he says. “I think they can tell I’m taking the matter seriously.” These words could well be his personal artistic manifesto. “If you’re engaged in a long task, which you know will take you some years,” he says, “I don’t think you can go about it frivolously.”

Pullman is a high-minded writer and a dazzling interviewee; quotations from arcane philosophers are never far from his fingertips. But it would be wrong to characterise him as an Oxbridge bookworm whose life has been spent in studious isolation. He was nearly 50 when he found fame as an author. Most of his career was spent teaching in schools, playing music and practising woodwork, while writing children’s books on the side. He speaks with as much enthusiasm about the history of science as he does about trees, his collection of ukuleles and the Batman and Superman comics he discovered as a child in Adelaide. “You couldn’t get them in England then,” he recalls. “I immediately fell in love with the form.”

While still regarded as one of Britain’s foremost atheists, these days Pullman has bigger fish to fry than mere Christianity. The Secret Commonwealth, published this month as the second instalment of a new trilogy titled The Book of Dust, sees Lyra Belacqua’s world subtly changed. The Magisterium has mutated into a devious and bureaucratic beast, allied to rapacious corporations and led by charming and serpentine plutocrats. At its head is Marcel Delamere, a suit-wearing Svengali who admits his desire to “undermine the idea of truth”.

But there’s another more shadowy and surprising villain here – the blind worship of reason. To explain, Pullman turns, as he often does, to the writings of William Blake. “My one enemy is what Blake called ‘single vision’,” he says, citing the poet’s condemnation of narrow thinking. “In the first trilogy I suppose that was the Magisterium, with its dogmatic insistence on religious truth. But the belief that everything can be dissected and measured is another kind of ‘single vision’.”

In the new book Lyra, now 20 and a student at Oxford, has been hoodwinked by a pair of hip philosophers whose books preach a disgust for mystery and wonder. She has put away her childish things, a canny twist of her character arc. “His Dark Materials was quite popular,” Pullman says, 17 million copies later, “and I felt liberated by the knowledge that many readers would be returning as adults to a world they met as children.” For Pullman’s now-grown readers, Lyra’s predicament may offer a poignant reminder of how much things have changed since the ’90s; how far we have travelled from her and from ourselves.

“I don’t write books in order to illustrate the problem. Far from it. I discover the problem in the course of writing the story.”

“To have the most interesting events of your life over by the age of 12 is a bit rough,” says Pullman of Lyra, explaining why he felt the need to return to her story, two decades on. By the time readers open The Secret Commonwealth, Lyra has already reached the North Pole, fought her parents, saved the world and killed God. We’ve seen her get drunk for the first time, fall in love and undergo a sexual awakening that was memorable for its emotional power. “I wanted more for her,” says Pullman, “and it struck me as entirely right for Lyra to go through this stage of scepticism, then a sense of self-betrayal, and then a gradual rediscovery.”

Even Pullman, a lifelong dealer in fantasy and wonder, suffered these adolescent upheavals. “I vividly remember my own experience of these discoveries,” he says. “It’s intoxicating, when you’re 15, to reject the stories you’ve been told. Then the alienation, agonies, doubts…” Like Lyra, the author led a peripatetic childhood, following his fighter-pilot father from Norfolk to Wales to Africa. Like Lyra’s father, his was a complex and heroic figure who died in an air crash. The seven-year-old Pullman’s adventures continued. “We sailed to Australia in 1954,” Pullman recalls, “by way of Aden, Mumbai and Colombo.” This time, the family was following his stepfather, also an airman, who was working on a new kind of unmanned Australian aircraft named the Jindivik.

“We came back by sea, but not through the Suez Canal,” he says. “We had to sail around the Cape of Good Hope, because by then Britain and France had engaged in the ridiculous, Brexit-scale catastrophe of a war with Egypt.” There is an echo of this roundabout journey in The Secret Commonwealth as Lyra navigates a Middle East fraught with political turmoil.

Pullman is a mild-mannered and humorous raconteur, but his contempt for politics remains furious. The former British prime minister David Cameron “will never be forgiven” for calling the referendum on Brexit; the current leader, Boris Johnson, has “an airy disdain for the truth”. Donald Trump, Scott Morrison and Jair Bolsonaro are deplorable in equal measure. Pullman expresses a wish to see the British constitution rewritten, the voting system overhauled and the Houses of Parliament bombed. “I’d evacuate everyone first,” he says. This sense of Lyra-like rebellion occasionally sparkles in Pullman’s conversation. He is, it goes almost without saying, an admirer of environmental activist Greta Thunberg.

The history of English children’s books is crowded with the horror of growing up: Wendy betrays Peter Pan by becoming an adult, Christopher Robin goes off to school and Alice endures the changes in her body, while the children of the Narnia books escape the same fate only because of a convenient train crash. In His Dark Materials, Pullman’s most radical flourish was to reject this idea of adolescence-as-tragedy: Lyra experiences the discomforts and doubts of growing up, but these don’t signal the end of her adventure, they’re a thrilling part of it, which she navigates with her typical energy and spirit. “People have asked me why I thought young children would be willing to read such a long story,” says Pullman, “but I think it’s because they like Lyra. She doesn’t understand it all either, but she’s willing to explore and they go along with her. As she finds things out, they find them out as well.”

Despite his career in the classroom, Pullman forcefully rejects the idea his books are educational or didactic. “I don’t write books in order to illustrate the problem,” he says. “Far from it. I discover the problem in the course of writing the story.” Certainty is suffocating, both for the storyteller and his world, and so his new book celebrates the unexplained. “I salute the great scientists with unbridled enthusiasm,” says Pullman, “but there are some things that can’t be measured. Love, grief, the feeling of being watched.”

Along with the daemons and witches of Lyra’s world, these phenomena belong to the Secret Commonwealth of the book’s title – a catch-all term for the supernatural, the preternatural and the fabulous. Even within his own authorial practice, the so-called Secret Commonwealth is at work, with significant plot elements revealed to Pullman as if through the generosity of supernatural forces. “I didn’t realise Marcel Delamere was Mrs Coulter’s brother,” he tells me of his two arch villains, “until I realised I had given them the same name.” Sure enough, the glamorous and merciless Mrs Coulter was born Marisa van Zee – her last name a Dutch version of the French Delamere.

The book tracks through a complex maze of human emotions that – in a typically Pullmanesque flourish – the author decides to illustrate using his two poodle-spaniel crosses, which he had ushered from the room before our interview. “When I see my dogs, I see them not as canine quadrupeds, but as members of the family, for whom I feel a great affection, tempered by irritation and the knowledge that they’re probably not going to live as long as me,” he says. “That’s part of what the word ‘dog’ means. That’s part of the way we live.” He returns to his beloved poet: “It’s ‘twofold vision’; Blake says we mustn’t abandon this.”

Even students of literature cower from Blake, the visionary English poet and printmaker whose rhapsodic writings have puzzled scholars for two centuries. But Pullman’s writing is remarkable for the esteem in which it holds even younger readers and their ability to grapple with heavy material. The latest instalment in Lyra’s story is no exception. For all its vivid colour, its globetrotting adventure and its carnivalesque cast, the book once again invites Pullman’s readers – children and adults – to confront themes of startling intellectual weight.

“Dust…” he says, his wonder at the word still palpable after years under its spell. “I think part of the reason I’ve returned to the story is that I wanted to know more about dust.” This mysterious molecule emanates from certain beings and objects in Lyra’s world, and in Pullman’s mind has resolved itself into a metaphor for consciousness. “We know we’re conscious,” he says, animated by the thorny topic, “but philosophers are still struggling with what David Chalmers calls ‘the hard problem’ – how do we get from matter to consciousness?” He is currently gripped by the concept of “panpsychism” – the theory that everything is conscious, not only human beings. “Once you accept that consciousness is a normal property of matter, and that everything is, in however dim and rudimentary way, conscious, everything becomes clearer. This is the question towards which The Book of Dust is working.”

Besides the new book, this autumn brings another gift for Pullman’s readers: a feverishly anticipated small-screen adaptation of the His Dark Materials trilogy, filmed in long form – Northern Lights alone will run to eight episodes. “I’m very excited about the prospect of that,” Pullman says. “You can’t tell any long story in two hours.” He knows this from experience. The 2007 adaptation The Golden Compass failed to impress his fans, and the author himself. “You used to be told that putting your book on the screen would open it out, but it’s the opposite,” he says. “The film had a terrific cast, but it failed because of the compression.”

The new adaptation is equally star-studded, with James McAvoy and Ruth Wilson in leading roles. Pullman’s favourite actor, the “excellent” Clarke Peters, plays the master of Jordan College, while the heroic airman Lee Scoresby – a character of obvious importance to Pullman – is played by the American composer Lin-Manuel Miranda. “I was rather surprised by that casting,” says Pullman, “but he looks and sounds wonderful.”

For now, though, he has the final part of The Book of Dust to finish, in which hairier battles are still to be fought. Lyra must traverse an unforgiving desert and – spoiler alert – reconcile with her estranged daemon Pantalaimon, the animal embodiment of her soul. She doesn’t fully understand the mysterious world she inhabits, nor do her loyal readers, nor even the erudite novelist who created her, but therein lies some of its magic.

“I don’t know how it will end, but I know it all comes down to Lyra,” says Pullman. After 2400 pages and 25 years of telling it, his passion for her story is clearly undimmed. I ask whether he sees himself in the brave young girl of his own creation. “Yes, probably,” he replies, quietly, “I hope so.”