As Quentin Crisp once pointed out in a lecture: if he were to bring a distinguished old Yorkshireman on stage, the audience might be perplexed; but if he brought a polished abstract sculpture with a hole in the middle, the audience would cry out, “Ah! Henry Moore!” So AA Milne’s long career as poet, playwright, polemicist, peace campaigner and novelist is completely eclipsed by four short children’s books which, as he put it in 1952, he created, “little thinking / All my years of pen-and-inking / Would be almost lost among / Those four trifles for the young”.

That “almost” is no longer needed. Pooh is one of a tiny handful of creations that are so enormously successful we forget the infelicity of their names: Boots, the Beatles, Star Wars, Winnie-the-Pooh. One of the great secrets of success is that, more often than not, it is not the kind of success you were hoping for. You want to be Hamlet but you’re hailed as a clown, and now you can never be any kind of Hamlet. You want to move on but your global hit exerts all the gravity of a planet and you are trapped in its orbit. Failure at least has the comfort of hope. Milne’s life story brilliantly illuminates what it feels like to be tested by huge, unlooked-for success.

Watch the official trailer for Goodbye Christopher Robin

It isn’t easy. Frankenstein was so eclipsed by his own creation that it has robbed him of his name. Milne had a long, successful career in the theatre – a world in which the writer becomes accustomed to a certain amount of petting and caressing. He gets to hear the audience call “Author! Author!” No one did that at Pooh events. They wanted to see the bear and – more troublingly – the boy.

Milne isn’t of course the only writer to find himself swallowed up by his own creation. You could say his friend and hero JM Barrie wrote with great commercial success after Peter Pan, but what does “after Peter Pan” mean? Peter Pan was, is and always will be; Barrie’s other works are of their time. The overarching drama of the Sherlock Holmes stories is the great detective’s struggle, not with Moriarty, but with his creator’s attempts to kill him off.

The tragedy of Milne’s success is that it trapped a real child in time like a fly in amber

Milne carved out his career in the suave, seductive world of London’s clubs in the 1920s and the green rooms of Shaftesbury Avenue. It’s good to be reminded of just how long and demanding an apprenticeship he had served before he discovered Pooh. His peerless dialogue has its roots in his playwriting career.

His study of classics and his work on Punch had given him the extraordinary ease and range he shows in the poems. Milne’s are probably the last poems written that really cry out to be memorised and recited. I can’t think of one more easily absorbed and enjoyed than “Disobedience”, nor one that captures so perfectly this childhood terror.

It’s good to be reminded, too, that Pooh was not universally adored, that writers who had admired Milne’s lightness of touch turned on what they saw as the mawkishness of Pooh. “Tonstant Weader,” said Dorothy Parker, “twowed up.” “Timothy Bobbin,” wrote PG Wodehouse, “goes Happily hoppity hoppity hop.” Cruellest of all is Richmal Crompton’s brilliant skewering of the cult of Christopher Robin in the poem “Homework”: “Anthony Martin is doing his sums.” Public adulation is salt in the wound of a writer who has lost the admiration of his peers. Milne’s letters to his brother Kenneth uncover the wellspring of his creativity with all its childish joys, shadowed by tragedy. They are a real find. Of course, when I was asked to write a film about Milne I left that out.

Carefree days … Will Tilston as Christopher Robin in Goodbye Christopher Robin. Photograph: Allstar/Fox Searchlight Pictures

Biography looks for what makes the individual different; drama looks for what we have in common. You can sell a million books if you write a good story well but a cultural phenomenon like Pooh needs something else. It needs to touch a raw nerve. That terrible review by Parker also covered Christopher Morley’s children’s book I Know a Secret – which really is a pile of mawkish mush. There was a fashion for sentimentalising children on which people like Morley successfully cashed in. Milne on the other hand searched it for its source and found something true and terrible and enduring.

The House at Pooh Corner stands in a glade between two dark shadows – the aftermath of one war that had just finished and the dread of one coming. No one who fought in the first world war knew it was the first world war. On the contrary, they had been told that they were fighting the war that would end all wars. It must have been with the most bitter irony and failure, then, that Milne’s generation watched their children march away to a war that they had been told would never happen. The Milnes received that dreaded telegram telling them their son was missing in action and presumed he was dead. This could have happened to anyone; this was feared by everyone. It’s there – something you can build a film around. It’s the shadow that makes the carefree days in the Hundred Acre Wood tremble and shimmer with their own fragility. They are suffused with a sense that happiness is possible and valid even though we know it is short-lived. It’s a feeling that is expressed with peculiar intensity in the political situation of the between-the-war years, but which applies to everyone everywhere all the time.

It is there, too, in the child who plays in the woods – in Christopher Robin himself. On the one hand he is Robin Hood revelling in the freedom of the Greenwood but he’s also a babe lost in the wood. What marks Christopher Robin out from other children in literature – from William, say – is that he’s often absent from the adventures. Often his role is to come and put things right. He’s more like a kindly uncle than a child. Through the carefree forest he carries a burden of responsibility.

Domnhall Gleeson as AA Milne in Goodbye Christopher Robin. Photograph: David Appleby/Fox

The other unusual thing about Christopher Robin of course is that he was – to some extent – a real boy. The difference between Winnie-the-Pooh and, say, Sherlock Holmes, is that Pooh did not just swallow Milne, it also swallowed Christopher Robin. Imagine if Barrie had called Peter Pan Peter Llewelyn Davies. When Czeslaw Milosz said, “When a writer is born into a family, the family is finished”, he meant that a writer will betray his parents and siblings. Milne on the other hand – however innocently – betrayed his son. The magic of the Hundred Acre Wood is that it takes something painfully fleeting and makes it stay for ever. The tragedy of Milne’s success is that it trapped a real child in that moment like a fly in amber and made it almost impossible for him to become that thing that every child wants to become – a grownup. Is there a threat more pathetic and painful than Christopher Robin’s cry “We’ll see how father likes it when I write poems about HIM”?

For all its shadow, what really abides about this story is the light, the sense that happiness – no matter how fleeting – is real. The fact that we are all moved and enchanted by the Hundred Acre Wood, that it calls to us, is proof that these passing moments are as real and essential as the more solid and enduring things with which we surround ourselves, that we find in them something true and paradoxically enduring, even eternal.