A few months ago, I was coming out of the lavatory at Maison Bertaud, a fusty old patisserie in Soho, when I saw the familiar full-moon face of Simon Callow – actor, playwright, director, indeed all round homme de théâtre – eclipsing the window. For a moment I experienced the giddy thrill of fandom, and had to restrain myself from striding across the room and pumping him by his hand while exclaiming, "You're Simon Callow, aren't you?" I had to restrain myself not only because I dissent from the assumption that notoriety is a licence-to-accost, but also because this rhetorical question would then have mutated into the no doubt unwelcome encomium, "I just love your talking book of The Twits".

Who knows, perhaps Callow would be completely relaxed about my regarding his reading of Roald Dahl's children's story as his chef-d'oeuvre, but somehow I doubt it. After all, not long after this near encounter I saw him effortlessly upstage Ian McKellen and Patrick Stewart, playing Pozzo to their distinctly sweet-cured Estragon and Vladimir in a grown-up production of Waiting for Godot. But while Callow's Pozzo may have been magnificent, no recollection of it will ever fill my heart with joy the way his fabulously orotund declamation of the opening lines of The Twits does.

It helps that it's such a very good entrée: "What a lot of hairy-faced men there are about nowadays. When a man grows hair all over his face it is impossible to tell what he really looks like. Perhaps that's why he does it. He'd rather you didn't know." To my mind Dahl's flatly authoritative statements have a universal sweep and psychological penetration to rival the first line of Tolstoy's Anna Karenina, with the added bonus of actually being true; I mean, there are undoubtedly many happy families that are altogether unalike, while – speaking with all the authority of the recently barbellate – I can assure you that when a man grows hair on his face, he definitely has something to hide.

The Twits is my favourite Dahl book – although I'm not sure that I can entirely divorce my enthusiasm for it from Callow's performance. The celebrated midden of Mr Twit's beard, his insertion of his glass eye into Mrs Twit's beer, her substitution of worms for his spaghetti – not forgetting the dreaded shrinks; indeed, all of the creepy mind games the ghastly duo engage in strikes me as a more veridical portrait of a marriage than anything ever staged by Strindberg. That the whole tale of spousal, child and animal abuse should be punctuated by lines that skilfully and inexorably tighten the dramatic noose is testimony to Dahl's genius as a fashioner of narrative, when the aside comes: "We can't go on for ever watching these two disgusting people doing disgusting things to each other." The only possible reply is: "Yes, we can – please, more!"

Certainly the appetite for Dahl's children's books seems undiminished in the 19 years since his death. They sell in huge quantities, and the linkage of the sprightly texts to Quentin Blake's spikily effective illustrations is as secure in the popular consciousness as any horse and carriage – or Dumbledore and Harry for that matter. Not that everyone exactly likes Blake's line; it's more that his shtick – an early form of the now commonplace "adult child" style – is a good enough visual simulacrum of Dahl's imaginary world. As in Blake's drawings, there are big white spaces in Dahl-world where any realistic detailing might well be shaded in by a lesser writer; and again, in common with Blake's vision, Dahl-world is at once lurid and curiously ill-defined. The passions are strong and clear – fear, hatred, avarice, love, greed (especially for sugar) – but they are played out against a backdrop that is only wonkily apprehended.

Dahl mimicked to perfection a believable child's-eye view, that, looking up from below, sees the adult realm as foreshortened, and adult foibles as grossly elongated. I say this with confidence, but I'm not the ideal critic of Dahl's children's writing, for the simple reason that I was never exposed to it during childhood. In theory, I could've read – or been read – James and the Giant Peach (1961), Charlie and the Chocolate Factory (1964) and The Magic Finger (1966), but by the time Dahl published Fantastic Mr Fox (1970) I was out of the zone. Children's literature is in this respect rather like pop music – until it reaches classic status it has a rigid generational stratigraphy.

No, the Dahl of my childhood was the grown-up author of Kiss Kiss (1960), Switch Bitch (1974) and Tales of the Unexpected (1979). This last short story collection gave its name to a long-running television series (nine seasons in all, between 1979 and 1988) that initially dramatised Dahl's tales but latterly used his basic format to generate and adapt many more, something in the manner of the 1950s Twilight Zone. Indeed, the lounge-lizard signature tune of the show, and its title sequence of silhouetted naked Princess Di mop-crops cavorting in front of a roulette wheel is synonymous in my memory with a louche demimonde, at once attractive and repulsive. In my teens, I thought Dahl's short stories pretty cool: they were lucidly macabre, full of cod-scientific details, often world-wearily sexy, and invariably ended with an unforeseen twist, thrust like a stiletto through the reader's psychic armature.

That being said, when I began thinking about this piece and tried to recall some of the stories, I drew almost a complete blank, apart from one about a middle-aged woman whose ex-husband convinces her that her smoking and drinking have rendered her frigid, and so drives her to suicide. This turned out to be "The Last Act" from Switch Bitch, and when I reread it I found a rich seam of misogyny and sadism. In "Royal Jelly" (in which an obsessive apiarist cures his own infertility with royal jelly then feeds it to his languishing baby daughter) I discovered a mordant authorial delight in not so much subverting as completely inverting the unconditional love of good parenting. True, when I started scanning the Dahl collections I was able to marry tale to twist, but mostly it was the twists alone that had stayed with me: the murdering wife feeding the weapon – a frozen leg of lamb – to her policeman husband's colleagues in "Lambs to the Slaughter" was far more memorable than the mise en scène, let alone Dahl's prose, which was never much more than workmanlike.

I was once with Martin Amis when he was asked if he'd ever consider writing a children's book. He thought for a few moments before drawling: "I might . . . if I had brain damage." I don't take that view – for me a great children's book transcends the age group of its intended readership as completely as a great science fiction or detective novel transcends its genre. And it takes a fully engaged writer to write one; after all, Lewis Carroll may have been a repressed paedophile, but he certainly wasn't cognitively impaired. With Dahl there does seem a case for suggesting that it was the beneficial loss of at least some of his faculties that transformed him from a so-so writer for adults into a masterful one for children.

I'm not too interested in raking over the Dahl biography in search of a psychoanalytic perspective on his work – plenty of others have been there before, and the writer seems to have been as mercurial, if not Manichean, a figure as we would expect. By some accounts he was a deeply altruistic man, responsible – among other things – for designing a valve to release the pressure on the skulls of hydrocephalic children when his son, Theo, was afflicted with the condition after a car accident. According to others he was a fulminating misanthrope, whose late excursion into political comment earned him the reputation of an antisemite (unjustified in my view). Was he the caring husband, who took over the rehabilitation of his wife (the actress Patricia Neal) after she suffered a series of cerebral aneurysms – or the callous adulterer who had an affair with one of her close friends, Felicity Crosland, before leaving Neal to marry her?

Then there's the vexed question of what sort of parent Dahl had been himself; with AS Byatt's A Children's Book shortlisted for this year's Man Booker prize, the richly ironic cloth of the dysfunctional families of the great purveyors of children's literature has been spread before us: a counterpane world of suicides, mental breakdowns and addictions. As for Dahl, was he the delightfully playful, loving and inventive paterfamilias that some of his five children have recalled – or the crushingly dominating patriarch, subject to terrifying rages that looms in others' accounts? Standing 6ft 6ins, was Dahl not so much a BFG as a Big Angry Giant?

Nor am I keen on probing further back, to the miserable boarding school boy's separation from his doting mother, or the early death of his father. Dahl may have got some of the furniture for his children's stories from testing Cadbury's products at school, his hated masters and the folk tales of his Norwegian grandmother – but if a children's writer doesn't gather material from his own childhood, where else is there?

I'm more interested in Dahl's children's fiction as a perfectly achieved analogue of his distinctive worldview, with those elements that are wholly unsuitable for children obliterated – as if by some insult to the brain – yet their ghosts informing what remains. Dahl had a good war: an RAF fighter pilot, he achieved record numbers of "kills", and his early short stories traded on his wartime experiences. And yet can there ever really be such a thing as "a good war"? Even from the air the obliteration of human life is a vile business, while especially from up above, the pretensions of human morality are miniaturised. In Dahl-world, political institutions are shoved well to the background, while the notion of an orderly society is never seriously entertained: at best we have a little community anti-authoritarianism, as in Danny, the Champion of the World.

Take Dahl's most famous work, Charlie and the Chocolate Factory: its boy protagonist's family is immiserated, his carers are incapable of looking even after themselves – so salvation comes through luck, and the arbitrary beneficence of a deranged feudal-capitalist with a happily mancipated workforce. Of course, the spur that initially drives Charlie on is a lust for sweet things that, were it transferred to an adult plane, could only result in a work entitled something like "Charlie and the Huge Seraglio full of Compliant Nymphomaniacs". The NAACP slated the black pygmy Oompa-Loompas in the original text – and so Dahl changed them to a fictional light-skinned subspecies – but he couldn't get rid of the brown sugar. My equation of sweets with sex is not facetious; in Dahl-world, oral gratification is pretty much the only thing that matters.

The misogyny that haunts Dahl's adult writing is also short-circuited in his sexless children's fiction, for here his heroes and heroines can be either orphans (James of giant-peach lusting, for instance), or else, as in The Magic Finger, an unnamed, and thus family-free, eight-year-old girl. The eponymous heroine of Matilda has parents who are neglectful to the point of being abusive. Sophie in The BFG is extracted from her natal home in order to experience good and bad surrogacy – from giants (ie Dahlesquely huge men). The infanticidal witches of The Witches stand proxy for all mothers – who kill that which they claim to love; true, the boy's Norwegian grandmother is a good enough parent, but then she's safely de-sexed by age and illness. Only Danny, the Champion of the World presents an idealised parent – and that, also, has to be a man.

Danny is the most naturalistic of the Dahl children's books, and its evocation of rural English life in the 1950s gives us more substantial clues to its author's political perspective than any of the other, more cartoonish works. True, there are the satirical personifications of consumerism (Veruca Salt) and mass media (Mike Teavee) in Charlie, while even in a squib such as The Giraffe and the Pelly and Me we get little vignettes of a ruptured organic society – but it's only in Danny that the whole Red Tory cloth of his convictions is unfurled. With its portrayal of happy poaching folk, existing in impoverished but mutually supporting symbiosis with a rough-shooting aristocracy, Dahl registers his wholesale revulsion from the modern world.

That Danny's father, William, is a pretty much perfect single parent, raising a boy to be self-sufficient as well as loving, is as much a function of the support network that surrounds their Romany caravan – the doctor, the policeman et al – as his own particular qualities. In Danny there's no such thing as family – only a small and male society. Reading Danny one thinks of the milieu of the Men of the Trees, and other proto-fascist eco-movements of the British interwar period.

Danny is also noteworthy for another of those strange ambivalences that permeate Dahl's work: while The Magic Finger is an out-and-out assault on blood sports, this book seems to embrace the killing of game, so long as it's perpetrated in a sustainable fashion. I said I wouldn't wax Freudian in my view of Dahl, but I can't help myself: I think it safe to see blood sports in his children's books as a synecdoche for all human violence. Dahl seems to suggest that there are good ways of killing things – the just war, the poached pheasant – and emphatically bad ways: the holocaust of pheasants that the grotesquely plutocratic brewer, Mr Hazell, is intent on unleashing.

This brings us fairly logically to Fantastic Mr Fox, the latest film adaptation of Dahl. It's a distinguished addition to a filmic canon that includes superb versions of The Witches and Matilda – although there are also notable clunkers such as Danny, the Champion of the World with an effete Jeremy Irons as the ideal parent. Filmed using stop-motion techniques, Fantastic Mr Fox looks beautiful: a vivid appliqué of pose-able figures threaded into sets that any kidult would love to play with. It's a relief to experience a children's film that has a genuine quiddity after all that remorselessly perfect computer-generated imagery. As for the players, I wonder whether George Clooney (Mr Fox) and Meryl Streep (Mrs Fox) have ever generated more sexual chemistry than they do with these husky, sassy voiceovers.

I took my eight-year-old to the screening. He's the youngest of our cubs, and the last one available with whom to burrow about in Dahl-world. His remark on the adaptation was telling: "I like it when they do the whole story properly, but then they put in anything else they feel like." Luckily, the "anything else they feel like" is in this case cleverly scripted by Noah Baumbach (of The Squid and the Whale) and director Wes Anderson. I had no problem with all the animal characters being snappy late 1950s American types, not unlike the advertising men and their wives in Mad Men; nor did I mind the introduction of a yoga-practising, karate-kicking fox nephew, with whom Fox Jr has a troubled rivalry. Nor, indeed, did I suspect any sinister subversion of the special relationship in Anderson choosing a fine trio of British character actors for the loathsome farmers, Boggis, Bunce and Bean (the last giving Michael Gambon an opportunity to smoke heavily on screen, albeit in puppet form). After all, while it's a great bedtime book, Fantastic Mr Fox would be slim pickings for a feature had a flickknife-toting rat (voiced by Willem Dafoe) not been interpolated.

All that's fine, and it's refreshing also that Anderson doesn't play up the convenient truth of environmental damage by humans, but rather subverts it by making this the cause – as it indeed is – of an advantageous vulpine incursion into the built environment. However, the recasting of the Foxes' marriage as one of near-equality, with Fox himself not so much a wild animal as a grandiose chicken-snatching addict, and Mrs Fox scratching against her own co-dependency, was certainly not true to the spirit of the book. For Dahl, Mr Fox was the fantastic one – he didn't need any lousy vixen to show him the way to dig.

I can't claim to have read every Dahl children's book, but those I have read, I've read aloud. It might be an idea for all literary critics to read the books they analyse aloud – it certainly helps to fix them in the mind, while providing a readymade seminar with your audience. The Dahl books I haven't got on with – Esio Trot and The Minpins spring to mind – were ones my kids didn't like either. The reason I never finished The BFG was that I kept falling asleep while reading it; the eponymous hero of the book is meant to give children dreams – or nightmares – but to me he simply gifted oblivion.

But if my view seems to be of a jaundiced introjection of adult attitudes into children's entertainment, please don't imagine for a second that I find this remotely objectionable. I don't adore Dahl's children's fiction in spite of its submerged misogyny, lust, revanchism and wilful neglect of identity politics – I love it precisely because of these attributes. Dahl understood intuitively the truth best exemplified by a famous scene in The Simpsons, when Bart overhears Marge saying to Homer "Kids can be so cruel" and, taking it as an injunction, cries out: "We can? Thanks Mom!" There then comes the sound of his rapid footsteps along the hall, followed by Lisa's pained cry: "Owwww! Bart, cut it out!" Dahl's books resound down through the generations with the demented call "We can!" and its pained response "Owwww!" And long may they do so.

• This article was amended on 23 October 2009. The original said that Charlie Bucket was an orphan. This has been corrected.

Fantastic Mr Fox is released on 23 October.