There are a few books for children that operate as a type of personality test. One is The Wind in the Willows: every person is a Ratty, a Mole, a Toad or a Badger. Another is Rudyard Kipling’s The Jungle Book: in each of us is a constellation of Baloo, Bagheera, Mother Wolf, Kaa and Shere Khan: the book is a kind of Myers Briggs personality test for the childhood soul.

Many, perhaps most, people first come to The Jungle Book through the 1967 Disney animation. There’s a great deal to admire about the film, but it’s a very different beast; Walt Disney famously gave one of his screenwriters a copy of the book, saying: “The first thing I want you to do is not to read it.” The film is sunlit; the book is darker, sly and knowing where the Disney is optimistic, written with human venality full in mind. As a child I was drawn to the bookprecisely because it did not attempt to convince me that we are better than we are and thereby wrangle me indirectly into good behaviour. Mowgli was what I knew, at the age of eight, the best children were: stubborn, boastful, loving, egotistical, loyal, brave and wild. You wouldn’t necessarily like Mowgli were you to meet him on a bad day, but you would not be able to ignore how intensely alive he was.

When I was asked to write a book set in the world of Kipling’s Jungle Book, I accepted with alacrity, despite knowing that Kipling would probably loathe the idea of someone else playing with his work, and given what we know of his politics, I would not be his first choice. Nor would he be mine; as a supporter of the British empire and of theBoer war, in which thousands died in concentration camps, Kipling is a deeply troubling figure, and was considered controversial and difficult even by his contemporaries. But as with Richard Wagner, his characters have taken on a life that extends beyond the man himself. I accepted, because the desire to take up a story I have loved since I was eight years old was too strong to resist.

I was not alone. There has been, in the last decade or so, a spate of writers taking on the classics of children’s literature. Some have re-entered the original fictional world; Pamela Butchart, known for her ludic, joyfully unhinged The Spy Who Loved School Dinners, has just published a Secret Seven story, set in the same time and place as Enid Blyton’s original; Hilary McKay’s Wishing for Tomorrow is the story of Miss Minchin’s Select Seminary after the events of A Little Princess have passed, and the multi-award winning novelist Geraldine McCaughrean won a competition, run by Great Ormond Street Hospital, to write an “official sequel” to JM Barrie’s Peter and Wendy.

A different beast … Disney’s 1967 film The Jungle Book. Photograph: Alamy

Others have modernised existing stories. Jacqueline Wilson’s Katy, a present day version of What Katy Did, reshapes the ending so that Katy does not throw off her spinal injury when she has learned grace and sweetness. Frank Cottrell Boyce’s Chitty Chitty Bang Bang Flies Again takes only Chitty itself from Ian Fleming’s original 1964 text and reintroduces the car into a modern family.

Writing inside a much-loved story is, inevitably, a daunting job, though Cottrell Boyce’s task was possibly made less so by the fact that he is an exceptional writer for children and Fleming, for all his talents elsewhere, was quite markedly not. The original book is surprisingly sparse and blunt; most of the lunatic details that make the 1968 film such a strange, vertiginous triumph – the Child Catcher, Lionel Jeffries’s erratic Grandpa Potts – were added by Roald Dahl, who co-wrote the screenplay.

It may be that behind the boom lies a desire to save the old stories, which are read less and less every generation, from dying out and to inject them with new life; it might be, less nobly, a desire to take control, as an adult, of that which one coveted as a child – it has a touch of megalomaniacal glee in it. The fashion coincides with the explosion of fanfiction across the internet. Beginning with Star Trek fandom in the 1960s, it rapidly expanded to include both classic and modern fiction. The cliche is that the majority of fanfiction is lovingly described sexual liaisons between Harry Potter and Draco Malfoy, but it extends into every genre, and is at once an act of homage and an appropriation of narrative power. It is evidence that fiction, despite the stringent legalities of copyright, is a slippery beast, and cannot belong to anyone.

I did not expect writing my way into the world of the jungle to be easy. In part, I was dogged by fear of other people’s fury; The Jungle Book is part of the cultural landscape, and Mowgli is not mine, nor Kipling’s, but common property. My book Into the Jungle is a series of origin stories for each of the main animal characters, and, for some, the backstory I have imagined for Bagheera will be sacrilege. More, though, the job was difficult because children’s fiction is, at its best, so strange. Its logic often arrives sideways.

Extending the cultural landscape … an illustration from Into The Jungle. Photograph: Macmillan

Britain, unlike Russia and France, is still inclined to treat children’s fiction as if it were something sweet and slim, akin to producing watercolour paintings of kittens; at its best, though (and its best is, of course, necessarily rare), it is a way of distilling the frenetic wildness of the untrammelled human imagination into something witty, sharp, generous, shrewd and honest. I was faced with the fact that The Jungle Book, and particularly its narrative voice, is so strange as to be inimitable.

Kipling’s narrator in the novel sounds constantly on the edge of irritation: wry, wary, caustic. This he shared with JM Barrie, whose voice is ruthless to the point of cruelty (the Darling children’s mother is dismissed in a single line: “Mrs Darling was now dead and forgotten”), and, to a lesser extent, CS Lewis. Kipling’s language was self-consciously archaic even for 1894, as if to align the stories with fables and myths.

It felt foolhardy to try to mimic Kipling’s voice, where mimicry could so swiftly descend into pastiche; so I attempted to keep the voices of the animals recognisably their own and the surrounding narration as simple as possible. It was a valuable lesson to be reminded of; looking at children’s fiction of the past, it is the characters that endure.

Fictional characters, though, do not grow and evolve as children grow and times evolve, and to return to old stories is frequently to face the conflict of wanting to cherish stories that contain elements intensely unpleasant to a contemporary eye. Tintin, for instance, is at once loyal, brave, witty and, in places, unambiguously racist. Georges Remi, alias Hergé, the inventor of Tintin, wrote for a Belgian wartime newspaper, Le Soir, which collaborated with the Nazi occupiers; in one story from that period, The Shooting Star, Tintin battles against a hook-nosed Jewish financier called Blumenstein. Roald Dahl, in an interview with the New Statesman in 1983, stated that “even a stinker like Hitler didn’t just pick [Jewish people] for no reason”. To read the children’s fiction of the 19th and 20th centuries is to ask if it’s possible to love aspects of a character or an author and reject those parts we cannot love, without ignoring or forgetting their existence.

One potential way through is proposed by Philip Ardagh, who has made amendments to Hugh Lofting’s The Story of Doctor Dolittle for the Macmillan Collector’s Library edition. The book is yet to be published; some will believe it is editing away the very history to which we should be vigilant; others will think it restores the book as a text that children can revel in. Children’s fiction, as the place in which hearts and minds are shaped, can be highly charged; the secret garden does not grow in primary colours.

Rex Harrison in Doctor Dolittle (1967). Photograph: www.ronaldgrantarchive.com

Although we appear to live in a time when there is a passion for the reinvention of classics and a desire to keep the old stories alive, there is nothing new in the desire to take on another’s ideas and twist them. Milton’s Paradise Lost took flamboyant liberties with the voice of Satan, and could be posited as the first piece of long-form fanfiction. In 1681 Nahum Tate rewrote Shakespeare’s King Lear, deleting the Fool entirely and creating an ending in which Cordelia and Edgar are married off, to mass jubilation.

In the mid 17th century the poet Katherine Philips, a prominent woman of letters in her day, took extracts from several of John Donne’s love poems and made them her own in “To the Excellent Mrs Anne Owen”. Philips passionately addresses a female companion, harnessing Donne’s eroticism while maintaining, by using borrowed imagery, the deniability necessary for propriety: her use of Donne allowed her freedom. The same impulse to borrow, build, add and reshape runs richly through other mediums: Jacques Loussier’s jazz renditions of Bach pull together Bach’s symmetries and jazz’s spontaneous irregularities to make something at once new and recognisable. Matthew Bourne’s dance production The Car Man takes the music from Bizet’s Carmen and turns it into a tale of desire in mid-20th century America.

Rewriting can also, of course, be a form of righting textual wrongs. Several children, working on the Nahum Tate model, have sent me alternative endings to my book Rooftoppers, when they found the original insufficiently definite. And, many years ago, having long found the ending of Little Women unconscionable, I wrote a new one.

Ultimately, though, I wanted to write my way into The Jungle Book because there has always, since I was young, been more I wanted to know, and nobody to tell me – and I believed that other children may have felt the same. I wanted to know more about the side-lined female characters; what made Mother Wolf so ferocious? I wanted to know how the many languages of the jungle worked; I wanted to know what was at the heart of Bagheera’s enigmatic silences. Writing within another writer’s world feels, at its root, like a form of exploration. At its best, it is a moment of reaching out, drawing a work from one time into another, one medium into another and attempting, in that moment, to build something at once old and new, to connect that world with this.

• Into the Jungle, illustrated by Kristjana S Williams, will be published on 20 September and The Complete Jungle Book on 4 October by Macmillan. To order a copy of Into the Jungle for £14.44 (RRP £16.99 ) go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.

• This article was amended on 10 September 2018, to clarify the author’s view.