I'm in a small room, sitting at a round table. A woman brings me a notebook. I open it and read the words scribbled 50 or 60 years ago. There is an idea on each page. Some are just a single sentence. Others are a paragraph, a line of dialogue, the description of an incident. What if someone committed a murder with a frozen leg of lamb? What if a poacher caught pheasants with drugged raisins? Reading this notebook is like peering into another man's imagination – and it happens to be the imagination of a writer whose books I have read and loved for almost my entire life.

Earlier this year, I was the writer in residence at the Roald Dahl Museum in Great Missenden, Buckinghamshire. It's a wonderful small museum devoted to Dahl's life and works. In the corner of one room, the BFG looms over visitors. In a corner of another stands Johnny Depp's costume from the movie of Charlie and the Chocolate Factory.

Trick or treat … Johnny Depp as Willy Wonka in Tim Burton's Charlie and the Chocolate Factory Photograph: cWarner Br/Everett / Rex Features

Best of all is Dahl's shed, transported piece by piece from his garden. You can see the armchair where he wrote his books and a table lined with curious mementoes - his hip bone, for instance, saved from the incinerator after an operation, and a big silver ball which he crafted from Kitkat wrappers.

During my residency, I did some events at the museum, visited local schools to help students with their creative writing, and worked on my own book. I also spent several afternoons in the museum's archives, looking at Dahl's notebooks, letters, photographs, clippings and manuscripts.

The archive itself is housed in a long, thin, windowless room, temperature-controlled to preserve its fragile contents. The only door is guarded by the archivist, Rachel White, who sits in her office, collecting and collating information about Roald Dahl. If you want to read anything in the archives, she will fetch it for you and bring it to a table in yet another room. Here I sat with my own notebook and a pencil; no pens are allowed in here, nor cups of tea or glasses or water, or anything else that might damage the manuscripts.

I began with the notebooks, then looked through a big box of photos and newspaper cuttings that Dahl kept for inspiration. It's like a dustbin of discarded ideas, overflowing with characters who never got the chance to tell their own stories.

Next I read the opening chapters of Dahl's first novel for adults, Some Time Never: A Fable for Supermen. It was published in 1948 to poor reviews, has never been reprinted, and is now exceptionally rare.

Dangerous imagination … a scene from the film of Chitty Chitty Bang Bang. Photograph: Ronald Grant Archive

Another box contains the drafts of Chitty Chitty Bang Bang, the screenplay that Dahl wrote, based on his friend Ian Fleming's novel. A pile of grey folders are stuffed with different versions, alongside notes from meetings with impatient movie tycoons. They're all interesting in their own ways, but one thing disappointed me: in all these drafts, there's no trace of the most memorable and terrifying character in the finished film. Rachel tells me that no one is quite sure who actually came up with the Child Catcher. He doesn't appear in Ian Fleming's novel (not much of the movie does), but he's nowhere to be found in Dahl's screenplay either. Perhaps those tycoons invented him themselves and slotted him into the film at the last moment.

Finally I came to my favourite of Dahl's books. It began as a short story, printed in the New Yorker in 1959, about two men stealing pheasants from a local landowner. Dahl was a great recycler of his own work – his characters often make a brief appearance in one book, then pop up again in another – and he couldn't leave this particular idea alone, expanding it and rewriting it for children rather than adults. Danny The Champion of the World was finally published in 1975, almost 20 years after the original story.

Rachel brings me another large grey box full of different drafts, each in its own folder. First there are Dahl's earliest handwritten versions, scrawled on yellow lined paper. A secretary would type them up. Dahl scribbled his notes and changes in the margins, and sent them back to be typed again. This continued until Dahl was content with what he'd written.

Reading through the different drafts is like watching someone grow up in front of your eyes. The ideas change. The characters develop. The writing tightens and strengthens. Here, for instance, are the last lines of Danny The Champion of the World as Dahl wrote them in his first draft:

It does not matter one bit whether you live in a caravan or a castle. What does matter is that you have a father or a mother or both who make exciting things happen around you, and who do exciting things with you, and who lead you into splendid adventures, for in that way you develop many of the good qualities you will need later on in life, such as courage, loyalty, steadfastness and perhaps above all a sense of fun.

It's not bad, is it? But it's not exactly brilliant. The sentiment is nice, and it's expressed carefully and succinctly, but there's nothing magical about these words, nothing that will make you put the book aside and stare into space while you think about what you're just read. There's even something surprisingly teacherly about these words; you might expect a vicar to extol the virtues of "courage, loyalty, steadfastness", but not mischievous Mister Dahl.

He obviously wasn't happy with it either, and carried on honing and polishing his words over many drafts. By the final version, although he hasn't altered his actual ideas very much, their expression is incomparably different, and that changes everything. Here are the last few words that you'll find at the end of Danny The Champion of the World:

A MESSAGE

to Children Who Have Read This Book

When you grow up

and have children of your own

do please remember something important

a stodgy parent is no fun at all

What a child wants

and deserves

is a parent who is

SPARKY

Reading a manuscript is a moving and strangely melancholy experience. As you turn the crinkly pages, you can't help imagining the writer doing the same thing himself, adding a line here, trimming a few words there, and chuckling at his own jokes. But it's also fascinating and inspiring to see how a great writer comes up with an idea and refuses to let go of it, never allowing himself to be satisfied, working away for years until he moulds it into the perfect shape.

Extracts from Danny the Champion of the World © Roald Dahl Nominee Limited 1975, 2013