In 1996 I received what was - and probably remains - the most exciting telephone call I have ever had. It was from a man calling himself Tony. "I'm phoning on behalf of Stanley Kubrick," he said.

"I'm sorry?" I said.

"Stanley would like you to send him a radio documentary you made called Hotel Auschwitz," said this man. This was a programme for Radio 4 about the marketing of the concentration camp.

"Stanley Kubrick?" I said.

"Let me give you the address," said the man. He sounded posh. It seemed that he didn't want to say any more about this than he had to. I sent the tape to a PO box in St Albans and waited. What might happen next? Whatever it was, it was going to be amazing. My mind started going crazy. Perhaps Kubrick would ask me to collaborate on something. (Oddly, in this daydream, I reluctantly turned him down because I didn't think I'd make a good screenwriter.)

At the time I received that telephone call, nine years had passed since Kubrick's last film, Full Metal Jacket. All anyone outside his circle knew about him was that he was living in a vast country house somewhere near St Albans - or a "secret lair", according to a Sunday Times article of that year - behaving presumably like some kind of mad hermit genius. Nobody even knew what he looked like. It had been 16 years since a photograph of him had been published.

He'd gone from making a film a year in the 1950s (including the brilliant, horrific Paths Of Glory), to a film every couple of years in the 1960s (Lolita, Dr Strangelove and 2001: A Space Odyssey all came out within a six-year period), to two films a decade in the 1970s and 1980s (there had been a seven-year gap between The Shining and Full Metal Jacket), and now, in the 1990s, absolutely nothing. What the hell was he doing in there? According to rumours, he was passing his time being terrified of germs and refusing to let his chauffeur drive over 30mph. But now I knew what he was doing. He was listening to my BBC Radio 4 documentary, Hotel Auschwitz.

"The good news," wrote Nicholas Wapshott in the Times in 1997, bemoaning the ever-lengthening gaps between his films, "is that Kubrick is a hoarder ... There is an extensive archive of material at his home in Childwick Bury. When that is eventually opened, we may get close to understanding the tangled brain which brought to life HAL, the [Clockwork Orange] Droogs and Jack Torrance."

The thing is, once I sent the tape to the PO box, nothing happened next. I never heard anything again. Not a word. My cassette disappeared into the mysterious world of Stanley Kubrick. And then, three years later, Kubrick was dead.

Two years after that, in 2001, I got another phone call out of the blue from the man called Tony. "Do you want to get some lunch?" he asked. "Why don't you come up to Childwick?"

The journey to the Kubrick house starts normally. You drive through rural Hertfordshire, passing ordinary-sized postwar houses and opticians and vets. Then you turn right at an electric gate with a "Do Not Trespass" sign. Drive through that, and through some woods, and past a long, white fence with the paint peeling off, and then another electric gate, and then another electric gate, and then another electric gate, and you're in the middle of an estate full of boxes.

There are boxes everywhere - shelves of boxes in the stable block, rooms full of boxes in the main house. In the fields, where racehorses once stood and grazed, are half a dozen portable cabins, each packed with boxes. These are the boxes that contain the legendary Kubrick archive.

Was the Times right? Would the stuff inside the boxes offer an understanding of his "tangled brain"? I notice that many of the boxes are sealed. Some have, in fact, remained unopened for decades.

Tony turns out to be Tony Frewin. He started working as an office boy for Kubrick in 1965, when he was 17. One day, apropos of nothing, Kubrick said to him, "You have that office outside my office if I need you." That was 36 years ago and Tony is still here, two years after Kubrick died and was buried in the grounds behind the house. There may be no more Kubrick movies to make, but there are DVDs to remaster and reissue in special editions. There are box sets and retrospective books to oversee. There is paperwork.

Tony gives me a guided tour of the house. We walk past boxes and more boxes and filing cabinets and past a grand staircase. Childwick was once home to a family of horse-breeders called the Joels. Back then there were, presumably, busts or floral displays on either side at the bottom of this staircase. Here, instead, is a photocopier on one side and another photocopier on the other.

"Is this ... ?" I ask.

"Yes," says Tony. "This is how Stanley left it."

Stanley Kubrick's house looks as if the Inland Revenue took it over long ago.

Tony takes me into a large room painted blue and filled with books. "This used to be the cinema," he says.

"Is it the library now?" I ask.

"Look closer at the books," says Tony.

I do. "Bloody hell," I say. "Every book in this room is about Napoleon!"

"Look in the drawers," says Tony.

I do.

"It's all about Napoleon, too!" I say. "Everything in here is about Napoleon!"

I feel a little like Shelley Duvall in The Shining, chancing upon her husband's novel and finding it is comprised entirely of the line "All Work And No Play Makes Jack A Dull Boy" typed over and over again. John Baxter wrote, in his unauthorised biography of Kubrick, "Most people attributed the purchase of Childwick to Kubrick's passion for privacy, and drew parallels with Jack Torrance in The Shining."

This room full of Napoleon stuff seems to bear out that comparison. "Somewhere else in this house," Tony says, "is a cabinet full of 25,000 library cards, three inches by five inches. If you want to know what Napoleon, or Josephine, or anyone within Napoleon's inner circle was doing on the afternoon of July 23 17-whatever, you go to that card and it'll tell you."

"Who made up the cards?" I ask.

"Stanley," says Tony. "With some assistants."

"How long did it take?" I ask.

"Years," says Tony. "The late 1960s."

Kubrick never made his film about Napoleon. During the years it took him to compile this research, a Rod Steiger movie called Waterloo was written, produced and released. It was a box-office failure, so MGM abandoned Napoleon and Kubrick made A Clockwork Orange instead.

"Did you do this kind of massive research for all the movies?" I ask Tony.

"More or less," he says.

"OK," I say. "I understand how you might do this for Napoleon, but what about, say, The Shining?"

"Somewhere here," says Tony, "is just about every ghost book ever written, and there'll be a box containing photographs of the exteriors of maybe every mountain hotel in the world."

There is a silence.

"Tony," I say, "can I look through the boxes?"

I've been coming to the Kubrick house a couple of times a month ever since.

I start, chronologically, in a portable cabin behind the stable block, with a box marked Lolita. I open it, noting the ease with which the lid comes off. "These are excellent, well-designed boxes," I think to myself. I flick through the paperwork inside, pausing randomly at a letter that reads as if it has come straight from a Jane Austen novel:

Dear Mr Kubrick,

Just a line to express to you and to Mrs Kubrick my husband's and my own deep appreciation of your kindness in arranging for Dimitri's introduction to your uncle, Mr Günther Rennert.

Sincerely,

Mrs Vladimir Nabokov

I later learn that Dimitri was a budding opera singer and Rennert was a famous opera director, in charge of the Munich Opera House. This letter was written in 1962, back in the days when Kubrick was still producing a film every year or so. This box is full of fascinating correspondence between Kubrick and the Nabokovs but - unlike the fabulously otherworldly Napoleon room, which was accrued six years later - it is the kind of stuff you would probably find in any director's archive.

The unusual stuff - the stuff that elucidates the ever-lengthening gaps between productions - can be found in the boxes that were compiled from 1968 onwards. In a box next to the Lolita box in the cabin, I find an unusually terse letter, written by Kubrick to someone called Pat, on January 10 1968: "Dear Pat, Although you are apparently too busy to personally return my phone calls, perhaps you will find time in the near future to reply to this letter?"

(Later, when I show Tony this letter, he says he's surprised by the brusqueness. Kubrick must have been at the end of his tether, he says, because on a number of occasions he said to Tony, "Before you send an angry letter, imagine how it would look if it got into the hands of Time Out.") The reason for Kubrick's annoyance in this particular letter was because he'd heard that the Beatles were going to use a landscape shot from Dr Strangelove in one of their movies: "The Beatle film will be very widely seen," Kubrick writes, "and it will make it appear that the material in Dr Strangelove is stock footage. I feel this harms the film."

There is a similar batch of telexes from 1975: "It would appear," Kubrick writes in one, "that Space 1999 may very well become a long-running and important television series. There seems nothing left now but to seek the highest possible damages ... The deliberate choice of a date only two years away from 2001 is not accidental and harms us." This telex was written seven years after the release of 2001.

But you can see why Kubrick sometimes felt compelled to wage war to protect the honour of his work. A 1975 telex, from a picture publicity man at Warner Bros called Mark Kauffman, regards publicity stills for Kubrick's sombre reworking of Thackeray's Barry Lyndon. It reads: "Received additional material. Is there any material with humour or zaniness that you could send?"

Kubrick replies, clearly through gritted teeth: "The style of the picture is reflected by the stills you have already received. The film is based on William Makepeace Thackeray's novel which, though it has irony and wit, could not be well described as zany."

I take a break from the boxes to wander over to Tony's office. As I walk in, I notice something pinned to his letterbox. "POSTMAN," it reads. "Please put all mail in the white box under the colonnade across the courtyard to your right."

It is not a remarkable note except for one thing. The typeface Tony used to print it is exactly the same typeface Kubrick used for the posters and title sequences of Eyes Wide Shut and 2001. "It's Futura Extra Bold," explains Tony. "It was Stanley's favourite typeface. It's sans serif. He liked Helvetica and Univers, too. Clean and elegant."

"Is this the kind of thing you and Kubrick used to discuss?" I ask.

"God, yes," says Tony. "Sometimes late into the night. I was always trying to persuade him to turn away from them. But he was wedded to his sans serifs."

Tony goes to his bookshelf and brings down a number of volumes full of examples of typefaces, the kind of volumes he and Kubrick used to study, and he shows them to me. "I did once get him to admit the beauty of Bembo," he adds, "a serif."

"So is that note to the postman a sort of private tribute from you to Kubrick?" I ask.

"Yeah," says Tony. He smiles to himself. "Yeah, yeah."

For a moment I also smile at the unlikely image of the two men discussing the relative merits of typefaces late into the night, but then I remember the first time I saw the trailer for Eyes Wide Shut, the way the words "CRUISE, KIDMAN, KUBRICK" flashed dramatically on to the screen in large red, yellow and white colours, to the song Baby Did A Bad Bad Thing. Had the words not been in Futura Extra Bold, I realise now, they wouldn't have sent such a chill up the spine. Kubrick and Tony obviously became, at some point during their relationship, tireless amateur sleuths, wanting to amass and consume and understand all information. Tony obviously misses Kubrick terribly.

But this attention to detail becomes so amazingly evident and seemingly all-consuming in the later boxes, I begin to wonder whether it was worth it. In one portable cabin, for example, there are hundreds and hundreds of boxes related to Eyes Wide Shut, marked EWS - Portman Square, EWS - Kensington & Chelsea, etc, etc. I choose the one marked EWS - Islington because that's where I live. Inside are hundreds of photographs of doorways. The doorway of my local video shop, Century Video, is here, as is the doorway of my dry cleaner's, Spots Suede Services on Upper Street. Then, as I continue to flick through the photographs, I find, to my astonishment, pictures of the doorways of the houses in my own street. Handwritten at the top of these photographs are the words, "Hooker doorway?"

"Huh," I think. So somebody within the Kubrick organisation (it was, in fact, his nephew) once walked up my street, on Kubrick's orders, hoping to find a suitable doorway for a hooker in Eyes Wide Shut. It is both an extremely interesting find and a bit of a kick in the teeth.

It is not, though, as incredible a coincidence as it may at first seem. Judging by the writing on the boxes, probably just about every doorway in London has been captured and placed inside this cabin. This solves one mystery for me - the one about why Kubrick, a native of the Bronx, chose the St Albans countryside, of all places, for his home. I realise now that it didn't matter. It could have been anywhere. It is as if the whole world is to be found somewhere within this estate.

But was it worth it? Was the hooker doorway eventually picked for Eyes Wide Shut the quintessential hooker doorway? Back at home, I watch Eyes Wide Shut again on DVD. The hooker doorway looks exactly like any doorway you would find in Lower Manhattan - maybe on Canal Street or in the East Village. It is a red door, up some brownstone steps, with the number 265 painted on the glass at the top. Tom Cruise is pulled through the door by the hooker. The scene is over in a few seconds. (It was eventually shot on a set at Pinewood.) I remember the Napoleon archive, the years it took Kubrick and some assistants to compile it, and I suggest to Jan Harlan, Kubrick's executive producer and brother-in-law, that had there not been all those years of attention to detail during the early planning of the movie, perhaps Napoleon would actually have been made.

"That's a completely theoretical and obsolete observation!" replies Jan, in a jolly way. "That's like saying had Vermeer painted in a different manner, he'd have done 100 more paintings."

"OK," I say.

Jan is right, of course. So why am I so keen to discover in the boxes some secret personality flaw to Kubrick, whose films I love so much? He was the greatest director of his generation. Jack Nicholson's "Here's Johnny!" Lolita's heart-shaped sunglasses. The Dr Strangelove cowboy riding the nuclear bomb like it's a bucking bronco. And on and on. So many images have implanted themselves into the public consciousness, surely because of the director's ever-burgeoning attention to detail.

"Why don't you just accept," says Jan, "that this was how he worked?"

"But if he hadn't allowed his tireless work ethic to take him to unproductive places, he'd have made more films," I say. "For instance, the Space 1999 lawsuit seems, with the benefit of hindsight, a little trivial."

"Of course I wish he had made more films," says Jan.

Jan and I are having this conversation inside the stable block, surrounded by hundreds of boxes. For the past few days I have been reading the contents of those marked "Fan Letters" and "Résumés". They are filled with pleas from hundreds of strangers, written over the decades. They say much the same thing: "I know I have the talent to be a big star. I know it's going to happen to me one day. I just need a break. Will you give me that break?"

All these letters are - every single one of them - written by people of whom I have never heard. Many of these young actors will be middle-aged by now. I want to go back in time and say to them, "You're not going to make it! It's best you know now rather than face years of having your dreams slowly erode." They are heartbreaking boxes.

"Stanley never wrote back to the fans," says Jan. "He never, never responded. It would have been too much. It would have driven him crazy. He didn't like to get engaged with strangers."

(In fact, I soon discover, Kubrick did write back to fans, on random, rare occasions. I find two replies in total. Maybe he only ever wrote back twice. One reads, "Your letter of 4th May was overwhelming. What can I say in reply? Sincerely, Stanley Kubrick." The other reads, "Dear Mr William, Thank you for writing. No comment about A Clockwork Orange. You will have to decide for yourself. Sincerely, Stanley Kubrick.")

"One time, in 1998," Jan says, "I was in the kitchen with Stanley and I mentioned that I'd just been to the optician's in St Albans to get a new pair of glasses. Stanley looked shocked. He said, 'Where exactly did you go?' I told him and he said, 'Oh, thank God! I was just in the other optician's in town getting some glasses and I used your name!'" Jan laughs. "He used my name in the optician's, everywhere."

"But even if he didn't reply to the fan letters," I say, "they've all been so scrupulously read and filed."

The fan letters are perfectly preserved. They are not in the least bit dusty or crushed. The system used to file them is, in fact, extraordinary. Each fan box contains perhaps 50 orange folders. Each folder has the name of a town or city typed on the front - Agincourt, Ontario; Alhambra, California; Cincinnati, Ohio; Daly City, California, and so on - and they are in alphabetical order inside the boxes. And inside each folder are all the fan letters that came from that particular place in any one year. Kubrick has handwritten "F-P" on the positive ones and "F-N" on the negative ones. The crazy ones have been marked "F-C".

"Look at this," I say to Jan.

I hand him a letter written by a fan and addressed to Arthur C Clarke. He forwarded it on to Kubrick and wrote on the top, "Stanley. See P3!! Arthur."

Jan turns to page 3, where Clarke had marked, with exclamation marks, the following paragraph:

"What is the meaning behind the epidemic? Does the pink furniture reveal anything about the 3rd monolith and it's emitting a pink colour when it first approaches the ship? Does this have anything to do with a shy expression? Does the alcohol offered by the Russians have anything to do with French kissing and saliva?"

"Why do you think Arthur C Clarke marked that particular paragraph for Kubrick to read?" I ask Jan.

"Because it is so bizarre and absurd," he says.

"I thought so," I say. "I just wanted to make sure."

In the back of my mind, I wondered whether this paragraph was marked because the writer of the fan letter - Mr Sam Laks of Alhambra, California - had actually worked out the secret of the monolith in 2001. I find myself empathising with Sam Laks. I am also looking for answers to the mysteries. So many conspiracy theories and wild rumours surrounded Kubrick - the one about him being responsible for faking the moon landings (untrue), the one about his terror of germs (this one can't be true, either - there's a lot of dust around here), the one about him refusing to fly and drive over 30mph. (The flying one is true - Tony says he wasn't scared of planes, he was scared of air traffic controllers - but the one about the 30mph is "bullshit", says Tony. "He had a Porsche.")

This is why my happiest times looking through the boxes are when things turn weird. For instance, at the end of one shelf inside the stable block is a box marked "Sniper head - scary". Inside, wrapped in newspaper, is an extremely lifelike and completely disgusting disembodied head of a young Vietnamese girl, the veins in her neck protruding horribly, her eyes staring out, her lips slightly open, her tongue just visible. I feel physically sick looking at it. As I hold it up by its blood-matted hair, Christiane, Kubrick's widow, walks past the window.

"I found a head!" I say.

"It's probably Ryan O'Neal's head," she replies.

Christiane has no idea who I am, nor what I'm doing in her house, but she accepts the moment with admirable calm.

"No," I say. "It's the head of the sniper from Full Metal Jacket."

"But she wasn't beheaded," calls back Christiane. "She was shot."

"I know!" I say.

Christiane shrugs and walks on. The sniper head would probably please Mr Sam Laks, on a superficial level, because it is so grotesque. But in general the most exotic things to be found here are generated from the outside, from the imaginations of fans like him.

"I was just talking to Tony about typefaces," I say to Jan.

"Ah yes," says Jan. "Stanley loved typefaces." Jan pauses. "I tell you what else he loved."

"What?" I ask.

"Stationery," says Jan.

I glance over at the boxes full of letters from people who felt about Kubrick the way Kubrick felt about stationery, and then back to Jan. "His great hobby was stationery," he says. "One time a package arrived with 100 bottles of brown ink. I said to Stanley, 'What are you going to do with all that ink?' He said, 'I was told they were going to discontinue the line, so I bought all the remaining bottles in existence.' Stanley had a tremendous amount of ink." Jan pauses. "He loved stationery, pads, everything like that."

Tony wanders into the stable block.

"How's it going?" he asks.

"Still looking for Rosebud," I say.

"The closest I ever got to Rosebud," says Tony, "was finding a daisy gun that he had when he was a child."

As I look through the boxes over the months, I never find my Hotel Auschwitz tape. Nor do I get around to opening the two boxes that read Shadow On The Sun. But, one evening just before last Christmas, I decide to take a look. The boxes contain two volumes of what appears to be a cheesy sci-fi radio drama script. The story begins with a sick dog: "Can you run me over to Oxford with my dog?" says the dog's owner. "He's not very well. I'm a bit worried about him, John." This is typed.

Kubrick has handwritten below it: "THE DOG IS NOT WELL." It soon becomes clear - through speed-reading - that a virus has been carried to earth on a meteorite. This is why the dog is listless, and also why humans across the planet are no longer able to control their sexual appetites. It ends with a speech: "There's been so much killing - friend against friend, neighbour against neighbour, but we all know nobody on this earth is to blame, Mrs Brighton. We've all had the compulsions. We'll just have to forgive each other our trespasses. I'll do my part. I'll grant a general amnesty - wipe the slate clean. Then perhaps we can begin to live again, as ordinary decent human beings, and forget the horror of the past few months."

This, too, is typed. But all over the script I find notes handwritten by Kubrick. ("Establish Brighton's interest in extraterrestrial matters"; "Dog finds meteorite"; "John has got to have very powerful connections of the highest level"; "A Bill Murray line!") "Tony!" I say. "What the hell is this?"

I believe I have stumbled on a lost Kubrick radio play. Perhaps he did this in his spare time. But, if so, why?

"No, no," says Tony. "I know what this is."

Kubrick was always a keen listener to BBC Radio, Tony explains. When he first arrived in the UK, back in the early 1960s, he happened to hear this drama serial, Shadow On The Sun. Three decades later, in the early 1990s, after he had finished Full Metal Jacket, he was looking for a new project, so he asked Tony to track down the scripts. He spent a few years, on and off, thinking about Shadow On The Sun, reading and annotating the scripts, before he abandoned the idea and eventually - after working on and rejecting AI (which was filmed by Steven Spielberg after Kubrick's death) - made Eyes Wide Shut instead.

"But the original script seems so cheesy," I say.

"Ah," replies Tony, "but this is before Stanley worked his alchemy."

And I realise this is true. "Dog finds meteorite." It sounds so banal, but imagine how Kubrick might have directed it. Do the words, "Ape finds monolith" or, "Little boy turns the corner and sees twin girls" sound any less banal on the page?

All this time I have been looking in the boxes for some embodiment of the fantasies of the outsiders like Mr Sam Laks and me - but I never do find anything like that. I suppose that the closer you get to an enigma, the more explicable it becomes. Even the somewhat crazy-seeming stuff, like the filing of the fan letters by the town from which they came, begins to make sense after a while.

It turns out that Kubrick ordered this filing in case he ever wanted to have a local cinema checked out. If 2001, say, was being screened in Daly City, California, at a cinema unknown to Kubrick, he would get Tony or one of his secretaries to telephone a fan from that town to ask them to visit the cinema to ensure that, say, the screen wasn't ripped. Tony says that if I'm looking for something exotic or unexpected or extreme, if I'm looking for the solution to the mystery of Kubrick, I don't really need to look inside the boxes. I just need to watch the films.

"It's all there," he says. "Those films are Stanley."

Although the Kubricks have always closely guarded their privacy inside Childwick, I come to the end of my time at the house during something of a watershed moment. Christiane Kubrick and her daughter Katherine are soon to open the grounds and the stable block to the public for an art fair, displaying their work and the work of a number of local artists. The boxes are going to be moved somewhere else. Many, in fact, have now been shipped to Frankfurt. On March 31, the Deutsches Filmmuseum will launch a major Kubrick exhibition, including lenses, props, cameras and some of the stuff that I found in the boxes. This will tour across Europe and hopefully visit London, if the BFI can find a suitable exhibition space. And the German publisher Taschen is soon to bring out a book on Kubrick that will reproduce some of the Napoleon archive.

Towards the end of my time at the Kubrick house, Tony mentions something seemingly inconsequential, but as soon as he says it I realise that the Rosebud I was after - the quintessence of Kubrick - has been staring me in the face from the very first day. From the beginning, I had mentally noted how well constructed the boxes were, and now Tony tells me that this is because Kubrick designed them himself. He wasn't happy with the boxes that were on the market - their restrictive dimensions and the fact that it was sometimes difficult to get the tops off - so he set about designing a whole new type of box. He instructed a company of box manufacturers, G Ryder & Co, of Milton Keynes, to construct 400 of them to his specifications.

"When one batch arrived," says Tony, "we opened them up and found a note, written by someone at G Ryder & Co. The note said, 'Fussy customer. Make sure the tops slide off.'"

Tony laughs. I half expect him to say, "I suppose we were a bit fussy." But he doesn't. Instead, he says, "As opposed to non-fussy customers who don't care if they struggle all day to get the tops off."

The thing is, nobody outside the Kubrick house got to see the boxes