The vista is straight out of a Merchant Ivory film rather than a Stanley Kubrick one. Past the olde English village with perfectly preserved church, through the first set of electronically operated gates, along the long, sweeping drive to the second set, and finally the third, and there, among the epic landscape of spreading trees and sweeping meadows, stands chez Kubrick. In its sprawling grandeur, it looks so quintessentially English that you wonder once again what brought the Brooklyn-born director here to the almost absolute seclusion of deepest rural Hertfordshire. It really is the most unlikely, but, at the same time, almost perfect setting for all those stories of Stanley Kubrick the recluse, the eccentric, the paranoid obsessive.

My youthful taxi driver is ever more gob-smacked with every electronic gate he has to jump out of the car to activate. 'Who lives here?' he asks, 'Tony Blair?' I tell him it is where Stanley Kubrick lived. He looks bemused and shakes his head. 'Never heard of him,' he says, and somewhere, you think, Kubrick is smiling. This is exactly how he preferred it, an anonymity that allowed him to get on with whatever it was he was doing in the increasingly long gaps between films that characterised the latter part of his life. A secluded life that was meant to keep the world at bay, but, as he found out to his cost, only made his reputation burgeon into the stuff of modern myth.

'He was sad about all the stuff those people wrote about him,' his widow, Christiane, a striking figure in artfully piled-up hair, heavy make-up and designer peasant smock, tells me, looking offended, though it is she who has brought up the subject of her husband's reputation for eccentricity so soon in our conversation, if only to dispel it. 'Those people' are the British tabloid press, of which Christiane is both scathing and dismissive. 'All those awful stories,' she sighs, looking pained, 'that he sprayed the grounds with insecticide because he was so afraid of intruders. This was a man who loved his animals, so how could he do such a thing? That he shot people and tried to pay them off because they are bleeding. Please? How could he not be hurt by this kind of lie? That he hated women, was phobic, obsessive, weird. I flinch,' she says, flinching, 'when I see all these things repeated about Stanley the so-called crazy man.'

Christiane is good company, effortlessly charming but fiercely protective of her late husband, her voice a mix of received English and aristocratic German, and only ever raised when she leaps to the defence of his reputation.

We are sitting in the library which used to be a screening room. The books on the shelves are not film books, but history books, most of which seem to be about Napoleon, the subject of the film that Kubrick most wanted to make, but, despite his clout and his unparalleled powers of persuasion, never got off the ground. 'This week,' ran a news item in the French paper, L'Express, on 23 September, 1968, 'several hundred books on Napoleon were shipped from Paris to the London office of Stanley Kubrick.' Now, 37 years later, they stand as testament to an enduring Kubrick obsession, for once left unfulfilled.

'He had plans to make it into an 18-hour television special,' says Christiane, 'that's why he saved all the books and stuff. Some articles have given the impression that Stanley was the kind of person who thought, "Oh here's this wonderful little turd of mine, I'm going to preserve it and keep it in a box."' 'No.' she says, sharply, while I marvel at her gift for metaphor. 'Nonsense. He was meticulous in his research and preparation, and very methodical in his working methods, but he was not a neurotic collector.'

Just how meticulous Kubrick could be is evinced by his extraordinary archive, an extensive collection of notebooks, photographs and research material of every conceivable kind, alongside invoices, contracts, budgets, and records even of film stock. After Kubrick's death, the family employed an archivist who sifted through several hundred boxes of material. It took him eight months, working 12 to 15 hours a day, but it now sits, sorted and filed, in several rooms in an adjacent building. Several American universities are interested in acquiring the Kubrick archive, a fraction of which is part of a touring exhibition of the director's work, currently on show in Germany, and scheduled to travel to Melbourne, Moscow, and Tokyo, but oddly not London. 'No one has offered yet,' says Christiane. 'I feel that they somehow don't really like Stanley here.'

Those that do, though, which is anyone with a passion for modern cinema, are in for a treat with the publication of The Stanley Kubrick Archives, an extravagant coffee-table book that mixes photographs, essays, facsimiles of the master's notebooks and story boards, and several previously hard to find interviews. Published by Taschen, who excel at this sort of sumptuous presentation, the book was overseen by Christiane, and her brother, Jan, who was Kubrick's long-time assistant and creative confidante. Given that there are several family photographs that Kubrick would have considered intimate, I ask Christiane, as tactfully as possible, if she is certain he would have approved of this undertaking, which, by his standards, is quite revealing. 'Oh, I think so. I mean, we were so careful, it's almost as if he were looking over our shoulders. This is a book for fans, and for film-makers, I think. It's great to have access to his preparatory work, to the technical stuff. It's really a glimpse of cinema history you are looking at.'

Christiane's life has been almost as colourful as her husband's. Born Christiane Susanne Harlan in 1932, in Braunschweig, Germany, she grew up amid an extended family of musicians and actors. Both her parents were opera singers, but she gravitated towards painting at an early age, filling sketchbooks with images of the upheaval of wartime. Her war drawings were later exhibited in Russia, but never returned.

She first met Stanley Kubrick in 1956 on the set of his third feature film, the war epic, Paths of Glory. He had seen her act in a German television adaptation of Chekhov's Three Sisters, and cast her as the only female in the film. She appears in the final scene as the innocent young German girl who sings an old German folk song to the assembled soldiers. 'He was shooting chronologically,' she remembers, 'so from the time he hired me at the start of the film to the time I actually had to do my scene, we had got to know each other, and were already attached to be married.'

Kubrick had already been married twice before, and she once, but their whirlwind romance endured, not least, one suspects, because she pursued her own creative destiny, and is now a highly respected painter. Her big still-life paintings of flowers and plants fill the next-door studio, which is bright and airy, a complete contrast to the library we are sitting in. Leafing though the book, she happens on a photograph of the Grady girls, the ghostly twins who appear to such disturbing effect in The Shining, Kubrick's masterful horror film. I ask her if, as some critics have suggested, the girls' image was based on the famous Diane Arbus portrait of two similar looking girls. 'No,' she replies, 'he cast them as he saw them from Stephen King's book. He liked how utterly serious they looked. He encouraged them to dream up horrible things during the filming. He even let them paint blood on themselves, and they really went to town. It became a game. He never ever let them become scared. That was Stanley,' she sighs.

It is six years since Stanley Kubrick's sudden death on 7 March, 1999, but his presence is palpable everywhere in this huge house. The obituaries from the time paint two distinct but overlapping pictures of the man: the cinematic genius and the oddball recluse. The words 'obsessive', 'paranoid', and 'disenchanted' recur, alongside the words 'auteur', 'perfectionist' and, of course, 'genius'. Christiane wholeheartedly endorses the last three, and blithely dismisses the first three, but one suspects the truth lies somewhere in between.

Kubrick was certainly meticulous going on obsessive in his preparation, and relentless in his pursuit of the perfect scene - just watch the browbeating of Shelley Duval in The Shining, recorded for posterity by his youngest daughter, Vivian, whose documentary on the making of the film remains the most revealing glimpse of the master at work. His dramatically decreasing work rate over the years - seven years between The Shining (1980) and Full Metal Jacket (1987), another 12 until Eyes Wide Shut (1999) - certainly suggests a deep disenchantment with the industry, even as it, in the form of Warner Brothers, allowed him an unprecedented amount of creative control and freedom. And, yes, he was a recluse, not least from the glare and unrelenting pressure of Los Angeles, of which he had a taste when Kirk Douglas hired him to direct Spartacus (1960), the one and only all-Hollywood movie he ever made. The film was a huge success, but Kubrick fled to Britain soon afterwards, chastened by the experience, and determined never to repeat it. Eventually Hollywood came to him in the shape of stars such as Ryan O'Neal (Barry Lyndon), Jack Nicholson (The Shining), and, of course, Cruise and Kidman (Eyes Wide Shut

The more Kubrick fled fame, though, the more it stalked him. His fear of flying - precipitated when a friend, whom he had worked with as a photographer on Look magazine in the Fifties, was killed in a plane crash - seems to have given rise to stories of all manner of obsession, all denied by Christiane. His obsessions were confined to film-making, she says, as demonstrated by his meticulous recreation of a New York street at Pinewood, the Vietnam jungle in a disused gasworks in King's Cross. All so that Stanley could sleep in his own bed after a long day's shooting.

During the making of Eyes Wide Shut, a man called Alan Conway went around impersonating Kubrick at various social events in London, before being arrested. 'It was an absolute nightmare,' says Christiane, 'This strange doppelganger who was pretending to be Stanley. Can you imagine the horror?' I nod, trying to. 'It took ages to find him,' she says, shaking her head, 'and then he pleaded insanity, which succeeded. Then they go and make a TV documentary parading him around. He seemed like such a disgusting person, and now there is a film about him and so on.'

She seems exhausted even talking about this episode. I don't mention that, while researching this piece, I came across various web updates on Colour Me Kubrick, the film in question, and that John Malkovich, who played a man who had his identity invaded in Being John Malkovich, is to play the man who impersonated Stanley Kubrick. It sounds like another Charlie Kaufman/Spike Jonze wheeze, but it isn't.

Given that the archive must contain a host of memories, was it hard for her to go through it in order to select what went into the book? 'Well, when you are unearthing stuff, you are also unearthing feelings. It brings out the whole scale of emotions, sometimes laughter, sometimes deep, deep sadness. When someone dies, and you come to the moment when you must go through the left-behind things, you go through some kind of hell, and some wonderful moments. It can be inspiring one day, and utterly soul-destroying the next. What is left behind says so much about what you have lost.'

I think of the huge archive, the sheer volume of associations. 'He didn't hoard,' says Christiane, as if reading my thoughts, 'he just didn't throw anything away. There's a difference.' I screw up my face in concentration, trying, for the life of me, to discern what that difference is. 'The thing you have to understand,' she says, patiently, 'is that, every time he finished a film, the film people would ring up and say, "We have all this stuff." So he would say, "Send it over, and I'll sort it out." But he never did. And the reason he never did,' she says, laughing as if realising the absurdity of it all for the first time, 'is that we have so much room. He didn't have to sort it out, he could just store it somewhere safe and forget about it.'

This makes perfect sense to me. I have a flat that would fit comfortably into this library, but I do the same thing. Suddenly, it doesn't seem so eccentric or obsessive, just endearingly human. I'm almost convinced that Stanley was just a regular guy, rather than one of the great obsessive artists of our time. Almost.

Towards the end of the interview, Christiane returns to a subject that has obviously been on her mind, her husband's insistence on privacy, on the primacy of his work over the personality that made it. 'I ask myself all the time, as any widow would, what is right to talk about, and what is not,' she says, 'and perhaps after he died, we said too much. If you love someone, of course, you have many cute stories to tell, and we told some of them. That makes me cringe now. The book seems a more respectful kind of remembering. It concentrates on the work. I'm sure,' she says, smiling, 'Stanley would have approved.'

The Stanley Kubrick Archives, edited by Alison Castle, Taschen £100, pp544. We have two copies of the book to give away. Details: http://blogs.theguardian.com/observer