"I have kept a diary," he said, "since I witnessed my mother's adultery at the age of nine." As he spoke, an elderly woman was slowly trundling her luggage trolley so close to our knees we had to draw our legs out of her path. Richard E Grant was unfazed, as though his narrative focus was so fine he had not noticed her at all. He is a disconcerting fellow. This morning, at the crack of dawn, he came off the red-eye from Newfoundland and, if he was a normal human being, he'd be crashed out in his pit, sleeping off his jet lag, not spring-heeling around this rowdy hotel. I can never trust manic energy, let alone a man with eyes the colour of turquoise. It's not natural.

However, I like to think the elderly woman took the bit about seeing his mamma in flagrante to her room to mull over. I was stuck on the 40 years of diary addiction. Diarists make me very uneasy. There's something disempowering and control-freakish about them, as if they're going to snatch the pen from my hand and stab me with it. He is not, he said (not for the first or last time), a shrink. He can only tell me he began his habit of diarising at a time when he was beset by guilt and loss. He knew something terrible and he couldn't tell anyone about it, so he wrote down his understanding of the aforementioned debacle as a way of off-loading the pressure of what he knew. And went on doing it because it became his way of dealing with the world.

"It means you are simultaneously inside and outside your own life all the time," he said, "watching yourself experiencing what is happening to you, then having a written conversation with yourself about it. It's a kind of control mechanism; an exploration and a way of keeping a record."

It can be quite ruthless, apparently. He wouldn't necessarily recommend it, though he has been told he has got rather good at it. Richard E never comes out with a straightforward boast. It's always something some kind person has told him that has given him the courage or the confidence to do the things he has done. Otherwise, he gives the impression, he'd have sweltered his life away in a paralysing miasma of self-disgust, watching himself making a complete tit of himself, knowing he would shortly be recalling and describing every microscopic detail of his tit-ship.

So excruciating is his self-consciousness, he has only once watched himself on the screen and that was nearly 20 years ago when he sat through the entirety of Bruce Robinson's glorious Withnail And I in such an agony of disillusion that by the end he was practically welded to his seat and had drawn blood from his wife's comforting hand. Thereafter, he was lionised up hill and down dale, but deep down he has always known he buggered up, let everybody down, missed the boat, exposed himself as a total no-hoper who would never work again. He also knows, and will say with perfect equanimity, that Withnail was his first big break, without which he would never have worked with Altman, Coppola and Scorsese, never been the movie star who, in the mid-90s published a memoir of his years in Hollywood that, yes, does credit to his addiction to diary-writing despite its catchpenny title With Nails, which doesn't mean anything except a lack of confidence in the undoubted charm and cleverness of its content. He shrugs that one off. What could he do? The publisher had to know best, after all - they thought he was worth publishing.

After 60 films, things haven't got any better. "You finish a movie and you think, there, you've done it, really well, or best you can. But if you watch it, you see it was just bollocks. You have to look at the discrepancy between what you hoped and imagined and the reality of yourself and all your shortcomings. You only see your own failure. I'd rather," he said, "stick with the first idea - just have the experience of working - and leave it at that. You've got to protect the old bravado."

It is also sort of true that Withnail has been a double-edged sword throughout his subsequent career. The identification was perhaps too indelible. Withnail was iconic. And he was Withnail. He was once telephoned by a casting director's secretary who, passing him on to the big man, was heard to say, "It's Withnail." I can see Withnail in him now and I can feel it's not fair. The eternally out of work wannabe actor, swilling alcohol, chain smoking, a sociopathic ratbag who'd pimp his best friend for a square meal and a bottle of plonk. I wanted to know what happened to him. You know, after the end of the film, where he stood all alone in the pissing rain watching the tragic, limp wolf in its zoo enclosure and you couldn't help feeling sorry for him. And Richard E, who is allergic to alcohol, doesn't smoke and is a thoroughly good egg, was pleased to tell me that Withnail never acted, contracted throat cancer and died of alcoholic poisoning at the age of 48, the list of disasters tripping off his tongue with all the emotion of a man reciting a laundry list. I wish I hadn't asked. No redemption then? No redemption.

Richard E has now lived for exactly as many years as Withnail. He intends, he says, to live for ever. It has been 10 years since he first thought of writing and directing his own story, and nine since he cocked the snook at Hollywood with With Nails, an acerbic account of his days in LA, for which he knew he would surely be punished. He does not expect to be invited back. "What is there now?" he said. "Famous people running away from explosions. That's it. They call it production values. Audiences will queue round the block to see an unimaginably highly-paid film star running away from a fantastically expensive explosion. They think it's their money's worth. I despair that's what people have to do."

It does, however, explain why so many "thesps", as Richard E insists on calling his fellow actors, move away from the obscenely big-bucks industry and into television and small production companies - "to enact being human beings instead of cartoon characters leaping from imploding buildings".

He once sat down with the actor Matthew Modine (Birdy, Full Metal Jacket) and discussed the Hollywood star system. What you have to do, they concluded, to make the transition from film actor to film star, is to be in movies where you are seen to be morally justified in killing people, while retaining your sex appeal. Literally getting away with murder while being perceived as a highly desirable fellow. Failing that, you have to take the other option; play a psychopath, a deaf-mute, an amputee or a blind man. This being so, Richard E somehow lost any ambition he might have had to climb the Hollywood ladder to the topmost wrung.

"Hollywood," he said, "is on what they call a shit-tide, meaning a tide where stuff comes in and goes out very quickly. People come in, get a part in something, get in a magazine, then they go away and you never hear of them again. The sun shines, the level of paranoia is bottomless and everybody you meet has an agenda. And that's it. Showbusiness, 24 hours a day. If you're doing well, you're a target, nobody's interested in you except how you can be of use to them. And you can't engage with anyone, you can only engage with their agenda. It is all," he ended enigmatically, "very anti-sex."

In the early years of his visibility as a British actor, Richard E Grant seems to have magnetised an abiding scepticism from the press. It has been as though showbusiness hacks were somehow thrown by the combination of low-key openness and casual irony that they took to be some kind of pretentiousness. The E was particularly bothersome to them. "What's the E for?" they'd go. And he'd tell them there was already a Richard Grant on the scene when he was applying for his Equity card, and his agent didn't think using his real surname, Esterhuysen, was going to be a thrill a minute on film credits, so the E was a sort of convenient compromise. By the time the first Richard Grant retired, the die, as it were, had been cast. Only they didn't believe him. He explained he was from Swaziland, where his father was a minister of education, and they took that as so much more old flannel, as though Hugh Grant (no relation) had claimed to be Chinese.

Somewhere back in Richard E's paternal ancestry, there were men who were Dutch or Hungarian and certainly Afrikaner. Yet he feels his father was an Englishman, working for the British government, and he, himself, is a Swazi who happens also to be English. Now living in Surrey, a brisk walk from where we sit, he will always classify himself as an immigrant. He used to wear two watches, one telling Greenwich Mean Time, the other the time of day in Swaziland. Swaziland was his home. Where he was born. Where he grew up and where his heart is. When called upon to sing at auditions, he would stand solemnly and belt out the Swazi national anthem. He didn't mean to be funny.

He patently enjoys talking about his homeland. The singular beauty of its landscape, what he refers to as the serenity of the indigenous population, the nefarious eccentricities of the European ruling class. "Swaziland is a small part of south-east Africa, the last country in the continent to gain its independence," he said, sounding rather as one of his father's kindly schoolmasters must have sounded as he stood by a British government-issue blackboard in front of a crowd of happy Swazi schoolkids. "The curious thing about Swaziland is that it is a one-tribe, single-language country, so, unlike almost everywhere else in Africa, they never had any intertribal warfare. It was a protectorate, which meant the colonials who lived there had been invited and regarded themselves as necessary and welcome by common consent. It was a very hermetically-sealed society," he went on, then dropped the schoolmasterly dirge, "It was a sort of equatorial Ealing. A most peculiar little enclave."

Nowadays, Swaziland could hardly be described in such rose-tinted terms if, indeed, it ever could. Its peculiarities include a notoriously profligate monarch who bats off any attempt at democracy, one of the world's highest HIV infection rates and subsistence on less than a dollar a day for the majority of the population.

The enclave of his memory has become a lifelong preoccupation for Richard E. The black/white social divide, the suburban White Mischief promiscuity of the colonial ladies and gentlemen, the contradictions between private and public lives, the pomposities and snobberies and hierarchies are all ludicrous in hindsight. But as children, we don't have that perspective. "When you're in the madhouse," he said, succinct as ever, "you don't know everyone's mad because it's your norm. You don't know anything else." He subscribes to the notion, "Give me a boy till he's seven and I will give you the man", and has had a good many years to brood from an adult perspective. And his affection for Swaziland is undiminished. Class barriers, hypocrisy, snobbery, highnesses and lownesses; these are the sources of our national comedy, he says. It was all so 1950s, so uniquely and typically English.

These are the things that shaped him, boy and man, like it or not. When, in the late 90s a producer asked him to write a screenplay for his own film, he felt he was already halfway there. Wah-Wah was in his head. All he had to do was write it down, which he did, in two and a half months flat. Then the fun started. His producer withdrew to take up social work in the West Indies and Richard E was left holding the script. From 1999 to 2004, he flogged what he refused to believe was a dead horse. Every producer, every finance company he went to, said no. "It's a chicken and egg situation," he said, "you've got to get name actors in order to get the finance, and in order to get the name actors you've got to bullshit that you've got the finance, while all the time you feel the whole thing could just unravel, the wheels come off the pram, everything conspires to make you sink into a pit of self-pity and despair." The idea of help from friendly thesps was fairly abhorrent to him. "When an actor asks you to read his script, your heart sinks. The number of scripts I've been given by actors that are so unbelievably terrible! It's well known that actors are lousy writers."

Still, he gritted his teeth and showed his dream script to Gabriel Byrne. And he liked it. So Richard E had found his father. And so it went on, one step forward, two steps back. Permission to film in his country granted by the King of Swaziland, more rewrites and castings and suddenly it's June 7 2004 and he has £4m, a star cast (Gabriel Byrne, Julie Walters, Miranda Richardson, Emily Watson, Celia Imrie) and seven weeks to make his movie.

"It was all a bit kick-bollocks-scramble-and-squeak," he said eloquently. "A bit like organising a fantastically huge wedding, and then just popping the little bride and groom figurine on top of the cake." He'd worried his guts out over his directorial debut, thinking he didn't have the technical know-how, but, when the crunch came, he hit the ground running. "I loved every nanosecond of it," he said. "I felt like the boy with the biggest train set, more Meccano gear than anyone else. I loved being asked 2,000 questions a day, storyboarding every move, knowing as though by instinct exactly where the camera had to be, because it was my story."

It was, he agreed, a kind of exorcism for him, but also "a fantastic treat" to go back into his past with actors to recreate the reality of his boyhood. It was as though the five years of boyhood encapsulated on film came - with the help of his friends and two lads, one 10, one 15, whom he calls his doppelgangers - to represent the whole cycle of his life, from the small boy who made a shoebox theatre with figures stuck on lollipop sticks, to the glove puppets, to marionettes on strings, to school plays, to amateur dramatics and drama school and film acting and then back again to Swaziland, watching his takes on a playback monitor a similar size to his original shoebox.

If Wah-Wah was a self-indulgence in its making, the finished product is a prime example of a genre rarely, if ever, attempted by British or American film-makers: a child's experience, impeccably observed through the narrow lens of the child's perspective. In the mid-20th century, there were French, Italian, Spanish, even Swedish examples of this, but Anglo-American influences gradually either sentimentalised or forgot the starkly one-dimensional reality of childhood, the solipsism of the child observing adult behaviour without the defence of detachment.

Wah-Wah opens with a small boy pretending to sleep in the back of a car while a man and woman copulate on the front seat, broadens to the edgy domesticity of the boy's home life, hones in on his facial tic, a mouth-gaping silent scream, then lingers on his private hobby as he holds two lollipop stick puppets, one in each hand, and has them shout Shuddup-Shuddup-Shuddup at each other. And so, with much Proustian detail, we go on. The boy and Swaziland get their independence in the end, or whatever it is we take for independence, and you can make what you will of the moral of the piece. Richard E says it's a love story, or it tells you how you pay for the choices you make in life. I think it's more a case of you don't always get what you want, but, like the Rolling Stones tell us, if you try sometimes, you get what you need. He quite liked that idea. He said he still felt the urge to do the silent scream, big as he is.

We chatted on about the film for a while: how he called it Wah-Wah because that was how his dad's second wife described the conversational tone of colonialists at their leisure; the country club's choice of Camelot for the am-dram treat for Princess Margaret's official visit to mark Independence Day; and how, driven by lack of white talent to include a black man in their production, they scrupulously whited-up his face with plimsoll cleaner so Margaret wouldn't notice. Even so, she made her excuses and left in the interval. Said she wasn't feeling well, apparently. It gradually emerged, to my astonishment, that give or take the odd tinkering with the timescale, Wah-Wah is not just true, but literally true, frame by frame. Richard E was surprised I was surprised.

· Wah-Wah opens the Edinburgh International Film Festival on August 17. Tickets can be ordered on 0131-623 8030 (or at edfilmfest.org.uk).

· Richard E Grant will appear in Simon Gray's Otherwise Engaged at the Criterion Theatre, London, from October 25, following a five-week regional tour.