It is a glorious New York spring day: blue skies, clear air, a fresh breeze. A day to skip along the pavement and feel happy to be alive. Just not in Woody Allen's world. Where it's a day, like every other day, ie, one in which to sit in a darkened room and ponder the futility of all human existence and the absence of meaning. Allen's own special brand of nihilism, as expounded in his films, is, of course, well known – equal parts despair and a sort of despairing joy – but entering his office makes me wonder if this is more than an intellectual choice; if he simply has a different physiology from the rest of us.

Because outside the birds are singing, the trees are about to burst into bloom, and yet his office, where he's worked for years, seems to be some sort of black hole, exerting a force field so powerful it swallows all light. The lobby, a gloomy marble sarcophagus, gives way to a suite of rooms lit only by dim puddles from low wattage bulbs. He's like one of those deep sea creatures who have simply evolved differently; the human equivalent of the Antarctic icefish, which has no red blood cells, or the vampire squid with a metabolic rate so low it's practically dead.

At the back is a screening room, which is gloomier still, decked out with a mossy green carpet and Draylon chairs that look like they last saw daylight sometime back in 1972 – but, then, it could be any time of day or night, any point in the past 40 years. There's a record player and a collection of jazz albums, and there, suddenly, in the gloaming, is Woody himself, who, give or take a whitening of the hair, and slight hunching of the shoulders, seems equally timeless. Even at 75 he looks pretty much as he's always looked, the spectacles in place, his eyebrows a cartoon question mark.

The reason for the interview is the UK release this week of You Will Meet a Tall Dark Stranger, the fourth film he's shot in Britain, and probably the best yet (if you can get over the shock of the opening scene in which Bridget Jones's mother is talking to Shirley Valentine. In a Woody Allen movie). In it, Alfie, played by Anthony Hopkins, leaves his wife, Helena, because "she became old and I wasn't prepared to accept it", and takes up with a twentysomething prostitute instead.

They are all here, the familiar subjects of Allen-esque despair. The feeling, as Alvy Singer explains at the beginning of Annie Hall, that life is nasty, brutish and cruel. But also too short. That death dominates life. And that nothing works out, ever. It's not a film a young man could have made. "No. I wouldn't have thought of it when I was young. It requires years of disillusionment, this is true," he says. The only happy characters in the film are the deluded ones, and the more powerfully deluded they are, the happier they seem. Helena, who takes up with a fortune teller and dabbles with the occult, is grinning like a loon by the end of the film.

"But then I've always felt that if the delusion works, it's great. I always think that people who have religious faith are always happier than people who do not. The problem is that it's not something you can adopt. It has to come naturally."

There's a brilliant sequence, which afterwards I think is the possibly the least romantic moment in any film ever, in which Sally, played by Naomi Watts, young, beautiful and trapped in an unhappy marriage, has a moment with her sexy, Spanish boss, Antonio Banderas. He obviously has feelings for her, as she does for him, and if she were a character in any other film, they'd eventually be together. Or maybe apart, but in a doomed, romantic way. Not here, though. It just doesn't happen, and they end up not together in the most banal of ways: the timing's off. She hesitates, and he falls in love with her friend instead. She takes consolation in her career but then that's thwarted too. It's a level of realism, the everyday realism of everyday life, that rarely reaches the screen.

In Woody Allen's universe there is no reason why some things happen and others not. His atheism allows no delusions of that kind, but what about age, I ask him? Do you, like Alfie, resist hearing that you're old?

"I do, I resist. I feel the only way you can get through life is distraction. And you can distract yourself in a million different ways, from turning on the television set and seeing who wins the meaningless soccer game, to going to the movies or listening to music. They're tricks that I've done and that many people do. You create problems in your life and it seems to the outside observer that you are self-destructive and it's foolish. But you're creating them because they're not mortal problems. They are problems that can be solved, or they can't be solved, and they're a little painful, perhaps, but they are not going to take your life away."

Problems such as, say, falling in love with Soon-Yi Previn, his partner, Mia Farrow's adopted daughter, and the sister to his three children – two adopted and one biological. It happened almost two decades ago, the hugest scandal at the time, with Farrow alleging abuse against another daughter, and although Allen and Soon-Yi have married, and have adopted two girls of their own, it still echoes down the years. Soon-Yi, 35 years Allen's junior, has no contact with Mia; Allen has no contact with his older three children. And You Will Meet a Tall Dark Stranger features that familiar Woody Allen trope: the May to December romance.

In previous interviews Allen has resisted any autobiographical interpretation to this, calling it a useful plot mechanism. Which it is. And here, as previously, in Whatever Works, Husbands and Wives and Manhattan, the relationship is both mined for comic effect – Hopkins counting down the minutes until his Viagra will work – and obviously doomed. Whereas Allen has previously spoken about his "luck" in finding Soon-Yi, and the tranquillity of their domestic life together.

In Wild Man Blues, the documentary Barbara Kopple made about Allen's love of jazz, in which she followed him and his band on tour, there are a few brief but telling insights into his and Soon-Yi's relationship. She's not afraid to call a spade a spade, is she, I say? "The crowd outside the concert was hilarious," she says at one point. "It was like for a rock concert, and yet you're an older guy." And, "you looked like a crazy person out there". And, "you must remember most people aren't coming because they like your music, they're coming because you are in the movies".

Soon-Yi, like Helena in the film, insists upon honesty. There's really no escape from reality when she's around. The difference is that, unlike Alfie, Allen seems to embrace it.

"Yes, yes, she's never taken me seriously really. And to this day – you know I just left her now – she sees me as a complainer, a hypochondriac, a kind of idiot savant. She thinks that I'm very good at what I do and absolutely terrible at everything else. And she's probably not far off. You know, it's that kind of relationship. She's not someone who sycophantically supports. You know, people thought when I first married her that, because of this big age difference, I'd married someone who'd idolise me. But that wasn't the case at all. She hadn't seen 90% of my movies, and to this day she hasn't seen 60% of them. She's just not that interested in them. And she's a stern critic of my work. She unashamedly hates my clarinet playing. Can't bear it. Can't bear my practising. Never comes to a concert. Thinks it's torture.

"In fact, if you were to see me around what you would call, but isn't really, my entourage, that is my wife, my sister, people close to me, loved ones, you'd think, 'The poor guy. They don't like his movies, they're critical of this, they're critical of that.' That's not really so true. They love me and are supportive in a meaningful way but they are very critical of what they would euphemistically call an eccentric. Although they think it's worse than an eccentric, it really is much more like an idiot savant."

Soon-Yi's behaviour reminds me of the story about the Roman emperors, I say, who were said to employ a slave during triumphal processions to whisper in their ear, "You're only mortal."

"Well I'm very aware of that. In fact, I'm hyper aware because of the fact that my wife is so much younger than me."

Do you worry about that?

"I worry about that only in that I hope that I live long enough to give her sufficient nourishment. It's almost a given that she'll outlive me. But I'd like to see the children up and launched."

But then, what comes across in You Will Meet a Tall Dark Stranger is how random it all is. How we meet one person but equally we could have met another person. Josh Brolin plays a writer, Roy, who becomes obsessed with the stranger in the window opposite. Does Allen think this is just the human condition? That we're all captivated by the stranger in the window?

"Yes, because you fill in the gaps. You see the woman in the window and you impute to that person the things you want to hear."

And, with Roy, it just seems completely arbitrary who he ends up with.

"Yes, we pick someone. And if your wife dies you get another wife. And in my movie Whatever Works, Larry David tried to kill himself by jumping out the window and lands on a woman, and that's a theme of mine, I was thinking about the other day… It was a theme of mine from the very first time I was ever associated with a movie, What's New Pussycat?, an abysmal film. Peter O'Toole, I think, was driving, and Ursula Andress parachuted out of a plane and happened to land in his car. And I touched on it again in Match Point: one of the characters at the museum is talking about how she hit him with a car and they wound up getting married. It's just all so capricious.

"Life is so much luck. And people are so frightened to admit that. They want to think that they control their life. They think 'I make my luck'. And you want to keep telling yourself that you're in control, but you're not in control. Ninety-nine per cent of it is luck, the luck of the genes, the luck of the draw, what happens during the day, the bomb that goes off on the other guy's bus."

His success has been luck, he says, although he's happy to take his failures as his own. He never watches his films – it's why he's so hazy on some of the details – never reads his reviews, finds the idea of his "legacy" abhorrent. And he rarely acknowledges either his tremendous work rate – he writes and directs a film a year – or his prodigious imagination. Or the fact that he's an entirely self-created man. He was born Allen Stewart Konigsberg, a Brooklyn Jew. Woody Allen, the urbane atheist of the upper east side, is very much his own invention.

At the start of the interview, while I'm fumbling to get out my first question, and he answers it in a sentence then sits and waits for the next, I think it's going to be a long, hard, pleasureless slog. There's a quote from an old Rolling Stone interview, which I've written in my notebook, that says: "During the entire session, he smiles three times – an event tantamount to the arrival of Halley's Comet, I later learned – and cracked not a s ingle joke."

And in recent YouTube footage I've watched of him, he seems to hide behind his glasses, appears, almost, to have no emotional engagement. So it's a surprise and a delight to see that he does smile. And he does joke. What's more, there's still a twinkle, the old twinkle, so familiar from Annie Hall and Hannah and Her Sisters, and the other golden age Woody Allen movies. It's still there. What I'm not sure is there – or at least if it is there, it's clamped down under a thick layer of slick sentences and smoothly articulated words – is Woody Allen.

There's a filmed interview with Diane Keaton in which she talks about how she met Allen – while auditioning for Play It Again, Sam – and how she came to be the inspiration for Annie Hall (her real surname is Hall), and how, after their relationship ended, she became his friend.

"The thing I don't know is how I came to be his friend. He's a very, very private person. And yet I am. I'm his friend."

Allen has mined so much of his life in his work, but more indirectly and obliquely than most people think. And it's what he hasn't mined that, in some ways, is most interesting. The heart of You Will Meet a Tall Dark Stranger, its emotional touchstone, and one of its central plot mechanisms, is a lost child. Two lost children. Alfie and Helena's dead son, and the child that Sally wants but hasn't been able to have. I read piles and piles of previous interviews but he never talks about his adopted daughter, Dylan (who now calls herself Malone ), and adopted son, Moshe (who changed his name to Moses), and his and Mia's biological son, Satchel (who is now an environmental activist and calls himself Ronan Seamus Farrow).

What strikes me, I say, is that if you look at your entire body of work, there are almost no children. It's almost as if the children are missing, and I just wondered in relation to your own life…

"There were never children in my films. I didn't care about children. I mean, I didn't dislike them but I just had no thoughts about children. If my wife, years ago, when I first married as a young man, had wanted to have 10 children, it was fine with me, or no children, it didn't matter. And then I remember when I did the movie Manhattan I made a list at the end of the movie of the things that made life worth living, and I got a letter from a lady saying, 'You didn't mention your child.' Because I had a child in the movie with Meryl Streep, a young boy.

"And you know I'd mentioned Louis Armstrong and Marlon Brando, and I figured, so, I didn't mention my child, so what? I mentioned things that were meaningful. It was only when I had children, over a decade later, that I realised what an egregious blunder that was. That of course someone with a child would mention it. So children have become very, very meaningful to me. Because they add an amazing dimension to your life. But even then, I'm aware of the fact that nature is nature in its most brutal way. You devote yourself to them, it's a one-way street, it's unconditional positive regard, no matter what happens, and they grow up and go out on their own and you become a very minor annoyance."

So do you have any contact with your children with Mia now, then?

"Oh. Contact with those children? No, no. I don't have contact with those children. I just have contact with my children."

Can you ever get over something like that?

"No, sure, that was a sad thing. And a very bad court decision, and not in the best interests of the children. But I found out, and I'm not saying this with bitterness, that the courts are not in the interests of protecting children. They're in the business of protecting themselves.

But doesn't it fill you with anger?

"You don't have any choice. There are things in life that are terrible, and you get over it. You move on. There's really nothing you can do. You try everything legally possible. In my case, I spent a fortune of money trying, and I tried for years, but that was the decision, and that's what it was. And at some point you realise you've lost this one, and there's nothing you can do. And now I've been married for, I don't know, 14 years or something, and I have kids growing up, and it's fine. That's all passed me.

"Now objectively – and I do feel I'm fairly objective about myself – I do feel that was a bad decision for everybody. The kids could have had… I'm a wonderful father. I really am a great father. I'm fabulous to my daughters, and the kids could have had wonderful opportunities with me but they were denied that. There was nothing I could do. There was nothing more I could have done. I had every lawyer I could, everybody testified for me that could, but I couldn't swing it."

It's hard to know what to make of this. The deliberate obtuseness about my question, which, I realise, when I listen back to my tape, is a Woody Allen special. He does it on several occasions, seems to confront my questions head on but then answers another question entirely. And then there's the emotional evasiveness of his reply. Which is fair enough – why spill your guts to a random journalist? But he really does give no hint of what's occurring beneath the surface. Of what, reaching the accommodation he articulates above, entailed. Of whether he really has put three children in a box labelled the past and pushed it out of sight. And if so, at what cost.

We've talked so much about delusion and denial, with reference to You Will Meet a Tall Dark Stranger, and yet there he is, seemingly, deluding and denying. To such a degree that it seemingly rules out any future hope of rapprochement or reconciliation. And, it is odd, no, to talk of "luck" in meeting Soon-Yi, and yet not acknowledge the bad luck that this entailed? A zero-sum game in which he gained a wife but lost a family.

But then he says he's ruthlessly unsentimental about the past, about his own personal history. Never keeps photos from his films, or scripts, throwing them away, or since he's realised he can get a tax deduction for it, giving everything to Princeton University. Although there's a hint of another Woody in comments Diane Keaton has made. She's talked on various occasions about his "strength", and has described him as having "balls of steel", although he dismisses this when I mention it. "She was probably just trying to find something nice to say."

He seems to have achieved domestic harmony with Soon-Yi but there's no doubt that the relationship with Keaton is in some ways the relationship of his life. (In Wild Man Blues, Soon-Yi says she's never watched Annie Hall, and who can blame her?) They're still close, still "very, very good friends", and while he says there's no way he'd entertain the idea of an Annie Hall sequel, he does want to work with her again.

He's written himself into a small part in the film he's shooting in Rome this summer, "but it's not a big deal, it's an amusing turn, so I'll do it. But I can't be the love interest any more. I can't play opposite Scarlett Johansson, it's not appropriate. So what can I do? I'd love to have a wonderful tour de force part with Diane Keaton but the problem is… doing what?"

A romantic part opposite Scarlett Johansson might not be appropriate, I say, but opposite Diane Keaton it would be.

"Nobody wants to see two septuagenarians get it on."

Why not? It'd be very romantic.

"People may say they do but they don't. They want to see Leonardo DiCaprio chasing after Scarlett Johansson. They don't want me flirting with Diane Keaton."

I think he's wrong, particularly since his films seem to have entered a new, mature period. And since Diane Keaton is the touchstone for his greatest films, his greatest characters, his entire sensibility towards women. He's written roles that have won his actresses Oscars, and he's one of the few directors who has both objectified and subjectified his female characters. They may be gorgeous beauties who drive men crazy but they're also fully realised beings themselves.

"I could never write female characters when I started out. And when I met Diane Keaton, and got friendly with her, and lived with her for a few years, I became so enamoured of her, I just fell in love with her. I became so enamoured of her as a human being, so in awe of her, that I started to write for her. I wrote Annie Hall for her, and then after that I could almost only write for women characters.

"They were cardboard figures before her, and I made no effort to change it, but after I met Keaton I could write women, and only write women, that was all that interested me."

And he seems to collect them still, always has his eyes open for a new ingenue, the next star. His next film, Midnight in Paris, which will open the Cannes Festival in May, stars the French first lady, Carla Bruni-Sarkozy. He met her at a lunch at the Elysée. "I had a small part, and I thought why not?" All the stories about her being difficult are rubbish, he says. And the film? He's such a harsh critic of his oeuvre, always hates the end result, is disparaging about his work, and yet it's been rumoured that he actually likes it.

"I feel affectionate towards it," he says.

Is that better or worse than normal?

"I'm always disappointed in my movies. We bring them in here to see them and it's always like taking a cold shower. It's never fun. But… this film I have great affection for because I have great affection for Paris, and that comes through in this film."

And not London? You've no plans to do for London what you did to Manhattan, set it to Gershwin and fireworks and celebrate it from on high?

"I love London. But London suggests… other sorts of stories."

Hmm. Well, you can't have everything. And You Will Meet a Tall Dark Stranger is not Annie Hall, because nothing else is, but it is a return to form from one of the all-time greats. And one day we'll have a Woody-Keaton combo again. He's promised us that.

You Will Meet a Tall Dark Stranger opens on Friday