'I think that something really good can come from trying to exorcise some of your own personal demons in your work," says Ethan Hawke, sipping from a steaming bowl of pumpkin soup. We are sitting in the window of a cafe just around the corner from Hawke's apartment in Chelsea, New York, talking about his new film, The Woman in the Fifth, a small French-British co-production, directed by Polish director Pawel Pawlikowski. Hawke plays an American writer in Paris, newly divorced, seeking custody of his daughter, who is drawn into a phantasmal relationship with a beautiful muse (Kristin Scott Thomas). The character seems like a napkin sketch of Hawke himself.

"In many ways it is," he says. "That jacket he wears – I had this blazer that's a little too small, like it's left over from prep school. The costume designer and I saw it as an extension of the character from Dead Poets Society: you know, his eyes have gone bad, he used to be quite promising but promising doesn't cut it at 40. He's turning in on himself. Pawel had this idea that depression is someone who can't see outside of themselves."

A paranoia piece in the vein of early Roman Polanski (Hawke calls it "an art film from 1962"), the film feels like Hawke's most thorough exploration yet of the darker claret running through his own strain of romanticism. It's been there from the beginning of his career, when, as he puts it, "the pure-bred romanticism of Dead Poets Society kind of injected itself into my psyche." Acclaimed as the actor of his generation for his portrayal of a coffee-house philosopher in 1994's Reality Bites, Hawke was simultaneously boning up on the films of Michelangelo Antonioni and the early career of Jack Nicholson. "I was a little late," he says. "Two decades too late." He was stuck in the 90s, eulogising the 70s.

Then came divorce from Uma Thurman, amid rumours of an affair with the couple's nanny, and vilification by the tabloids as Public Ratbag No 1. Hawke holed up in the Chelsea hotel for a couple of years, and returned to the theatre. When he did resurface on screen, it was to contemplate adultery (Richard Linklater's Before Sunset), rob his own parents (Sidney Lumet's Before the Devil Knows You're Dead), or skim money from his fellow cops (Brooklyn's Finest). If the 90s felt like a walk in the park for the actor, the last decade has seemed like a lightning dance, a canter beneath gathering clouds.

"Even this action movie I did, Assault on Precinct 13 – I saw a piece of that on TV the other day and I just looked depressed," admits the 41-year-old. "The whole movie was depressed. It's Detroit. It's snowing. My character's a pillhead. It definitely infiltrated my life." His charisma has a slightly haggard intensity these days, his face gaunt, his trademark goatee a little scruffier than you remember it; he looks like a starved version of himself. But he is still a fabulous talker, making unbreakable eye contact during long, soulful riffs on the importance of keeping your personal flame alight that recall the Huck Finn-ish sneaker-clad boulevardier who talked Julie Delpy into getting off a train in Before Sunrise. But there's no mistaking the black halo Hawke wears these days. The guy has taken a beating – the worst of it, one suspects, self-administered.

"I call it the black years," he says of the period following his divorce in 2004, sequestered in the delicious post-punk rot of the Chelsea hotel. "It was really difficult. It was difficult in ways I couldn't even see at the time. There was the obvious way in which it was difficult – the death of a dream, the inability to parent in the way that you want to. But for me it was – what's that Dante quote? 'At the midpoint of my life, I've come to the part of the forest where the straight way is lost.' Nothing teaches you like getting levelled. And I got levelled in my early 30s. Nothing went exactly the way I thought it would. Wait a second: love isn't real and, holy shit, I put all this energy into not making the same mistake my parents did and I just re-enacted them all! I thought I was so much smarter than everybody. And I'm not."

What saved him was his first love: theatre. He took a role as a sleazoid actor in the David Rabe play Hurlyburly, then Hotspur in a production of Shakespeare's Henry IV ("I put every ounce of anger that I possibly possessed into that play and I tried to ventilate it. I thought, I can't control what people think about me, but the one thing I can control is whether I'm improving as a performer. And my bet is that if I work on doing that I'll probably also, as a domino effect, improve as a person. That's been the goal of the last five or six years.")

In 2008, he married the nanny, Ryan Shawhughes, the same nanny he was rumoured to have had an extramarital affair with; the couple now live together with their two children, Clementine (three) and Indiana (six months). He is finding fatherhood a lot easier the second time around. "I'm really the right age now. I like being home. When Mia [his eldest child] was born, I was 27. It was three years after Reality Bites came out. I look at pictures of myself and wonder: what was I thinking?" What was he thinking? "I think I thought when I was younger that I would, like, get things and then it would all be easy. You'd be able to walk whatever road you wanted and then sadly you would start having arthritis, but by then you'd have kids and everything would be fine. I didn't really understand what a meandering road adulthood is."

In 2009, he wrote a lengthy Rolling Stone profile of the singer and actor Kris Kristofferson, at one time the biggest multi-platform star in America, who took a single bullet (Heaven's Gate) and heard himself described as "washed up" on the radio. An act of self-nourishing hero worship, smuggled autobiography, and half-time pep talk, the article seemed to be Hawke's way of rallying himself before returning to the fight. The piece ends with Hawke digging out his 10-year-old son Levon's copy of Margery Williams children's classic The Velveteen Rabbit: "Real isn't how you are made," says the Skin Horse. "It's a thing that happens to you… It doesn't happen all at once. You become. It takes a long time. Generally by the time you are Real, most of your hair has been loved off, and your eyes drop out and you get loose in the joints and very shabby. But these things don't matter at all, because once you are Real you can't be ugly, except to people who don't understand."

Of all the actors of his generation, Hawke has probably stayed most loyal to indie's art-for-art's-sake ethos. He has been offered plenty of superhero parts over the years and turned them down; he auditioned for Titanic, then watched as DiCaprio's career soared. Does he wish he had cashed in a little more? Hasn't he earned the right to don silver spandex and halt the progress of tidal waves?

"This is the biggest struggle of my life, to be honest," he says. "I never know to what extent I have to feed the snake, you know. The times in my life I've tried to sell out have failed miserably. I did this Angelina Jolie horror film thinking it would be a big hit and it was terrible. When I've followed my heart it goes well. One of the most successful movies I did was Before Sunset: we made that film for the sole reason that we wanted to make that film. The trick is to shoot from your heart, and then when the kind of work that you like is back in fashion again, you'll seem like you've stood your ground."

He laughs – a strange conspiratorial cackle, in which you detect a slightly embattled note. It makes you want to hug him and tell him how much you love his recent work (Brooklyn's Finest, Before the Devil Knows You're Dead), but that you wish he would cut himself some slack and play off that still-burning charisma a little more. Instead, I end the interview the way everybody ends their meetings with Hawke, by telling him what they should do with the third Before Sunrise film, currently at the planning stage. Everyone does this, apparently: journalists, fans, directors, strangers.

"I'll go meet some fancypants director for a movie. They'll go, 'Oh this guy wants to meet you for this, that, the other.' I go, 'OK, I'll meet the guy.' We start talking about their film for a bit and then they go: 'I know what the third film should be.' And then he tells me. It's flattering, but we have to just tune it out and figure out what we want to say."

What does he want to say? "One of the things I want to get into the film is an acknowledgement of – and this is everybody's struggle – how do you keep your innocence alive? How do you keep your sense of romance alive, your sense of joy alive? But match it with realism, to get rid of all the fake naivety. To see the world for what it is. It's a very difficult aspect of life. What do you do?"