It begins, as all Jean Claude van Damme films should, with a brutal and bloody brawl. Fists flailing and kicks flying, Van Damme marshals machine guns and knives, hand grenades and revolvers, dodges bullets and dispatches villains in a wartime wasteland. All the time the camera tracks around him in one smooth, uninterrupted three-minute shot with no cuts. It is beautifully choreographed and directed, but all rather so what - until we see a prop wall fall, and it becomes apparent that this fight sequence is a scene from a film inside a film. Van Damme confronts the director to complain about the faulty set. "It's very difficult for me to do everything in one shot," he says wearily. "I'm 47 years old."

It is at this point that you realise that JCVD is not just another Jean Claude van Damme film, and that the prop wall is not the only barrier to come tumbling down; JCVD is all about Van Damme demolishing the wall between fiction and real life in a performance that easily bears comparison with Mickey Rourke's portrayal of an ageing fighter in The Wrestler. I could say JCVD will make you forget every Jean Claude van Damme film you have ever seen but, being Guardian readers, it's not too likely you will have seen many Van Damme films to forget. Better, perhaps, to say that you can safely forget everything you think you know about Van Damme; in this new film he plays a world-weary, beaten-down former action movie star in a startlingly honest performance that is scarcely believable from someone more used to letting his high kicks do the talking.

In JCVD - the title, of course, is his initials - Van Damme plays himself: no longer the celebrated "Muscles from Brussels", but a washed-up erstwhile star who is running out of money, embroiled in a bitter custody battle for his daughter, and who keeps losing jobs to his nemesis Steven Seagal. (Seagal even beats him to one role by agreeing to cut off his ponytail.) There's more: while attempting to get a money order at a Belgian post office Van Damme inadvertently becomes caught up in a robbery. The police believe Van Damme has cracked and is the perpetrator, while inside the real criminals, despite being Van Damme fans, use the situation to their advantage by framing the actor. Imagine an episode of Extras, scripted by Charlie Kaufman and directed by Michel Gondry, and you are somewhere close to JCVD. Which raises the questions of what on earth Van Damme is doing in such a great film, and when did he become such a good actor?

Van Damme, his handler warns me ahead of time, is very tired. He has not slept for the last 24 hours, having been locked in an editing suite working on his next film, and his answers are liable to roam and ramble aimlessly. When I finally get to talk to him, he does occasionally answer a question I do not ask and sometimes wanders down some conversational cul-de-sacs - but he is, amazingly, also thoughtful, candid and refreshingly humble. He tells me that the inspiration for JCVD was a French television documentary in which he had talked candidly about his public image and career mistakes. The director of the documentary wrote a screenplay that played with the idea of Van Damme being embroiled in a bank robbery in which hostages have been taken. The conceit was that Van Damme the screen action hero would be as powerless when faced with a real crisis as anyone else.

That script was seen by a young French director called Mabrouk el Mechri. "I was approached by Mabrouk," Van Damme says. "He told me he was a huge fan of mine and that he had grown up with my films, and he said he wanted to make a film with me. He knew everything about me and he had an idea for a film that would be half-real, half-fiction, something really special." El Mechri says he believed "Jean-Claude Van Damme was a prisoner of what he became," and that he wanted to make a film that would "rediscover the man who stands in the shadow of his own image, and explore what that would feel like". El Mechri was dermined to separate the actor from his own mythology and refused to allow any members of Van Damme's entourage on the set. When they met, Van Damme says: "He told me that I was a great actor, but I was doing the same film again and again. He said he wanted to do something different with me."

It isn't only Van Damme, speaking in his native French, who impresses in the film. El Mechri employs a fractured narrative structure that revisits scenes from multiple perspectives and recalls Rashômon, films with fluid camerawork and muted colour tones, and announces himself as an inventive new director. And the critical response to JCVD has seen Van Damme receiving all-too-rare acclaim for his acting. But Van Damme wasn't enthused at first. "When I saw it for the first time," he says, "I was disturbed. It was not like I thought: 'Hey, I have made a great movie.' No; it left me disturbed for a couple of days."

That isn't surprising; Van Damme's acting, if that is what you can call it, is at times almost uncomfortably affecting. I asked him he thought hardcore Van Damme fans would make of this detour into arthouse cinema. "I don't know if this film is going to help my box-office career," he says. "The fans who liked my earlier films, who watch films to escape and have some fun, this won't be their cup of tea. But I think I will gain a different kind of audience."

Van Damme has taken a circuitous route towards becoming an arthouse star. He was born in Belgium and was introduced to martial arts by his father at the age of 11. After winning karate and bodybuilding championships, he moved to Hong Kong and then, in 1981, to Los Angeles - he arrived there barely able to speak English, but still hoping for a film career. He worked as a pizza delivery man and carpet layer until he won a small role alongside Chuck Norris in Missing in Action. His big break came when he was walking the streets one day and spotted a film producer whom he impressed with a demonstration of kickboxing.

That chance meeting offered him the chance to forge a career, which he did in early films such as Bloodsport and Kickboxer, which showcased Van Damme's martial arts skills. In the 90s he graduated to sub-Stallone action films such as Timecop, and below-par Schwarzeneggeresque fare such as Universal Soldier. Compared to Sly and Arnie, the Muscles from Brussels was a bargain-basement attraction, the Eurotrash video version of Hollywood's genuine action stars.

In the following years Van Damme slipped off the radar of the multiplex moviegoer. His career hit the skids, floundering in that part of hell reserved for the stars of films destined to bypass cinemas and go straight to the rental stores. Meanwhile, his personal problems were stacking up: five marriages, an addiction to cocaine, an arrest for drunk driving and accusations of spousal abuse.

What makes his performance in JCVD so compelling, though, is that after a career of beating up others, Van Damme beats himself up; we see him confronting his failures and facing up to the man he has become. The most extraordinary exhibition of that honesty is a seven-minute semi-improvised monologue to camera, with lights and rigging equipment behind him, in which he makes an unflinching assessment of his life: the marriage failures, the drugs, the debts and the estranged children. "When you got it all, you travel the world," he tells us directly. "You're the prima donna of the penthouse, so you want something more. I tried something and I got hooked. I was wasted mentally and physically, but I got out."

He continues, his eyes watery and his voice cracking: "It's not my fault if I was cut out to be a star. I asked for it, believed in it. When you're 13, you believe in your dream. Well, it came true for me but I still ask myself what have I done on this earth." He is weeping when he answers his own question. "Nothing! I have done nothing!" It is horribly mesmerising and a complete blurring of the line between fact and fiction. JCVD is not a documentary, but watching that scene is surely as close as one can get to a glimpse inside the mind of Jean Claude van Damme.

The first casualty of celebrity is often self-awareness, but the reason JCVD works so well is that its star is painfully aware of how he and his films are perceived. "I felt completely naked doing that monologue," he says now. "It was like I opened the fruit, peeled the skin, I cut the pulp, I got the pit and cut that and showed that to the audience." I ask him if everything in that monologue was based on his real life. "Real acting is telling the truth," he replies. "You have to believe what you say, and if you believe what you are saying, then acting is easy." But the debts and the drugs; was it really like that? "In real life, what happened was more than that," he says. "It was softened for the film."

The greatest enemy any action hero has to face is the clock; the ageing process is not easy for anyone, but for these performers it poses particular challenges. When you are feted for your physique and your ability to take on physical challenges that appear impossible, what do you do when time has eaten away at your muscles and stiffened your joints? Schwarzenegger became a politician; Stallone became Rocky and Rambo again, to critical derision; and Seagal became a joke. Van Damme says the experience of making JCVD has inspired him to take more risks, artistically, but he is not yet ready to say farewell to the genre that made his name. "I am planning on making sequels to Bloodsport and Double Impact," he says, "but the action films I will make in the future will be more believable and character-based. I am now on my second cycle of fame, and I want to make films that smell real and are truthful."

He has finally achieved some stability in his personal life: he remarried his bodybuilder second wife, is reunited with his children and has settled down in Hong Kong - and the experience of making JCVD has given him renewed confidence about the future. "I now truly believe it is impossible for me to make a bad movie," he says. "When you see me in JCVD, the way I am in that film, it is very difficult for me to go back to doing three flips in the air and landing on the table saying, 'I'm Jean Claude van Damme.'"

Our time is almost at an end and Van Damme needs to get some sleep. I have one final question: is he afraid of getting old? He pauses for a second. "You know, I looked at my face in the mirror this morning," he says, "and I like being old. My face has more content and when I train in the gym now, I am not training to be strong or handsome - just better than I was yesterday. These days the race is just against myself."

• JCVD is released on 30 January