Xavier Dolan’s It’s Only the End of the World is histrionic and claustrophobic: deliberately oppressive and pretty well pop-eyed in its madness – and yet a brilliant, stylised and hallucinatory evocation of family dysfunction: a companion piece in some ways to the epic shouting match that was Dolan’s earlier movie, Mommy. This is a pressure cooker of anxiety, a film with the dials turned up to 12. Watching it, listening to it, is like having your head in the speaker bin for a Motörhead concert.

That’s not everyone’s cup of tea. I’ve heard it denounced as “insufferable”. Dolan has made insufferable films in the past – his fey, musing films like the interminable Laurence Anyways, in 2012 – but this isn’t one of them, and the uncompromising ear bashing here is an intentional, black comic effect.

It is an adaptation of a stage play of the same name by the prize-winning French dramatist Jean-Luc Lagarce, who died of an Aids-related illness in 1995. It is the story of a prize-winning French dramatist who returns to his hometown after an absence of more than a decade, with the intention of telling his family that he is dying.

For this, Dolan has assembled an A-list French cast: Gaspard Ulliel plays Louis, the writer in question; Nathalie Baye is his genial, wittering widowed mother. Léa Seydoux is Suzanne, his sullen, punky sister who respects what Louis has achieved, albeit in a mood of resentment that he has not carried her along with him as a kindred spirit. Vincent Cassel – his inverted triangle of a face permanently set in a scowl – is his brother, Antoine, who has a blue-collar job as a tool-maker and is married to mousy and submissive Catherine, played by Marion Cotillard in a style not far from Olivia Colman. However much these family members might have wanted to keep things nice and polite, it is of course futile. The moment Louis steps through the door, the screaming starts.

It is a nightmare: stylised, unreal. We see them in the woozy way Louis sees them. Or perhaps this is the dream that he is later having about the family reunion. For most of the film, Dolan brings his camera tight in for extreme closeups on the characters’ faces. In fact, the action is almost just a sequence of faces, either square on or in profile, and they are almost always quarrelling or shouting. And Dolan keeps a clamorous orchestral score surging through the querulous dialogue. Occasionally, his own memories will cause a power surge of euphoria to crash through, but these are soon submerged again in the ongoing melee. Louis looks very ill, but it is not merely his illness. It is a form of nervous breakdown, mingled with guilt and fear. Being back among his family is causing something like anaphylactic shock.

The point is that they are not always like this. Dolan shows us that Louis’s family probably rub along reasonably well in Louis’s absence. He has caused this pain, whether he wanted to or not. They resent his success, to some degree. But it is more that they are hurt that he refused to contact them while his career was exploding, except in a supercilious series of cryptic little postcards. By returning home, he has triggered an outburst of precisely that toxic discontent which drove him away, only a hundred times worse, as if saved up for him: a kind of mass Tourette’s aria of anger.

Louis has to find the right moment to tell them that he is dying – a revelation of victimhood that will reverse their status relationship and possibly make them resent him more than ever. And perhaps he feels that staying in this family felt like dying, and leaving them felt like living. The paradox is unbearable.

It’s Only the End of the World is a deeply pessimistic film on the subject of “family”: which emerges as not a supportive, nurturing institution, but something unbearable and clinging. The movie might appear to raise the possibility that we can soothe the pain by talking about things – and that talking about things is dramatically inevitable. Yet these assumptions are also upended. It’s Only the End of the World is confrontational absurdism: a fascinating, sustained assault.

• It’s Only the End of the World is screening at Sydney film festival 2016