When Werner Herzog says a film is the most frightening and most surreal he's seen in at least a decade, you know need to steel yourself. He's right. Here's the best, and the most horrific, movie of this year's Toronto film festival.

It's a documentary about the Indonesian death squads of the mid-1960s who tortured and killed communists. But it's also a film within a film, as director Joshua Oppenheimer urges the ageing gangsters to recreate their acts on increasingly elaborate scale (prosthetics, props, drag outfits, soundtrack, location shooting). They grin and mug just as they also take it very, very seriously. A strangulation scene is interrupted by the call for evening prayers. But they return after their ablutions.

At first you suspect this will riff on familiar ground, with the main interviewees, former members of paramilitary organisation Pancasila Youth, explaining how they were inspired in both their look and sadism by the movies. The most charismatic of them, Anwar Congo, who has a radical hair dye job a third of the way through in reaction to the rushes, is slightly haunted by his acts, he admits, which he tries to forget with music and dancing. And booze. And marijuana. And ecstasy. Others are scornful of any regret: "You feel haunted because your mind is weak. It's just a nerve imbalance."

The frank corruption of politics, the glee of the media ("One wink and they were dead!" grins a local publisher), the killers' happy embrace by the government ("We need gangsters to get things done", says a senior minister) – this sheer giddy chutzpah has remarkable cumulative effect. We watch these gangsters (everyone emphasises their belief that the word stems from "free men") talking trash to female caddies on the golf course, waxing lyrical about the merits a life of "relax and Rolex".

Some have criticised Oppenheimer for not interviewing anyone who survived the ordeal. It doesn't matter. We know this was genocide. We know that they'd be likely to feel fairly aggrieved. These men hoist themselves and do more besides. The most extraordinary scene comes during one of the recreations. One of Anwar's neighbours, who is moonlighting as the victim, laughingly suggests they use in the film a story that he has. It's of a man – ok, it's his stepfather, he says – who was dragged from his bed at 3am by the death squad, to the sound of the screams of his wife and children (that's him, he laughs, that's me!). The next day they found his body beneath a barrel and then buried it by the side of the road, "like a goat", so frightened were they that they too would be taken. The percolation of reaction among the men listening is the most compelling thing you'll ever seen.

It's often said of documentaries that they deserve to have as wide an audience as possible. This doesn't deserve; it demands – not for what it says about present-day Indonesia or even about its former horrors. But because almost every frame is astonishing.