One must hope that audiences appreciate the extraordinary scene with which the documentary Cartel Land opens: deep within a forest in Mexico, a group of men cook crystal meth, and discuss their metier.

“We know what harm we do with all the drugs,” muses one of them, masked. “But what are we going to do? We come from poverty. If we were doing well, we’d be like you, travelling the world or doing good jobs” – he addresses the director directly, but it could be many of us, almost accusing our good fortune, in contrast to his lot. “But if we start paying attention to our hearts, then we’ll get screwed over. We will do this as long as God allows it. And every day we make more, because this is not going to end, right?”

He’s right. No, it is never going to end; Mexico’s narco-cartel war now counts an estimated 100,000 dead and 20,000 missing so far. But there’s a twist: as we embark on this film exploring Mexico’s nightmare, we have no idea who these men are, or who they work for, although this is a scene for which many documentary directors would give their right arm – and may have had to, literally.

‘I’d never been a war reporter… I’d never been to a conflict zone in my life’: Matthew Heineman during the filming of Cartel Land in Mexico. Photograph: Hans-Maximo Musielik / Dogwoof

Mexico’s war has set new standards in difficulty and danger for journalists and film-makers. Reporters are regularly and horribly killed trying to chronicle it, and while even mass killers such as Osama bin Laden and Radovan Karadžic entertained journalists, the cartels and those who back them maintain a terrifying inaccessibility – they control the message their way, with one-way communiques of their choosing and hallmark ultraviolence.

So reporters and film-makers have hitherto tended to cover the war mainly through contact with its victims, though one important recent film, The Legend of Shorty, reached the family and lair of Joaquín “El Chapo” Guzmán, boss of the Sinaloa cartel, who recently escaped a second time from a top-security jail in Mexico. But now, with Cartel Land, US director and cinematographer Matthew Heineman captures a new depth –plumbs a further circle of hell – in his vivid portrayal of Mexico’s carnage. He does this by attaching himself initially to the Autodefensas vigilante group, which took a stand in the western coastal state of Michoacán against the Knights Templar cartel.

The Autodefensas were an armed citizens’ revolt against the cartel’s reign of terror and extortion in their daily lives, in the absence of any attempt to end it by the authorities. They came from all walks of life – the group filmed by Heineman was formed by a doctor – and began to roll back territory against the cartel, village by village, town by town. The Templars are a new and small cartel, so that such a movement was possible in the fluid cartel-scape of Michoacán, in a way that would be unthinkable in, say, terrain under the grip of the Sinaloa cartel, Mexico’s most powerful, or in Tamaulipas – controlled by the Gulf cartel and the terrifying narco-militia Los Zetas.

A scene from Cartel Land Photograph: Everett/REX Shutterstock

In those places, as the veteran correspondent Dudley Althaus notes: “ain’t nobody taken up that [vigilante] baton”. But the Autodefensas uprising in Michoacán provided reporters with a chance to cover forbidden terrain; many journalists came to tell the story for the nightly news, and then left, job done. But for his film – executive-produced by Oscar-winner Kathryn Bigelow – Heineman stayed and stayed, until he was under the skin of this horror, cheek-by-jowl with its faces, recording its human emotions and savage reality. The result is raw and relentless film-making; truth-telling cinema that captures a dark kernel of Mexico’s nightmare like no other to date. It will stand among the great documentaries on Mexico, and the most powerful in years.

“I’d never been a war reporter,” says Heineman. “My last film was about healthcare in America, and I’ve never been to a conflict zone in my life.” This film was originally about vigilantes in Arizona – freelancers who patrol the Mexican border against migrants and drug-traffickers, and it retains some scenes with them. “Until,” says Heineman, “my dad sent me a clipping about the self-defence militias in Mexico. Immediately, when I read it, I knew I wanted to create a parallel story about vigilantes on both sides of the border.

'It’s so hard to understand the pleasure they take in their sadism, often hopped up on meth or other drugs' Matthew Heineman

“But then I just got deeper and deeper in. I got access I never thought I would, and felt a huge responsibility to tell this story – to take people behind the headlines and dramatisation on TV like Breaking Bad. I wanted to show what is really happening in our neighbouring country, with which we have so much in common. The drug war is something that connects both countries, and it’s a war we fund with our consumption of drugs in the US.” (In the film’s wake, says Heineman, addicts and former addicts are among those who come to panel discussions, anxious to learn about the origins, and human cost, of their habit.)

Indeed, Heineman got deeper in than almost any non-Mexican to have covered this war. “From the first moment I set foot in Mexico, I wanted to get into a meth lab. Most of the meth consumed in the US comes from Mexico, and much of it from Michoacán. Everywhere we went, I asked people: do you know somebody who is connected? But for four months, we failed. Then, finally, we got a call: be in this town square at 6pm. We were met by masked men, who drove us through towns, smaller towns, and then fields, until we stopped – and a separate car arrived to drive us.

“When we arrived, it was night-time. I don’t shoot with lights, so I couldn’t believe that I had come this far, and wasn’t able to see anything with my camera in the darkness. But then the head chef started showing me the lab with a flashlight, and with this flashlight, I lit the scene.” The torchlight gives Heineman’s cinematic coup a surreal and highly effective drama.

Heineman conveys what it’s like to live in cartel territory with footage of adults and children being buried in a mass grave, killed while picking limes because their employer had failed to pay the required extortion money. An aunt to some of the children stares at the camera, as she explains: “They were the innocents.”

A subsequent scene is even more harrowing. A woman with “her soul sucked out”, as Heineman puts it, tells the story of her kidnapping by the Knights Templar, and being forced to witness the torture, mutilation, decapitation and incineration of her husband and four other people. She also stares directly to camera as she speaks in a flat tone, her face pitted. After the orgy of killing, “they played with me and did whatever they wanted”, but then they released her, saying that “what I had witnessed was my punishment. It would make me crazy and suffer all of my life.”

In one scene from Cartel Land, Heineman shoots a meth lab in dramatic torchlight. Photograph: Everett/Rex Shutterstock

“I was not mentally prepared for the shootouts, torture, and other intense moments that I experienced first-hand,” says Heineman. “But far harder, and scarier in a way, was listening to her story. Not just the fact that the cartel would do this, but the sadism of her punishment was madness.”

“They were laughing like crazy people,” the woman recalls , invoking a usually unaddressed but crucial theme in Mexico’s violence – its recreational nature. “It’s so hard to understand the joy with which they do all this,” says Heineman, “the pleasure they take in their sadism, often hopped up on meth or other drugs.”

The principal torturers had the noms-de-narco Chaneque and Caballo, and in the unforgettable scene that follows, the Autodefensa vigilantes go after them. The shootout is loud and terrifying, the quarry captured and hauled into the open, manhandled, punched and kicked. “And when you see these guys,” reflects Heineman, “the men who did these appalling things are so frighteningly normal-looking, like they could be the guys at the local gas station.”

Heineman captured this episode by recourse to the two things that propel much good journalism: luck and hunch. “We were all exhausted,” he recalls, “desperate for a break, after shooting 18-20 hours a day for three weeks. We got to the airport, and I had this sinking feeling that we weren’t finished, that things were happening that we hadn’t been able to get on camera. I pulled my bag off the scale, sent the crew on and called my local fixer: ‘Will you go back in?’”

And despite not being a war cameraman – though perhaps by now an accidental war reporter – Heineman found himself filming one of the great passages in all war photography: “Suddenly I was alone with my camera in the middle of this shootout, too scared to process how dangerous it was – I just focused on my craft, on exposure, framing, and technique. I don’t know anything about being in a shootout, but I know how to film – so I did.”

The Guardian documentary: From California gang to Mexican vigilante: the family man fighting the drug cartels in Mexico. Guardian

The film has been very well received in the US, but Heineman says some critics have suggested that it lacks explanatory context. This arises, one presumes, from that infuriatingly ubiquitous desire to wheel on “experts” for their views, or punctuate observation with commentary. “That is precisely what I did not want to do with this film,” retorts Heineman. “My goal was to capture the story as it unfolds in real time, as people experience it, through my subjects’ eyes, not outside experts or some official government statement. He adds, with a laugh: “Though the Mexican government did send an attache, wanting to be part of this.”

At the heart of the film, is the journey of the charismatic Dr José Mireles, a small-town physician known as “El Doctor”, who founded and leads the Autodefensas in Michoacán. Unimpeachable in his motives, Mireles epitomises the story Heineman thought he was telling when he first arrived: “At first, it seemed to be a tale of good fighting evil.”

But the cruel beauty and philosophical depth of the film lie in its capturing of a very different reality, as the vigilantes themselves are joined – Mireles says “infiltrated” – by gunmen who appear to be former cartel members and who promise rapid progress with their ample weaponry. Mireles is a complex character who is betrayed by his closest lieutenants. By the final act of the film, the vigilantes “start to become the evil that they are fighting against, in concert with a corrupt government,” says Heineman.

“Things changed, and changed scarily for us, too,” says Heineman, “because we didn’t know who we were talking to. Were they the Autodefensas we thought they were, was it the cartel, was it the police, or the police working with the cartel? What was going to happen? Who can we trust, who trusts us?”

The Cartel Land trailer

This story unfolds on two levels. One is universal, the narrative of Animal Farm or Lord of the Flies: the inevitable corruption of power, especially that achieved by force of arms. “The film is both timely and somewhat timeless,” says Heineman. “The story of everyday people rising up to fight against evil to protect their families – it’s a story that we’ve seen play out throughout history and across the world today.”

On the other level, Heineman’s film contributes cogently to the blowing of the myth that the Mexican state is fighting, rather than conjoining, organised crime – so that this film is also a tributary, with a lusty current, flowing into a river of great work by Heineman’s – and my – colleagues.

Reporter Anabel Hernández argues that far from escaping from jail first time round in a laundry truck, “Chapo” Guzmán left in police uniform with a police escort, the day after the justice minister arrived on the scene to react to his “escape”. She more recently exposed the hand of federal authorities in the mass kidnapping of 43 students in Iguala, handed over to a narco death squad for execution. In Guillermo Galdos and Angus Macqueen’s The Legend of Shorty, Guzmán’s men coolly boast about paying off the soldiers supposedly hunting him down.

The tunnel-building that enabled Guzmán’s second, recent escape can only have been a deep inside job. Reporter Sandra Rodríguez Nieto is peeling away circumstances surrounding the recent murders of photojournalist Rubén Espinosa and activist Nadia Vera, leading to questioning by investigators of the governor of Veracruz (who denied any involvement). Writer Lydia Cacho has for years exposed state complicity in narco violence and criminal syndicates.

And now we have Cartel Land, which seems to encapsulate the whole story in one, jaw-dropping closing scene, a bookend that echoes the opening. “After filming in the meth lab, I asked to come back, and we made a date to meet up at a designated spot the next night,” explains Heineman. “But they didn’t show up, nor the next night, nor the next. But the fourth night we decided to make our own way to the lab.” The film crew was stopped, questioned and “told to wait at a nearby pool hall, in the middle of nowhere. It turned out this is where the meth cooks hung out, and it was one of the most cinematic scenes I’ve ever witnessed. There was a man talking to me in English, explaining it all: the connection between the meth cooks, the Autodefensas, and the government. But I knew that it would be dangerous to bring the camera out. Eventually, we got the green light to go back into the lab to get the shots that I needed.” Several weeks later, on our final shoot, I randomly ran into the man from the pool hall, who had all of the juice about what was really happening, and he agreed to be filmed. This interview ended up being integral to the final act of the film. What a lot there is in that final moment.”

There certainly is. The man wears an official state police T-shirt, and speaks through a bandanna in fluent American English. “Certainly”, he explains, “cartels, mafia – whatever you wanna call them– are involved in this. They’re involved in everything. Everything has gotten corrupted. The Autodefensas and the people who cook the meth are pretty much the same team”. But, he adds, “We cooks, we gotta lay low now that we’re part of the government. Us selling drugs, cooking drugs, it doesn’t look right. But it’s always going to happen. It’s just not going to stop, period. It’s a never-ending story”.

Cartel Land is released on Friday 4 September