The first thing that viewers of the slippery new thriller American Animals will see is an unusual title card which reads: “This is not based on a true story.” The next thing we know, the words “not based on” have disappeared before our eyes. So is this film, which recounts a 2004 heist in Lexington, Kentucky, a drama or a documentary? Onscreen interviews with the criminals themselves – four bored young men who hatched a plot to steal valuable books (including Darwin’s On the Origin of Species) – tip the balance in favour of documentary. On the other hand, the film features fictionalised versions of the same people played by an able young cast, as well as the sort of glossy cinematography and adrenalised score one would expect from a high-calibre heist movie. The real-life subjects even interact occasionally with their surrogates. In one scene, Evan Peters, who plays the ringleader Warren Lipka, sits beside the real Lipka in a car and asks: “Is this how it happened?” The effect is rather as if Pirandello had made Reservoir Dogs. More than that, it demonstrates the unstable shape of documentary in the age of alternative facts.

Fiction cinema acknowledged as far back as Citizen Kane that there is no such thing as definitive truth when it comes to storytelling; the documentary genre has taken rather longer to reach the same conclusion. The turning point was Errol Morris’s 1988 film The Thin Blue Line, which incorporated reconstructions based on differing statements in a murder case, and felt as much like a thriller as a documentary. Daring hybrids in the last 15 years or so have made the genre a breeding ground for mischief and ambiguity. Joshua Oppenheimer’s 2012 The Act of Killing invites murderous Indonesian gangsters and paramilitaries to restage their own appalling crimes, all in the style of their favourite movies. In Stories We Tell, from the same year, the actor-turned-director Sarah Polley explores the subject of her own paternity while laying bare the fabrications of the form: we see her coaching her father on reading his lines, making him start again if he fluffs one. Two British landmarks – Clio Barnard’s The Arbor, where actors lip-synch eerily to recordings of real people, and Carol Morley’s Dreams of a Life, which uses reconstruction and speculation – have also played their part in muddying the waters.

American Animals takes the confusion further. Bart Layton, its British writer-director, has a background in documentary; his hit 2012 film The Imposter, about a family duped by a conman, also straddled the line between genres. “I don’t tend to worry about categories,” he tells me. “For me, it’s all about finding the best way to tell the story. We’ve all had that experience where you see ‘based on a true story’ at the start of a film, and you get that sinking feeling. You’re waiting for the photos of the real people at the end, and you know that everything will be wildly Hollywood-ised. With American Animals, I wanted to see if there was a new way to tell a true story. It’s definitely not a documentary but nor is it a straight fictionalising. The facts were outlandish enough without the need to invent anything.”

Poppy Dixon, one of the producers, visited the subjects back in 2010 when they were still serving time. “My job was to put them at ease. When we explained how Bart wanted to tell their story, they thought it sounded cool.”

She believes the unique texture of the film makes it well-placed to comment on the nature of memory. “Sure, there’s Donald Trump’s idea of truth. But we all recognise that we have imperfect memories. It’s honest to represent that, and the particular form of American Animals allows this to happen.”

Layton embraced the discrepancies in his subjects’ accounts: “I wanted to be clear from the beginning that these are some pretty unreliable narrators,” he says. Audiences who smelt a rat in Catfish or were duped by the Joaquin Phoenix mockumentary I’m Still Here may find it refreshing to see a film which confesses openly to its own irregularities.

“The financiers were nervous about the nonfiction bits,” Layton says. “They were wondering if I was going to take what could be a commercial movie and spoil it with these documentary elements. My feeling was that once you see the real people, you’ve got skin in the game. However far the characters go, you know it all really happened.”

Layton is not the only filmmaker currently blurring the distinctions between fiction and reality. He singles out Chloé Zhao’s The Rider, which casts a real-life rodeo rider and his family and friends as themselves in a retelling of events from his life – an approach that has been a mainstay of Iranian cinema since the 1980s. “It’s really rich and unusual,” Layton notes. “You don’t see those faces on screen normally, and you can see they’re not actors.”

The Australian filmmaker Kitty Green is also a pioneer in the genre. Her 2017 Netflix film Casting JonBenet shows actors auditioning for a (nonexistent) film about the unsolved murder of the child beauty-pageant star JonBenét Ramsey.

The result is a disturbing collection of their on-camera revelations and their responses to the murder, with Green using fiction as a tool for unearthing intimate emotional truths in the absence of objective ones.

Of course, there’s a risk that cross-breeds such as American Animals, The Rider and Casting JonBenet could leave conventional documentaries looking impoverished. “I don’t know if anyone’s done quite what Bart does,” Dixon says. “I’m not sure if anyone’s been brave enough. But it’s a challenge to documentary-makers, isn’t it? A really good true story is always better than a fictional one.”

American Animals is released on 7 September, The Rider on 14 September