This is weird. The documentary film-maker Errol Morris says he likes the Guardian – "It's my favourite paper" – but, sitting in the lobby of a sleekly manicured hotel in New York's SoHo district to talk about his work, it's not clear if he likes documentaries very much. "This is going to get me depressed," he groans. "I feel as if I became a documentary film-maker only because I had writer's block for four decades. There's no other good reason. I don't know what I should be doing. I'm tired of everything – mostly of myself."

It's weird not because Morris is being downbeat – after all, he once had a magazine column entitled The Grump; a typical post on his Twitter account reads: "I think I'm suffering from Irritable Mind Syndrome" – but because it makes him sound like a failure. How can that be? His first feature, Gates Of Heaven (1978), is rated by Roger Ebert as one of the 10 best films of all time. The Thin Blue Line (1988) helped get its subject – a prisoner sentenced to life for killing a police officer – sprung from jail. The Fog Of War (2003), which offered unprecedented access to Robert McNamara, architect of the US war against Vietnam, won an Oscar.

Talk to any film-maker and they'll say the same thing: Morris, who was born in 1948, is a revolutionary. His documentaries create reality as much as they capture it. He sees them as art, as essays, as anything but anthropology. Railing against cinéma vérité and its philosophy of "shaky camera equals truth", he opts instead for dramatic reconstructions, obtrusive soundtracks, striking angles. He also invented and employs a machine called the Interrotron (its name a synthesis of "interview" and "terror") that makes interviewees appear to be talking directly to the film viewer.

Tabloid, his new film, is as strange as any Morris has ever made. It's the story of Joyce McKinney, a beauty queen and former Miss Wyoming who in 1977 hired a private aeroplane and with accomplices travelled to England to track down her boyfriend Kirk whom she believed had been abducted by the Mormon church. After kidnapping him, she whisked him off to a cottage in Devon where, depending on whose story you believe, she spent the next three days either raping him as he lay spread-eagled in bed or having a merry feast of "fun, food and sex".

As soon as she was captured by the police, a Fleet Street circulation war broke out. Was she, as the Daily Express claimed, an old-fashioned heroine straight out of a medieval romance? Or was she, as the Daily Mirror claimed to have discovered, an ex-hooker specialising in S&M? For a while, she became a "celebrity", stealing Joan Collins's thunder at the premiere of The Stud and being kissed by the Who's Keith Moon, before escaping back to America. Cue more high dramas. Phone taps, mysterious burglaries, dog-cloning. "Joyce is an epic character," says Morris. "Love should stop short of insanity. This is the counter example."

Characters: Morris loves them. Gates Of Heaven was about idiosyncratic California businessmen running a pet cemetery; Vernon, Florida (1981), about a strange nook of America whose locals carried out insurance fraud using self-dismemberment, featured an odd cast of worm farmers, turtle keepers and turkey hunters; Fast, Cheap & Out Of Control (1997) talked to a hairless-mole-rats expert, a topiarist and a scientist who designed bug-like robots. They paved the way for documentarians such as Louis Theroux and Jon Ronson. But they have also led some critics to accuse him of trivialising his subjects.

"'How dare you call Joyce a character?' That's a question I've been asked a lot," Morris admits with a disconcertingly bright-toothed smile which reveals the relish this self-professed "Jew from Long Island" takes at being seen as an outsider at the documentary club. "There's this idea that interviews never involve an element of performance. Or that they shouldn't: it becomes a kind of moral prohibition. But would you prefer to be called flat, lacklustre and boring? I don't think so. I think calling someone a character is a compliment. And I find Joyce one of the strangest characters I have ever encountered. I like preserving her crazy dream on film."

Joyce, it turns out, is less keen. She has taken to showing up unannounced at screenings around America to rail against it. At first, when I call her to ask why, she's not pleased to hear from me. "You're from the Guardian? They usually lie." She says she's stressed. "A horse fell on me. I can't walk. My leg's in a brace. I totalled my car, too. I'm car-less." She's also busy – talking on another line with a friend who's been entrusted to collect the ashes of her dead horse, and on another line to a lawyer: "The journalist says the Errol Morris film is gonna play in England! We can sue him in England!"

Tabloid tells the story of Joyce McKinney, who in 1977 flew to England to track down the boyfriend she believed had been ­abducted by the Mormon church. Photograph: Kobal Collection

For the next half hour she delivers a highly eloquent if almost entirely impossible-to-follow denunciation of her former boyfriend's faith. My notes read: "Mormons – penis electrodes – Utah – pastor's dog – black people."

"About Kirk," I begin.

"Errol Morris says I kidnapped him. That's a slander. I rescued him from the cult Mormon machine. Did you know I did my doctoral studies in film? I thought Errol Morris was a respectable film-maker but he used misleading jump cuts and composite shots and made my story a funny story. It's not! I'm an Erin Brockovich-style victim of a financial corporation masquerading as a religion."

Joyce is as relentlessly dramatic as she appears in the film. She oscillates between sentimentality and low blows (she dismisses one of the accomplices in the film as "that little midget pilot"), and is capable of shifting from sobs and conspiracy theories to hysterical laughter within a few sentences. "I was popular – like Kate Middleton. Errol Morris is a deceiver and a master manipulator. His trashy, perverted film makes me look like a bimbo, slut, sexpot. I'm suing him for $50m. Maybe more. The creep!

"I thought he had a brain. I had no idea he was a nutcase. He does freaks. My life was destroyed by paparazzi and he said I had a sordid past! My mum tried to kill herself and is in a coma. Now my father is $325k in debt because of medical bills. Errol Morris was working in collusion with the Mormons. And he made fun of my dogs!"

When I put the accusations to Morris, he emits a long sigh. "Joyce likes to portray herself as a victim – of the Mormon church, the tabloids, of me. If she is, she's a kind of crazily triumphant victim. Look: if you go to the UK with a posse of heavies, and take with you a Los Angeles police department regulation Smith & Wesson, handcuffs, a toy but realistic pistol, chloroform, rope, with the intention of 'abducting' your former boyfriend, you're not a total 100% victim if the tabloids come after you. A story like this is made to order. It's like a gift from heaven. It adds to the imaginative possibility of life."

Perhaps. But it's hard not to wince at the antics of the tabloid journalists in the film. Peter Tory of the Express admits his paper, which paid Joyce £40,000 for her story and would go on to portray her as a nun, embellished details about her chaining Kirk to his bed: "I think it was ropes but chains sound better." Kent Gavin, a photographer for the Daily Mirror, boasts of tricking another former boyfriend into offering nude photographs (whose authenticity she denies). Later he even starts laughing at the memory of seeing her hanging off a hotel balcony threatening suicide.

Isn't this precisely the kind of exploitative behaviour that gives the press a bad name and that reached its awful acme in the News Of The World phone-hacking scandal? "Actually, I love tabloids. When I grew up it was the National Enquirer. I remember my favourite cover story; I saved it for years and still have it somewhere – 'Man Kills Wife Then Carves "I Am Sorry" On The Stomach'. And there was a picture! You didn't know exactly what it was a picture of, but it had really bad lettering. That to me was the essence of tabloids. It was all blood and guts. Usually there was the bad pun and the ridiculous headline, too – 'Headless Body Found In Topless Bar'."

For Morris, then, tabloids represent a species of American poetics whose energy, pulpiness and earthy humour is the antithesis of the New Yorker or the New York Times. At their best they tell epic stories of the everyday and tap into a strain of folk surrealism. But, he laments, the golden era of the tabloid has passed. "The culture has changed and now they're just about celebrities and politicians and a different kind of scandal-mongering.

"Maybe it's to be found on television – on America's Most Wanted or in reality programmes. As for the tabloid reporters in my film, were they guilty of bad taste? Of sleaziness? Yes! But they didn't cross the line into criminality. That's what makes them different from the current scandals."

Have you ever tapped a phone? Morris pauses. "A friend once said: if the ends don't justify the means, then what does? Here I have to be careful. I don't want to go into all the things I did investigating Randall Adams in The Thin Blue Line. But I thought there'd been a terrible, terrible case of injustice – and I was determined to get the evidence I needed to show an innocent man had been sentenced to death. There's a difference between wanting to resolve an investigation and trying to pump up a story because you're trying to sell newspapers."

Any other crimes you'd admit to? "Well, I've always been interested in murderers – back in the 70s I was the only person to have interviewed Ed Gein, the guy whom Psycho was based on – but there was one other murderer I really wanted to interview. She was locked up in a mental hospital and I wasn't allowed to interview her. So I broke into the hospital. The door was open but I was caught. At that time I was a student at the University of California and I had all these letters of recommendation – I shouldn't be revealing this to you – from the head of the school of criminology. Later he said: 'Try to avoid doing that.'"

It's a story that is testament to the director's doggedness. As a combative student he had an ashtray flung at him by the distinguished historian of science Thomas Kuhn whose philosophical doctrines he kept questioning. Werner Herzog, whom he befriended in his 20s, said he'd eat his shoe if Morris ever finished Gates Of Heaven, but as Les Blank's film Werner Herzog Eats His Shoe (1980) later documented, he ended up doing just that.

For a couple of years he worked as a private detective investigating rogue traders and securities frauds on Wall Street. "In The Thin Blue Line, to all intents and purposes I'm a private detective working not for a Fortune 500 company, but for myself. That's much, much worse. When you're working for yourself and your own obsession with finding the truth, you're at your own mercy."

Morris's interest in murder has its funny side – he married art historian Julia Sheehan at Brooklyn Criminal Court: "It was between two prostitution cases. The judge hit on my wife's sister. He had a huge box of rat poison at the base of the American flag" – but mostly it's part of his lifelong interest in what he describes as "making a connection with people who generally fail to make connections".

"The people who fascinate me have created an alterative universe for themselves. Take Ed Gein: his brother had died under mysterious circumstances. His mother had died and he was left alone in the most desolate place imaginable. Gradually he created a fantasy world for himself using the parts of dead people."

Perhaps it's this idea of "fantasy world" that makes the America of Morris's films as lonesome as an Edward Hopper painting, as unsettling as a David Lynch movie. On a good day, their subjects are eccentrics, oddballs, dreamers. On a bad day – and as with Fred Leuchter, the Holocaust-denying electric-chair expert in Mr Death (1999) – they're deluded, irrational, isolationist.

According to essayist and friend Lawrence Weschler, Morris reveals America as "absurd in the old existential sense. It's in flight from coherence." Touching The Void director and documentary historian Kevin Macdonald likens Morris to Woody Allen: "He's a great comedian with fantastic turns of phrase, but basically he has a very pessimistic view of mankind." He's right. Morris is gabby, amiably self-deprecating and liable to come out with lines as blackly funny as those in a play by one of his favourite writers, Samuel Beckett: "I once went to see a distasteful psychiatrist at Princeton. He asked me: 'Do you ever have unwanted thoughts?' Unwanted thoughts? What other kind are there?"

But it's hard to believe Morris is merely performing melancholia or acting out the character of Errol the Grump for the sake of this interview. "Maybe existence is ultimately a lonely thing," he suggests. There's a pause. "Or comic. It can be both, can't it? Some of the most tragic visions are also the most absurd. You might say the same about Joyce McKinney. There's something ludicrous about her story. But also something moving, too. Ludicrous and moving: that's what makes it tragic.".

• Tabloid is released on 11 November.

• This article was amended on 31 October 2011. The original incorrectly spelled the name of the director of Werner Herzog Eats His Shoe as Les Back. This has been corrected.