Werner Herzog has teamed up with David Lynch to make a film about a disturbed actor who kills his own mother with a sword. Jeremy Kay sits in on the final day of filming

Werner Herzog's output has swung perilously close to mainstream of late. His 2006 Vietnam war action film Rescue Dawn was mostly well-received, while his Bad Lieutenant was released last month to widespread admiration. Has the famously idiosyncratic German director gone straight? Not a chance. His latest movie, which screens at the Edinburgh international film festival this week, sees him back on the fringes of artistic expression.

My Son, My Son, What Have Ye Done has baffled many US critics, but bewitched others, who have declared it a nightmarish gem. Based loosely on the true story of Mark Yavorsky, a San Diego actor who in 1979 killed his mother using a sword from a production of The Oresteia (itself a story of matricide), the movie lunges back and forth in time. Herzog gives us the narrative in splinters, switching from police procedural to flashback, marrying passages of stygian gloom with moments of absurd comedy; there are echoes of early works such as Aguirre: the Wrath of God and Even Dwarfs Started Small.

The film was the subject of intense speculation even before shooting began. For one thing, it would mark the first collaboration between Herzog (director) and David Lynch (executive producer). Then there was the cast: Willem Dafoe, Chloë Sevigny and Michael Shannon, the latter a rising star who outshone Kate Winslet and Leonardo DiCaprio in 2008's Revolutionary Road. And let's not forget the charged subject matter. When a friend extended an invitation to visit the final day of the US shoot, it was hard to resist.

My first thought, as I arrived at the Westin Bonaventure hotel in downtown LA last April, was that this was simply too ordinary a building to house a movie that promised such mysteries. The crew had been toiling since early morning to set up the first shot. Herzog arrived at midday, in combat trousers and a black fleece top, looking invigorated and happy, as if he'd just completed a rigorous trek through the Amazon.

In fact, he had. Herzog had just emerged from the Amazon basin in Peru, where he had been shooting at a site not far from where he filmed Aguirre almost three decades earlier. Shooting needed to take place at the time of year when the Urubamba river is at its highest, most treacherous level: in this crucial scene, Shannon's character Brad McCullum (based on Yavorsky) looks on as his friends die in an accident on the water.

As Herzog conferred quietly with the crew at the hotel's breakfast bar, among the business travellers and attendees at an ice-skating symposium, Eric Bassett, Lynch's business partner of the last 10 years, filled me in. "A little voice inside McCullum's head tells him not to take part in the river trip. He listens to it and this keeps him alive. Later, it tells him to kill his mother."

Suddenly someone yelled: "Action!" Shannon, wild-haired, wearing a moustache and a smart blue blazer, got up from a piano next to the breakfast bar. Grace Zabriskie, a Lynch regular who played Sarah Palmer in Twin Peaks and portrays McCullum's ill-fated mother here, walked up to him. The pair shuffled off towards a bank of elevators and rode up to the second floor. Udo Kier, playing McCullum's drama teacher, followed close behind in a plum velvet suit, his laser-beam eyes trained on the carpet.

"Cut!" The trio returned to the piano and Herzog consulted with Shannon. Kier swigged from a bottle of water and chatted with the crew. Zabriskie studied her shoes. Herzog called them back, and they went again, and then again, although this time the director conferred with Zabriskie while Shannon sat by the piano, his head lowered. Again they were off, and again he called them back for another take. "What's the scene about?" I asked a nearby grip. "They ride an elevator," he revealed, adding: "I will say this about Werner. Once Werner's filming, he's focused."

It was clearly going to be a long day. What were we on now, take eight? "Thirteen," a production assistant mouthed to me. Men and women in headsets held clipboards and shifted from foot to foot. Herzog, looking good in his 60s, seemed impervious to physical discomfort as he consulted with his cast, peering through the camera lens and thrusting his arm after filming to signal that they would have to do it all over again.

"Cut!" Arm thrust. "Cut!" Arm thrust. "Cut!" Round and round they went, until gradually the alchemy seemed to work: Shannon, Zabriskie and Kier dissolved into McCullum, his mother and his drama teacher, gliding around before us in solemn unity. Finally, Herzog was pleased. He had got his shot. To my untrained eye, it was a carbon copy of the previous one, and the one before that. But the director was happy.

Some of the crew took a brief lunch break. Others set up the next shot on the second floor, in which the characters emerge from the elevator and walk past the hotel gym, where a man wearing an oxygen mask could be seen running on a treadmill. He would need the mask: Herzog kept him busy.

The film tells the story of McCullum in non-linear fashion. We learn within minutes of the matricide. Dafoe's detective gets involved, and the police lay siege to McCullum's home, which constitutes the bulk of the action. Sevigny plays McCullum's fiancee. The scene I watched was set in Toronto: McCullum, fired from a production of The Oresteia for being too disruptive, returns to his hotel after watching a performance.

Beware of the running director

Herzog likes to shoot guerrilla-style. The hotel granted him permission to shoot, but it was up to the crew to work around the guests. At one point during the second shot, running in a crouch behind his cinematographer as if about to board a helicopter, Herzog dodged furniture and gave one startled onlooker a vigorous thumbs-up in thanks for not interfering with the actors gliding towards him.

At reception, I bumped into Herb Golder, a classics professor at Boston University who brought the story of Yavorsky to his friend Herzog; together they wrote the screenplay. "This young man walks out of a rehearsal of Aeschylus's Oresteia and kills his mother," Golder said. "The story was intriguing, so I hired a detective who found the guy, and we developed a relationship." Yavorsky was found guilty of involuntary manslaughter. A judge subsequently ruled him innocent on grounds of insanity, and he was sent to a secure hospital. He was later released, and died in 2003.

In the movie, Golder explained, McCullum is disillusioned by life and finds solace in drama, which be believes to contain real truth. "It's really about a precocious man who lives in two worlds and finds himself depressed by the artificiality of everyday reality. He starts looking for a world elsewhere, and the tragedy is that this results in the death of his own mother."

At 5.07pm, the crew called it a wrap. Everybody was whooping and clapping each other on the back. I grabbed a few moments with Shannon, who spoke in a booming, breathless voice and strode about like a sprinter winding down after a race. "Werner is a legendary director and he knows that. He's very certain about what he wants," he said.

He walked off, and I spied Herzog on his way outside to catch a taxi. "Shannon is going to be very big after this," he told me in his slightly strained Germanic monotone. "I told him I wanted him for the lead, so as a warm-up I invited him for a small part in Bad Lieutenant." Herzog said he was about to fly to Kashgar in China for the final segment of filming on My Son, but asked me not to mention this until after he had secured the permits. He was planning to strap a camera to Shannon's head as he walked through a busy market. "I was there a long time ago and always wanted to go back and film there."

Looking over his shoulder for the taxi, he added: "With this movie I wanted to make a point – that you can make a high-calibre theatrical feature for under $2m [£1.4m]. What you see on the screen will look like it's $40m. That's the way to handle things when there's a crisis in movie financing."

And what of Lynch? "We like and respect each other. We know each other from a distance. I said I have a project, and he said we should do it together, and I said we would not do it together, but his name would be on it as executive producer." A taxi pulled up. What was My Son about, I asked. "It's a murder story," he said, smiling faintly as he stepped inside. "A bizarre murder story."

My Son, My Son, What Have Ye Done is at the Filmhouse, Edinburgh (0131-623 8030), tonight and Friday.