In between the stark futurism of A Clockwork Orange and the floodlit horror of The Shining, Stanley Kubrick made an 18th-century picaresque costume drama that was far less widely loved than either of those films but infinitely more devastating. Barry Lyndon follows the adventures of an opportunistic Irish nitwit, Redmond Barry (Ryan O’Neal), as he clambers inelegantly up the social ladder in search of a title and a fortune. Those who disliked the picture on its release in 1975 cited the pace, which even a snail might consider a tad slow. Defenders, such as Alexander Walker of the Evening Standard (“cinema to marvel at”) and Nigel Andrews of the Financial Times (“a near-masterpiece”) were outnumbered by doubters: Margaret Hinxman in the Sunday Telegraph found it “stupefyingly dull”, while Derek Malcolm in the Guardian’s pages accused the director of “tickling us to sleep”. Even Steven Spielberg, who later brought Kubrick’s unmade project AI: Artificial Intelligence to the screen, likened the experience of watching Barry Lyndon to “going through the Prado without lunch”.

What is now apparent is that the slow-burn approach allows emotion to reach us in acidic drops rather than obliterating waves. A doleful narrator, Michael Hordern, pre-empts everything that occurs on screen, even revealing, 45 minutes before the end of the film, what the outcome will be. “What is important is not what is going to happen,” Kubrick insisted, “but how it will happen.” Painterly wide-shots keep the action at arm’s length, while the dominant camera move is a slow backwards zoom that leaves the characters dwarfed by the landscape. They are playthings and puppets, jerked around cruelly by fate for reasons that remain obscure to them. The cast had good reason to feel the same way, though at least they knew who was pulling the strings.

Watch the trailer for the 2016 rerelease of Barry Lyndon

That was Kubrick, whose relationship to his actors has long been a source of fascination. His casting was always devilishly inspired. This, don’t forget, was the man who chose Tom Cruise for Eyes Wide Shut only after first crossing Steve Martin and Woody Allen off the list. Barry Lyndon is full of typically oddball choices: the Rising Damp star Leonard Rossiter, who had appeared briefly in 2001: A Space Odyssey, plays a priggish army captain, while Patrick Magee, the terrorised writer in A Clockwork Orange, resurfaces here as a card-sharp in eye-patch and face-powder. Magee was familiar with the director’s exacting methods. “The catchwords on set are: ‘Do it faster’, ‘Do it slower’, ‘Do it again,’” he said at the time. “Mostly, ‘Do it again.’”

O’Neal only survived the year-long shoot because of “a strong suspicion that I was involved in something great”. This strangely docile performer was a natural fit for Redmond Barry, who is too namby-pamby to retrieve a ribbon from a woman’s cleavage even when expressly invited, too witless to understand that you don’t negotiate with highwaymen. “God, [Stanley] works you hard,” the actor he wrote in his shooting diary. “He moves you, pushes you, helps you, gets cross with you but, above all, he teaches you the value of a good director.”

Kubrick’s unusual directing methods began with the auditions, which weren’t really auditions at all. The veteran British actor Murray Melvin was summoned to Dublin for a costume fitting for the part of Reverend Runt, companion to Barry’s wife, Lady Lyndon. “My agent said: ‘Don’t get carried away. People are coming and going. They get the part, he fires them.’ I arrived early in the morning and they did the costume. After a few hours there was still no sign of Stanley. ‘He’s on set,’ they said. ‘Have some lunch.’ At about six o’clock in the evening, I heard someone say: ‘He’s coming!’ The build-up was fit for a Roman emperor. I saw the entourage first, then he appeared. ‘Oh Stanley, you’re alive!’ Everyone gasped. He said: ‘Hello, Murray.’ He had a look at the costume. Asked me to turn around. Then he said, ‘Thank you, Murray’, and walked off. Two days later, I was told I got the part.”

Kubrick (centre, right) on the set, with Ryan O’Neal on left. Photograph: Warner Bros

Melvin was contracted initially for four scenes over three weeks. “I stayed for 24 weeks. Most of my scenes he made up as we went along. ‘Listen, Murray, we’re doing this scene and I think you should be in it …’” The model-turned-actor Marisa Berenson, who was 26 when Kubrick asked her to play the neglected Lady Lyndon, was on set for the entire year-long shoot. “Stanley was a very visual person so he needed to always see things in front of him to decide if he liked them or not,” she tells me. For that reason, the cast were not allowed stand-ins while shots were being set up: everyone had to be available, in costume, at all times. “He never allowed you to go anywhere. For three months, I was just sat there in costume, not working. He kept saying: ‘I may need you tomorrow.’”

Each morning brought fresh challenges. “The toughest part of Stanley’s day was finding the right first shot,” said O’Neal in 1975. “Once he did that, other shots fell into place. But he agonised over that first one. Once, when he was really stymied, he began to search through a book of 18th-century reproductions. He found a painting – I don’t remember which one – and posed Marisa and me exactly as if we were in that painting.”

Watch Murray Melvin and Dominic Savage in a scene from Barry Lyndon

The production was complicated by the use in some scenes of a highly sensitive lens developed specially for Nasa, which Kubrick used to capture authentically the texture of candlelight. “One scene with thousands of candles took a week to set up,” Melvin recalls. “Then it was scrapped. The second AD [assistant director] told me: ‘Stanley had a look and he doesn’t like it.’ That was when I fell in love with Stanley. I thought: ‘That’s power!’ A week to set up – and he doesn’t like it!”

Berenson also became fond of Kubrick, despite finding his lack of direction disconcerting. “He gave us tremendous freedom and let us do exactly what we wanted. Once he’s chosen you, he has total confidence. But he never directed me. I like to be directed, I like feedback, but there was never any idea of what he was looking for. He was a man of few words.”

Few words but many takes. Melvin estimates that for his climactic scene, a showdown with Barry’s mother (Marie Kean), Kubrick called “Action!” no fewer than 150 times. “Having finished it, the cinematographer John Alcott was heard to say, ‘I bet he uses the first one.’ To me, Stanley wasn’t difficult. He was demanding. You had to rise to the occasion – if you didn’t, you sank. But you got nothing from him. Just ‘Do it again.’ I was punch-drunk by the end of it. I was destroyed. I couldn’t even speak English any more. I’d say to Marie between takes, ‘What am I doing wrong?’ ‘I’ve no idea’, she’d say. ‘But keep going. I’ve a restaurant booked for tonight. We’ll have a nice bottle of wine.’”

Dominic Savage was 11 years old when he played the younger incarnation of Lady Lyndon’s son, Lord Bullingdon. “I remember Stanley with some affection,” he told me in 2009. “He was very tactile. His way of directing was incredibly subtle and low-key. He was so interested in me, and that made me feel interesting.” Savage went on to become a director himself. “I try to be as un-director-like as possible, and Stanley was like that. He was an experimenter. One scene we shot was rewritten three or four times in the days leading up to filming. I found it more amusing than disorienting. It was nice to see someone not necessarily sure of something. But he knew filming was a way of discovering that sureness.”

Another scene took O’Neal several days to nail to Kubrick’s satisfaction. Once it was finally in the can, the director moved on to the next set-up. “He found a way to walk past me, giving instructions to the crew – ‘Let’s move on to 32, move those lights into the foreground’ and so on – but as he passed me, he grabbed my hand and squeezed it. It was the most beautiful and appreciated gesture in my life. It was the greatest moment in my career.” Berenson says she feels “blessed” to have appeared in the film. “Not a day goes by without someone talking to me about Barry Lyndon. It puts a spell on people. I think it’s going to live forever.”

Barry Lyndon is on release nationwide from 29 July.