It’s Oscars season and Ang Lee is out in the cold. This is not a familiar place for him to be. The 62-year-old director, who was born and raised in Pingtung County, southern Taiwan, and studied film at New York University in the early 1980s, is accustomed to pleasing all of the people (industry, audience, critics) most of the time. Only John Ford, Frank Capra and William Wyler have won more Best Director Oscars; Lee has two, a tally equalled by Steven Spielberg and Clint Eastwood. He received his awards in 2006 for Brokeback Mountain, which he dedicated in his speech to his late father, and in 2013 for Life of Pi, which prompted him to thank the “movie god”; he beat Spielberg on both of those occasions. He had already collected the prize for Best Foreign Language Film in 2001 for Crouching Tiger, Hidden Dragon, which is still the highest-grossing foreign-language film released in the US. (It took $130m, more than twice the sum made by its closest rival, Life Is Beautiful.)

In Lee’s three award-winners, you have a taste of his range: a tender gay love story, a spiritual adventure rendered almost entirely in CGI and an elegant martial-arts adventure in which warriors walk up walls, skip nimbly across rooftops and leap between trees. Then there are the delightful comedies of manners (Pushing Hands, The Wedding Banquet, Eat Drink Man Woman) that introduced him to the world, the eloquent literary adaptation that started his Hollywood career (Sense and Sensibility) and the downbeat blockbuster (Hulk) that nearly ended it. Just when you think you’ve got his number, he’s off. What runs through these superficially dissimilar films, however, is a level-headed curiosity about human behaviour. A line from his sexually explicit espionage thriller Lust, Caution puts it best: “If you pay attention, nothing is trivial.”

There have been stumbles. Hulk sank. Taking Woodstock did not, on the whole, generate hippie-ish feelings of peace and love, though it remains a fascinating attempt to view history from the sidelines, not unlike his sexual revolution drama The Ice Storm and his American Civil War story Ride With the Devil. Lee’s latest film, Billy Lynn’s Long Halftime Walk, adapted from the novel by Ben Fountain, is similarly oblique: it’s a war film that focuses mainly on what happens once the heroes are home from Iraq. Their bravery is seized on by jingoistic strangers while their glory is exploited by bloodsucking businessmen. Lee always seems happiest here in the margins, telling stories about the misunderstood and the overlooked.

And now he’s there himself. Four months ago, the film was considered a sure-fire awards contender. That has proved not to be the case. “Quite the opposite,” the director admits with a chuckle, gazing into his cup of tea in a hotel room in London. His conversation is peppered with those chuckles. Sometimes they serve as a kind of social lubricant; like his soft voice and gentle manner, they enable candid statements to slip into the conversation almost unnoticed. But this particular chuckle feels as though it’s there to cushion the hurt.

Has the snub been painful? “Yes. I’m very proud of the movie. Perfect or not, I think it’s important. Everyone involved was committed to finding a new film language and that’s a very noble endeavour.”

He is referring to the striking – and, to my eyes, disastrous – decision to shoot the film in digital 3D and hyper-sharp 4K resolution, at a rate of 120 frames per second (rather than the usual 24). The picture is intended to feel immersive, to blur boundaries between audience and image, so that we experience sensory overload along with the shell-shocked soldiers. Instead, the technology makes it resemble an unforgivingly overlit rehearsal in which even the most experienced actors (Kristen Stewart, Vin Diesel, Steve Martin) look cruelly exposed. The music, with its standard emotional cues and crescendos, adds a layer of kitsch, as though someone had commissioned a Hollywood score for a home movie.

Lee concedes that he may have gone too far. “I took a big leap,” he says. “It may be too much for people to get used to. Also, the theatres don’t have the right layout or equipment and we still need to find a way to eliminate the 3D glasses. But I do think it’s a legitimate format. And if we’re going to shoot digitally, why should we be tied to 24 [frames per second]? It isn’t any good. There’s no reason to stay with it now we have digital.” He sounds exhausted, like someone at the fag end of an argument that has been raging all night.

Lee is surprised by how little interest Hollywood has in the new technology. “Maybe everyone is too fixated on the beauty of the old way. They don’t want to leave the paradise, the innocence. It’s frustrating.” But if he has stopped hoping that his film will be appreciated, he will settle for it making future innovations possible. “I don’t mind being the marine lying on the barbed wire so people can step on me and plant the flag. Though, of course, I’d rather be planting the flag in one move.”

Would he have made his earlier films in this style if the technology had been available? “I don’t think like that. Art comes out of what you don’t have. It is the compensation. In literature, you can’t visually see, you don’t have pictures, so you describe it and it appears in the mind better than ever. You create art by embracing those limitations.”

Lee knows a thing or two about the good that can spring from limitations. When he made Sense and Sensibility in 1995, his command of English was rudimentary. James Fleet, who starred in that film, told me last year: “Ang was at a bit of a loss with people arguing back. Actors saying, ‘I don’t think my character would do that.’ He would sit there in the French windows with his baseball cap on. He must have been thinking, ‘What am I going to say to these people?’ He didn’t have the language. You’d talk to him and you wouldn’t be any the wiser at the end of it.”

Occasionally, Lee would inadvertently hurt his actors’ feelings as he struggled to give direction. Emma Thompson joked that she felt like slitting her wrists after he told her not to give a “dull” look to the camera. Alan Rickman responded to Lee’s request that he “reduce, reduce, do more” by spluttering: “Well, which do you want, reduce or do more?” And the director mercilessly mimicked Hugh Grant. “I couldn’t find the words so I asked Hugh, ‘Can you please not do this . . . ?’” He performs for me a savagely accurate impression of the stammering fop routine that characterised most of Grant’s work in the 1990s. No wonder the actor coined two nicknames for him: “Fang Lee” and “the Brute”.

“It was weird for the actors because they usually work with either mean directors or nice ones,” Lee reflects. “And I was a nice guy saying mean things.”

But Sense and Sensibility turned out a treat. “When you see it, it’s all Ang,” Fleet told me. “It’s his personality. There’s a calmness to it, a compassion. That’s exactly what he’s like. Here is someone who didn’t seem to be in charge and yet his personality is still imprinted on the film.”

When I ask Lee whether he feels that something was lost once he became fluent in English, he responds instantly: “Yes. I don’t have the privilege to be blunt any more. I’ve got to do what everyone else does – ‘That was brilliant’, instead of ‘That was boring’. I don’t want to say I’m bullshitting actors but, well, I have to talk more smoothly.” When Lee was shooting Hulk, he was visited on set by Tobey Maguire, whom he had directed in The Ice Storm and Ride With the Devil. After the end of a take, Lee issued the standard instruction “Check the gate” (movie-speak for making sure there is no dirt or debris behind the lens) and Maguire cried out: “Don’t say, ‘Check the gate.’ Say, ‘Check gate! Check gate!’” Lee laughs fondly at the memory. “He missed the old me.”

As a director, he has been awfully good at turning actors into stars. In The Ice Storm, he caught Maguire before Spider-Man and Elijah Wood before The Lord of the Rings; Sense and Sensibility was Kate Winslet’s second big film, and Heath Ledger wasn’t a megastar until after Brokeback Mountain. For all the faults of Billy Lynn’s Long Halftime Walk, the performance by the newcomer Joe Alwyn is not among them; his prospects now look excellent. With the ­exception of Chow Yun-Fat in Crouching Tiger, Hidden Dragon and Tony Leung Chiu Wai in Lust, Caution, Lee has never given a lead role to an established star.

“This is quite so,” he says. “I like the combination of innocence and experience – like putting Emma with Kate. I found that pleasurable. More experience provides more depth but then the audience worries less for the experienced actor. With the unknown actor, you see them trying to give the director what he wants, trying not to screw up, and that genuine effort makes you worry for them. You don’t worry for Kate any more. But at 19, she makes your heart race.”

One film for which he is reluctant to take credit for shaping the performances is ­Brokeback Mountain, which came to him after passing through the hands of Pedro Almodóvar and Gus Van Sant (who had planned to make it with Leonardo DiCaprio). “I didn’t do much,” he shrugs. “But the actors were all good. They just gave it to me.

“I was very tired, not ambitious. I thought I was just doing this cheap art-house film. I’d done two big movies [Crouching Tiger and Hulk] and I was whacked. I made it very straightforwardly – secure performances, simple shots.”

The film has a personal significance for him. After being burned by the response to Hulk, he had briefly considered retiring, only to be scolded by his father, who had never even wanted him to be a director in the first place.

“He said, ‘You’re 49. You can’t retire. It’s a bad example for your kids. Go ahead and make a movie!’” Two weeks later, his father suffered a stroke and died. “So I had to do it. He’d told me to. I didn’t have time to grieve. I went straight into production on Brokeback. I was tired and reluctant but the movie nurtured me back to health. That’s when I felt there was a movie god who loves me and that I should continue because this is what I’m meant to do.”

“Billy Lynn’s Long Halftime Walk” will be released in 2D on 10 February