Wait: is it really almost 20 years since Terrence Malick stumbled out of the desert and into his comeback? Apparently yes – it was Christmas 1998 when the director released his grandly oblique war movie The Thin Red Line, his first film since 1978.

By pure good luck, I was in New York and saw it the weekend it opened, in a cinema humming with the sense of a moment. Back then, this was special. For two decades, no one had thought there would even be another Malick film. The Bigfoot of US cinema had made two unalloyed classics before the age of 35 in Badlands and Days of Heaven, announcing himself as the world’s hottest young auteur. Then he disappeared – or at least quit the movies. Remarkably, given the stakes, the new film was a triumph. Nearly everyone at the sold-out screening was rapt. Nearly. Midway through the three-hour film, a man in the front row got to his feet. “This is bullshit,” he shouted, and began walking out. “Bullshit!” he repeated, loudly.

On Friday, Malick’s latest film, Song to Song, premieres at SXSW in Austin, Texas. The location makes sense – Austin is the setting for the movie, in which a beautiful-people buffet of Ryan Gosling, Michael Fassbender, Natalie Portman and Rooney Mara play lovers and rivals against a backdrop of the city’s music scene. Yet the mood feels different, as it has for much of Malick’s recent career. Of course, there are diehards to whom he remains sacred. For a certain kind of movie star too, he is still the one director for whom they will risk the raw indignity of being dropped from the finished film. (Occasionally, a ticked-off actor will appear in the press like a defector from a secret state. “[He] just cuts to birds,” complained Christopher Plummer in 2011, having starred, more or less, in The New World, the account of Pochahontas and Captain John Smith with which Malick followed The Thin Red Line.)

But the irate New Yorker of 1998 would now find more takers for his take on Malick. Since then, the gaps between films have shortened and the catcalls increased, the graph of public opinion a ski-slope. Seven years separated The Thin Red Line and The New World, generally admired, occasionally adored; six went by before The Tree of Life, a dense collage of childhood memory and cosmic awe that brought the first squints of unease. Then the trouble really started. A year later, Malick released To the Wonder, a whispery portrait of a sombre Oklahoman falling in and out of love, played by Ben Affleck with his usual air of blank constipation. Last year, it was joined by Knight of Cups, with Christian Bale cast as a jaded Hollywood scriptwriter glancing over a conveyor belt of attractive young women. Critics reached for pitchforks.

Malick, it is safe to assume, loses little sleep over opinion. For that we should applaud him. Who knows, it may well be that he is now working so far ahead of the curve that we will only realise how great his later films were shortly before we all die. But for now, even among his natural fanbase, much of his recent output has felt irritating, his reputation further battered with each film. Which means Song to Song feels as critical a moment as was The Thin Red Line.

With the modern Malick, the problem most often griped about is the surface of it all, for ever wobbling on the brink of self-parody with the Hello-Sun-Hello-Sky camera-work and gnomic voiceovers on the nature of desire. The comparison to perfume advertising has become its own cliche, winked at last month when Malick unveiled an actual perfume ad for Guerlain starring billowing fabric and Angelia Jolie. (It looked like a Terrence Malick film.) But the issues go deeper.

Take Knight of Cups. If you tensed slightly at the thought of Malick sending Bale on a search for the perfect life partner, the reality was lumpier still: a film about bearded men brooding meaningfully while women frolicked nearby, occasionally barefoot. The sight of the film’s female cast, their characters not always given names, scampering ahead of the camera was glum – and odd for a director whose first two films drew much of their brilliance from the spiky deadpan of Sissy Spacek and Linda Manz.

For those grown weary of stoned profundity, meanwhile, a low was reached when a kohl-eyed starlet (Imogen Poots) wandered through a Los Angeles sunrise, approached a homeless figure asleep on a concrete bench and left a flower beside them. A flower.

Out-of-time and out of date are different things. One of the stranger aspects of Malick’s career is that having first made films exclusively set in the past, it was only when he moved into the present that they felt old-fashioned. Modern young(ish) people have become his muse. Now, Malick might cut to birds or a ketamine party (both will look lovely). But it feels persistently fuddy-duddy.

Partly it’s his frame of reference – not all films about young musicians would involve the Red Hot Chili Peppers. But in the political tinderbox of 2017, a director can say a lot simply through the people they choose to make films about – and it’s a choice to make them about emotionally stunted LA screenwriters and lovesick Austin rock stars. Of course, what really concerns Malick is the stuff that floats above the world. The problem is, sadly, most of us are still stuck down here.

Then again, there are no films he could make that would be good enough to compete with his own past. The truth is, while in the 90s his absence was frequently mourned, in fact most film lovers savoured it. Having sat out 20 tatty, hair-gelled years, there was no Dune or Color of Money to ding the mystique. Instead, they were flawless relics of the kind of 70s splendour the stupid film business didn’t make any more. Malick was gone, and his vanishing act was both a glorious pre-internet mystery and an implicit scold. He was too good for us.

So imagine the disappointment when he popped up again. It was as if JD Salinger had suddenly appeared on a late-night chatshow, discussing how he liked giving flowers to the homeless. I don’t know that Malick has ever quite been forgiven.

But here’s the thing. Around the same time Poots appears in Knight of Cups, there is a shot of a dog from underwater, manically trying to grab a ball in its jaws. Of course, the very motion that causes then keeps pushing the ball just an inch out of reach. It is perfect – a teasing philosophical idea, caught in a single, endlessly eloquent image. In the course of one of the more annoying films I’ve seen in recent years, it instantly became one of my favourite scenes in cinema. It probably crosses my mind every day. Which is a way of saying that, even now, when it comes to Song to Song, and every Malick film from now until doomsday, although I’ll understand if you want to walk out, I’m staying put. And don’t shout “bullshit” too loudly.