When 2001: A Space Odyssey was first released, few would have predicted it would still be feted nearly half a century later. In fact few would have tipped it for even short-lived glory. At its premiere – its premiere – there were 241 walkouts, including Rock Hudson, who asked: "Will someone tell me what the hell this is about?"

Even its champions were stumped. "Somewhere between hypnotic and immensely boring," thought the New York Times; "Superb photography major asset to confusing, long-unfolding plot," reckoned Newsday. But bafflement was the intention, explained its creators. Said Arthur C Clarke, whose 1948 story The Sentinel was the starting point for Stanley Kubrick (Clarke's novelisation postdated the film): "If you understand 2001 completely, we failed. We wanted to raise far more questions than we answered."

A cop-out? Far from it: 2001 is magisterial. Its impeccable serious-mindedness is nothing to scoff at; what some saw as ponderous now seems merely prescient. It was both the last space-travel movie shot before men actually landed on the moon, and the first to turn a genre that had been the preserve of B-movie cheese into the highest form of art.

It looks not just as fresh as the day it was made but as fresh as the day you first saw it: iffy ape costumes aside, it's one of the few 60s movies that stands up to contemporary technical scrutiny. At the time, it must have marked a quantum leap forward: suddenly, space seemed credible. To watch even now is to be awestruck: all those exacting details (the 700-word instructions for using a zero-gravity toilet), the pacing – at once lulling and urgent, the audio – soaring Strauss waltzes spliced with dead air. In space, of course, no one can hear you speak.

And so to the difficult matter of what on earth it's about. On a bare-bones level, it concerns three artefacts: one left on Earth at the dawn of man by space explorers keen to steer the evolution of the apes, another buried deep in the lunar surface, and programmed to signal word of man's first journey into the universe (in Kubrick's words, "a kind of cosmic burglar alarm") and the third in orbit around Jupiter – another alarm, this time for when man breaks out of his own solar system.

And that's what happens in the film, when a team of five men (three in hibernation) jet off to investigate the second. But the mission goes awry – arguably the fault of the chatty command computer, Hal – and the sole surviving astronaut is swept into a force field that hurls him on a journey through the galaxy. From there it's to a human zoo, built from his own subconscious, where he ages fast, dies, is reborn and enhanced – "a star child, an angel, a superman" (said Kubrick) – before returning to Earth to advance evolution.

2001 is a film whose ambition is only matched by its achievement in pulling it off. It was the world's first – and perhaps only – metaphysical exploration of the workings of humanity, from the beginning of time to the far-flung future, and it's small wonder sci-fi has never really recovered. It has really only been going backwards, relying either on splashy effects or psychological conundrums handled so tritely that they barely seem related to 2001 at all.

Some complain that it is chilly, inhuman. Perhaps. But the dying song of Hal, warbling out Daisy Bell (A Bicycle Made for Two) as his plug is pulled, must be one of the most haunting scenes in cinema, mechanic or not.

Not to mention, of course, that 2001 provides the most open-and-shut case for cinema being primarily a visual medium: from the twirling, dancing orbits to the extended acid(ic) trip at the end, it is, quite simply, a knock-out. And it features Leonard Rossiter as a Russian astronaut. The question back in 1968 would have been: how could this possibly be number 1? The question today is: how could it not?