My favourite childhood reading in the 1950s included the Eagle comic, especially the adventures of Dan Dare, Pilot of the Future – where the brilliant artwork depicted orbiting cities, jet packs and alien invaders. When spaceflight became real, the suits worn by the Soviet cosmonauts (and their US astronaut counterparts) were already familiar, as were the routines of launching and docking. My generation avidly followed the succession of heroic pioneering exploits such as Yuri Gagarin’s first orbital flight and Alexei Leonov’s first spacewalk.

But of course the day most etched in our memories is 20 July 1969, when Neil Armstrong and Buzz Aldrin planted the first human footprints on the moon’s dusty surface. The exploit seems even more heroic in retrospect, given how primitive and untested the technology was. Nasa’s entire suite of computers was less powerful than a single smartphone today.

The Soviet “firsts” in space had caused alarm in American political circles – and so galvanised a massive countereffort. John F Kennedy declared in 1961 that the US “should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the Earth”. No single space project, he said, would “be more impressive to mankind, or more important in the long-range exploration of space”. The US committed up to 4% of the US federal budget to Nasa’s Apollo project, the strategic prioritybeing to surpass the Soviets – and despite the rhetoric it was not primarily an idealistic goal, still less a scientific one.

In December 1968, to worldwide acclaim for Nasa, Apollo 8 successfully circled the moon 10 times before returning to Earth. During this mission William Anders snapped his famous “Earthrise” photo – iconic among environmentalists ever since – showing our fragile blue Earth contrasted with the sterile “moonscape” in the foreground.

The Apollo programme remains the heroic peak of manned spaceflight

The later Apollo missions succeeded, with the exception of Apollo 13 in April 1970, when astronaut Jack Swigert famously radioed mission control with the words, “Houston, we’ve had a problem here.” The shuttle’s power supply had failed, but thanks to repairs and heroic improvisation by Jim Lovell (portrayed in a memorable 1995 film by Tom Hanks) and his fellow astronauts, the mission concluded safely. Armstrong’s “small step for man” came just 12 years after the first successful Soviet space mission, Sputnik 1. Had the pace of advances been sustained, there could have been footprints on Mars by now – indeed that’s what many of us expected in the 1970s. But the Apollo programme ended in 1972 with Apollo 17’s safe return to Earth. It remains, a half-century later, the heroic peak of manned spaceflight.

In the subsequent decades, hundreds have ventured into space – but they have circled the Earth in low orbit, mainly in the International Space Station (ISS). Their voyages simply do not offer the same inspiration that the pioneering Soviet and American space exploits once did.

But the Apollo programme was motivated by the US strategic imperative to “beat the Russians”. Once that had been achieved, there was no longer any justification to continue the massive expenditure that it had consumed. In fact, it would have been naive to think that manned spaceflight would continue to surge forward as it did in the 1960s. All technologies advance in “spasms”, and sometimes stagnate if there’s no commercial or social demand. Of the 12 men who walked on the moon, only four are still living. Will there soon come a time when no human has firsthand memory of standing on another world? Many, myself included, would be saddened to see human exploration of space simply fade into history. The question, of course, is whether the manifestly exciting potentialities of space will include a practical and affordable role for humans. Machine learning is advancing fast, as are sensor technologies. And there is still a huge cost difference between manned and unmanned missions. The hiatus on manned spaceflight stems not only from the lack of a Kennedy-style political imperative, but also because the practical case weakens with each advance in robotics and miniaturisation.

There are nonetheless plans, from the US, Russia and China, to return to the moon and build a “base” there – but will there be sufficient motive and political will to send people, given what robots can do?

The main obstacles are political and cultural: the American public demands that such missions come with almost no risk. Nasa’s space shuttle was launched more than 130 times, with only two of its missions ending in disaster. But those episodes were national traumas, because the shuttle had unwisely been promoted as a safe vehicle for civilians, such as the schoolteacher Christa McAuliffe, who died in the 1986 Challenger explosion. Professional test pilots and adventurers would readily accept an implicit failure rate of 2% – or indeed, much more.

If I were an American, I would not support Nasa’s manned programme – I would argue that private companies should “front” all manned missions as cut-price, high-risk ventures. SpaceX, led by Elon Musk, and the rival effort Blue Origin, bankrolled by Jeff Bezos, will soon be offering orbital flights to paying customers. Indeed, SpaceX hopes to launch a Japanese billionaire, with a few friends, on a five-day trip around the far side of the moon. The phrase “space tourism” should be avoided by promoters of these private-sector schemes, because it creates a false complacency that such ventures are routine and safe, more like extreme sports or intrepid exploration.

‘Private companies should front all manned missions.’ The SpaceX Falcon Heavy rocket lifts off from Nasa’s Kennedy Space Center in Florida, April 2019. Photograph: Joe Raedle/Getty Images

The risks are daunting, but despite them there would still be many volunteers for a trip to Mars – some perhaps even accepting “one-way tickets” – driven by the same motives as early explorers and mountaineers. Indeed, it is time to eschew the mindset that space ventures should be national (or indeed international) projects – along with pretentious rhetoric where the word “we” is used to denote the whole of humanity. There are some endeavours – tackling the climate crisis, for instance – that can’t be done without concerted international action. The human exploitation of space need not be of this nature; it may need some public regulation, but private or corporate risk-takers can best provide the impetus.

And of course, space technology has burgeoned, despite the hiatus in manned flight. Modern society now depends heavily on orbiting satellites for satnav, communication, navigation, environmental monitoring, surveillance and weather forecasting.

Moreover, space probes have journeyed to all the planets of the solar system. Nasa’s New Horizons beamed back amazing pictures from Pluto, 20,000 times farther away than the moon. The European Space Agency’s Rosetta landed a robot on a comet. These spacecraft took five years to design and build and then nearly 10 years journeying to their remote targets. The Cassini probe spent 13 years studying Saturn and its moons and was even more venerable; almost 20 years elapsed between its launch and its final plunge into Saturn in late 2017. It is not hard to envisage how much more sophisticated today’s follow-ups to these missions could be.

In the coming decades, giant robotic fabricators will be able to construct solar-energy collectors, telescopes and other giant structures in space. Some aspiring space pioneers, like Bezos, envisage a longer range future in which much industrial production actually takes place off Earth. And the entire solar system – planets, moons and asteroids – will be explored by fleets of tiny automated probes, interacting with each other like a flock of birds.

But no feasible probes can reach beyond our solar system. Our knowledge of deep space – stars and galaxies – comes from telescopes. But space offers a huge boost to astronomers. Telescopes orbiting far above the blurring and absorptive effects of Earth’s atmosphere have beamed back sharp images from the remotest parts of the cosmos. They have surveyed the sky in infrared, UV, x-ray and gamma ray bands that don’t penetrate the atmosphere and therefore can’t be observed from the ground. They have probed with high precision the “afterglow of creation” – the microwaves pervading all space whose properties hold clues to the very beginning, when the entire observable cosmos was squeezed to microscopic size. Perhaps most fascinating of all, we’ve learned that most stars are orbited by retinues of planets, just as our sun is orbited by the familiar planets: there are millions of Earth-like planets orbiting other stars in the Milky Way. But do any of these harbour life – even intelligent life? To me, that’s the most fascinating question of all.

These “exoplanets” are too faint to be studied by existing telescopes. Giant robotic fabricators will be able to construct, in space, enormous lightweight telescopes to map the cosmos far beyond our own solar system. It’s realistic to hope that by 2068 – the centenary of the iconic Earthrise picture – such telescopes will have an image of another “Earth”, orbiting a distant star, that harbours life – yielding a still wider perspective on the role of life and intelligence in the cosmos.

• Martin Rees is the astronomer royal and a former president of the Royal Society