Who can resist this hinge in time, when one year ends and another begins, without indulging in a bit of light speculation about the future, if only to contemplate where, if anywhere, on Earth might be a safe space at the end of March.

This interregnum between Christmas and new year is also when Whitehall’s records of old arguments and negotiations and decisions are released, at least partially. It’s a moment usually treated as an exercise in history, but really it’s a snapshot of the way people anticipate the future. The latest releases, published at the end of last week, are particularly interesting because they reflect the way Whitehall copes at a moment of extraordinary upheaval. They cover the early 1990s, the years after the Berlin Wall was breached. The Soviet Union was tottering. Nelson Mandela was newly freed, and a technological revolution was waiting to be unleashed. Yep, a new world was waiting to be born.

So on the one hand there’s what to do about this extraordinary and unfathomable thing, the internet, and the “email will never catch on” view of Whitehall officials in 1994 (a view widely shared even by people who claimed to understand it, if only because its implications were so alarming). “One issue is whether we should advertise that it is possible to send messages to the prime minister, and – presumably – get a reply,” wrote John Major’s principal private secretary, Alex Allan. “I am cautious about rushing into it.”

Thatcher dismissive of Mandela after first phone chat, files reveal Read more

On the other, in an exchange from 1990, Mandela is seeking to persuade Margaret Thatcher that although she had previously considered him a terrorist, he could in fact be South Africa’s Mikhail Gorbachev, someone she could do business with. Britain’s ambassador to South Africa, Sir Robin Renwick, was a big fan. “Mandela has a natural dignity and authority. He is not as intelligent as Mugabe but a great deal nicer.”

The two judgments mirror the classic futurologist insight, that the way you see the world now shapes the future that you imagine. Whitehall does power and bureaucracy: no surprise that it was better at thinking about Mandela and his part in the future of South Africa than it was at anticipating the disruptive power of digital communication.

But it also illustrates another distinction, between planning for the future, which involves thinking about a world ahead that follows broadly along established trajectories, and imagining how the future could be. If you think of it as Tony Blair v Jeremy Corbyn, it is clear that there are limitations to both approaches.

Technology in particular has a way of defeating the most ambitious agenda. It is impossible, say, to stand at the foot of the astonishing staircase of locks at Devizes on the Kennet and Avon canal without feeling some sympathy for the pioneers who invested more than £1m to build a southern inland water route linking the river Avon to the Thames – only to be driven out of business by the Great Western railway before they had got their money back. Or, even shorter lived, America’s transcontinental Pony Express, which lasted just 18 months before the advent of the telegraph demolished its business model.

The thing about tech is that even if you can see it coming, you can’t be sure quite how it will arrive or what it will do when it gets here. The goldmine of NHS data could transform the way health and illness are understood, but finding a way to do it that is both technically possible and morally acceptable has already cost billions and remains a distant dream. We run old political models such as representative democracy, slow and hierarchical, against the flat and the instant of digital communication and struggle to manage the consequences.

It takes an unusual combination of imagination and dogged persistence to make real change. My Christmas reading is Wilding by Isabella Tree. She and her husband have taken almost 20 years even to begin to show how their vision of a managed return to the wild for their West Sussex farm may offer a solution to the global collapse of biodiversity and a future for more marginal agricultural land. It takes a bold reimagining of the countryside to flog the farm machinery. And a great deal of taxpayers’ money.

But for the ultimate demonstration of the limitations of predicting the future when imbued with a powerful sense of the failings of the present, visit the Anglo-Saxon exhibition at the British Library. In 1014, Wulfstan, the archbishop of York, gave a downbeat view of the year ahead: “Beloved men, realise what is true: this world is in haste and the end approaches; and therefore in the world things go from bad to worse.”

Sixty years later, according to the Anglo-Saxon Chronicle, nothing had improved. “And always after that [the Norman Conquest,] it grew much worse.” Well, the Anglo-Saxons were out of power. But at least the world hadn’t ended. In fact, the exhibition suggests, the world that was being created and, in the Domesday Book, catalogued, is recognisably the basis of the world today. So what, when viewed from the next millennium, will survive of 2019?

• Anne Perkins is a former Guardian correspondent