On a clear day, a Frenchman with a pair of binoculars could get a decent glimpse of the challenges facing Britain’s railway network by peering across the Channel at Dover. Until last Christmas, a high-speed train would travel at half pace along the old line at the bottom of the cliffs, giving commuters a direct fast link to Folkestone and the capital via HS1’s tracks. Then a combination of storm and tide so damaged the sea wall that passengers from Dover and beyond now have to transfer miserably on buses as engineers work to repair the damage.

In this part of the south-east, more and more people are travelling by train, and paying a high price for it: a season ticket costs more than £6,000 a year. But the capacity of the network is being sorely tested and, despite huge investment and the development of high-speed track as far as nearby Ashford – still rests on creaking infrastructure in many places. Network Rail chairman Peter Hendy, charged with reviewing in detail what its engineers can achieve for its £38.5bn budget, said at a recent transport conference: “We’re discovering that some of it is pretty badly built. Seen Dover?”

One man who has seen plenty of Dover of late is Steve Kilby, the Network Rail manager who must restore the line after cracks appeared in the sea wall. “I got the call at 12.15pm on Christmas Eve. I was on annual leave – but that was the end of that.”

The cracks turned out to be the visible manifestation of deeper problems. The beach beneath it had shifted dramatically, exposing the sea wall’s foundations to battering from the waves. The track was sitting not on solid ground but on a viaduct, buried in chalk, that was being eroded from above and below by the sea. The line was closed and the wall quickly shored up with 200 lorry-loads of rock armour – huge boulders on the beach.

Repairs on the damaged line near Dover. Photograph: Martin Godwin/The Observer

Commuters became frustrated by the subsequent apparent lack of action. Kilby admits that at first they had little idea of the best way to rebuild the route. Records of the construction of the sea wall in the 1920s were sparse, and there was nothing about the newly discovered mid 19th century viaduct. In the end, engineers decided the Victorians had built it for a reason.

“So it came back to a viaduct,” says Kilby. “But if I had a 250 metre-long bridge to do anywhere else in the UK, it would take a year to design and a year to build. We’re committed to nine months.”

Concrete piles 30 metres deep are being cast in the ground, while 90,000 tonnes of rock is being brought on a barge from Norway to reshape this wave-battered stretch of shore, called Shakespeare beach, and build a solid new structure. But a changing climate means the future looks ever more unpredictable. “Around the country, we’ve been having these events that normally occur once in a 100 years, 250 years, even 1,000 years,” says Kilby. In 2014, he worked at Botley in Hampshire, where a landslide blocked the line between London and the south coast, and 80 metres of embankment was washed away by torrential rain.

Ageing infrastructure and the need for more capacity are the railway industry’s two big challenges. Network Rail chief executive Mark Carne says the best way to meet burgeoning demand is what he calls the “digital railway”: bringing forward the replacement of signalling by decades so that more trains can run on the existing network. Carne told the transport select committee earlier this month that it would make the best of the tracks Britain already had: “The network will be more reliable and more resilient. It’s going to cost a great deal of money – but less than the alternative.”

Concrete piles are being sunk at Shakespeare beach to shore up the Dover-Folkestone stretch of rail. Photograph: Martin Godwin/The Observer

Carne says the current system is safe, but inefficient, with signalling one of the most fragile parts of the infrastructure. A detailed business case is being drawn up but the digital railway idea has influential supporters in the Department for Transport as discussions over the next five-year plan begin.

Network Rail hopes that, with transport secretary Patrick McLoughlin tipped to be reshuffled out of the job after the EU referendum, a new minister will champion the project. It would have legacy potential, while avoiding the public antipathy generated by a third airport runway or HS2.

But Carne is not without opponents: some in the industry say he has overstated the benefits of the digital railway while appearing to disparage its current operation. Roger Ford of industry magazine Modern Railways says: “We’ve had a digital railway since 1985. Mark Carne’s ill-informed comments about us being in the dark ages are quite ridiculous.”

He added: “The capacity claims made for Carne’s plan [up to 40% more trains] are extreme. It is being promoted as the universal panacea, an alternative to building new infrastructure. But if you build infrastructure, you’re adding to the network, not just heating it up. Would you rather spend money on a fifth track into Waterloo – which will be there for ever – or squeeze a quart into a pint pot?”

Carne appeared to have taken note of such criticism when he appeared before the select committee: he was careful to state that the digital railway could only deliver certain improvements, and could not replace other work.

Network Rail chief executive Mark Carne says hi-tech signalling will solve capacity problems.

Another expert, Jim Steer, director of rail research group Greengauge 21, said no one disputed that upgrading the UK’s signalling to European Train Control System standards was a move in the right direction and could boost capacity. But he added: “If it was available everywhere, great, but it’s ambitious in terms of budget. The previous plan was to time its adoption with when renewal [of signalling systems] fell due and you were going to have to spend money anyway. I’m not sure it can be done by 2028, and where is the money going to come from?”

Although Network Rail’s stock with the government has risen considerably since the early 2015, when cost overruns led to the cancellation of major projects, few expect to see large sums being made available in the near future, even if Carne insists it is money well spent.

While the industry debates the technological future and ways to cram ever more passengers on to the network, the past may still come back to bite it. Steer warns: “These railways were built before people understood the basics of soil mechanics – and with climate change that has become a real issue.”

They know that too well in Dover, where commuters hope by Christmas to have once again their link to resilient, high-speed tracks – the modern railway that appears tantalisingly close but is for now out of reach.