The privatisation of the railways had nothing to do with trying to make them better. It was driven by the ideological fervour of a few Tory MPs, before attracting the support of the privatisation unit in the Treasury led by Steve Robson (later Sir Steve), who loathed British Rail with a passion and later was a non-executive director of RBS during its worst excesses.

British Rail was almost the last in the long line of the series of privatisations initiated by the Tory governments of the 1980s and early 1990s. Margaret Thatcher had opposed the idea, and it was only the arrival of her replacement, John Major, which led to the concept being put in the party’s manifesto for the 1992 election.

Oddly, despite the plan being highly controversial, the issue was hardly discussed in the hustings, and the unexpected Tory victory resulted in ministers scrambling around for ideas on how to go about it. The usual justifications were given for the sale and break up of an inefficient British Rail, which famously sold stale sandwiches: the private sector would inject commercial flair; competition would be encouraged; taxpayers’ money would be saved thanks to greater efficiency; the government would no longer be seen as responsible when things go wrong; and, though this was unstated, the trade unions would be smashed.

Ah, the irony. Not one of these aims has been achieved. The first difficulty was that actually British Rail was a lean, mean machine that after 50 years of existence had, at last, found a good structure and operated with considerable commercial flair. Older readers will hark back wistfully to the InterCity ads (though they might prefer to forget those which featured Jimmy Savile). Moreover, the curly-edged sandwiches had long gone. Indeed, it was British Rail pioneered the idea of packaging them.

Secondly, the model that was chosen and developed on the hoof was fraught with difficulties, as the series of crises that has ensued demonstrates. Separating the infrastructure from the operations was a cataclysmic mistake, stimulated not by any desire to improve the lot of passengers but driven by the need to try to create a competitive environment. By handing over the tracks to Railtrack, the idea was that “open access” operators would then be able to compete with one another to provide services. However, that was seen as a bit too risky, since no one would take over the loss-making lines. Consequently, 25 franchises were created, with bids being based on the amount of subsidy the train companies would get for running services. That effectively squeezed out any “open access” operators, because of the risk that the best and most profitable services would be cherry-picked, leaving the franchisee requiring a greater subsidy. As a result, there have only ever been a handful of services run outside the cumbersome franchise system. In other words, the structure was contradictory from the outset.

Then there was the Railtrack debacle. The company that owned the track and stations was floated on the stock exchange, despite warnings that privatisation of an infrastructure provider was a daft concept. Its shares briefly soared for nearly five times their original price but then collapsed as a result of the Hatfield accident, which exposed Railtrack’s incompetence, and a project to modernise the West Coast Main Line that got out of control, necessitating a state bail out.

Far from making the system more efficient, the structure led to a vast number of legal interfaces that had to be policed at great expense by consultants, lawyers and accountants. The fragmentation of British Rail into more than 100 companies was the worst aspect of privatisation and only now is the government looking at ways of patching together operations and infrastructure. But given that the former is in private hands while the latter has been renationalised, this will be no easy task.

As for passengers, there is both good and bad news. Thanks to the way that the government has had to keep to contracts drawn up with the private sector, investment, nearly all funded by the taxpayer, has flowed into the industry. There has been a great increase in services, much new rolling stock and some fabulous station improvements. However, it has all come at a great cost, not least to passengers who now pay for two-thirds of the cost of running the system, whereas previously fares only contributed a third. The new system has gobbled up huge sums of taxpayers’ money that British Rail executives could only dream about. Not only has subsidy, at around £4bn every year, been around double (discounting inflation) what it was under British Rail, Network Rail has built up a debt of around £50bn – in other words £2bn per year – which clearly will have to be written off at some point in the future. It is, though, the trade unions, in particular Aslef, which represents train drivers, that have laughed all the way to the bank. Wages have soared, with £60,000 being pretty routine in many companies, because the drivers have been able to move between companies while retaining the threat that withdrawing their labour will paralyse the network.

Over a quarter of a century, the railways have got far better. However, a series of recent debacles have highlighted the high cost of improvements and the inefficiency of the fragmented system. These include the electrification programme on the Great Western, the ongoing industrial dispute and poor performance on the Govia Thameslink services and continued above inflation rises. The Labour Party is on strong ground when pushing for the renationalisation of the industry. But, when in power, it will find it hard to row back on some of the structural faults in the system, such as the high cost of projects, the incomprehensible fares system and the lack of any guiding mind to determine long-term strategy. It is not just a matter of bringing the services into public ownership. The railways need a coherent structure with a strong organisation at its core but with the ability to respond to local demands. Given that parts of the system such as freight services and the ownership of the rolling stock cannot easily or cheaply be brought back into the public realm, creating a new structure will require a delicate balancing act.

Christian Wolmar is that author of a dozen books on transport including On the Wrong Line: how ideology and incompetence wrecked Britain’s railways.