I confess that I never expected, nor wanted, to share membership of an exclusive club of two with the transport secretary, Chris Grayling. And yet, as of yesterday’s extraordinary U-turn on what to do about the failing and over-extended east coast mainline, Grayling and I are the only living British politicians to have nationalised a railway.

In fact, we both nationalised the same railway. In 2009 I took the east coast mainline – which runs between London and Edinburgh and is a vital artery for the UK economy – back into public ownership. National Express, the private company that had been running east coast, wanted to hike prices and at the same time wriggle out of its obligations and debts to the government. It was a poor deal for fare-payers and taxpayers alike. So I took the railway away from a private company that was failing and created a new, public company to run it instead.

I was warned by the private sector that we would fail and would fail badly. “The state can’t run a railway,” I was told. Well, they were wrong. Ticket sales went up. Over £200m was raised for the taxpayer. Customer satisfaction reached record levels. And yet, in 2013, the coalition government decided not only to put the franchise back up for tender but to ban the successful, efficient and popular public company that had been running the service from even bidding to carry on.

Why? Because ideology still plays a peculiar and outsized role in policy-making when it comes to our railways. On the left, nationalisation is pursued for its own sake – as a panacea that will somehow fix train travel all by itself. On the right, private management is viewed similarly. No matter the evidence or the experience, the Conservative party has been allergic to direct state involvement in running our railways.

That fixation on the private sector has led to some very peculiar, almost comical mismanagement. Grayling has spent much of the last year trying desperately to avoid taking the very course of action that he announced yesterday. He bailed out Brian Souter and Richard Branson (the current franchisees) to the tune of £2bn. For a man notorious for his opposition to welfarism, this was extraordinary generosity at the taxpayer’s expense. It was partly in opposition to this ridiculous and dogmatic act of self-harm – characteristic of this Brexit-obsessed government – that I resigned as chair of the National Infrastructure Commission. It was a terrible deal done exclusively for political reasons.

Thankfully – under considerable public pressure, and thanks to a dogged campaign – the government has backed down. Those of us involved in, and passionate about, railways should give thanks that Grayling’s obsession with privatisation at all costs has been stopped, this time, in its tracks. Colleagues working under him in the justice system were not so lucky; they are still dealing with the consequences. But the fight is not over yet.

Embarrassed, perhaps, by his climbdown Grayling is still refusing to wholly learn from the mistakes of the past. Instead he wishes to bind his successors’ hands and force them to repeat them. He has insisted that the new public management of the east coast will be temporary and that it will be passed back into private management as soon as possible. This is a mistake.

Instead, the government needs to give its new creation the space to thrive. And then, when the time comes, that public company should be able to bid – in an open competition – to carry on. If a private company is able to beat the operator on value, on customer satisfaction, on investment, then so be it. If not, why throw away what has been achieved for the sake of proving a political point?

Britain’s railways don’t need to be wholly nationalised and they don’t need to be operated solely by private companies. Both of these myopias harm our ability to get the best deal for passengers. A mixed market – where public companies like the one Chris Grayling has just (reluctantly) created compete with the private sector – can deliver the best deal for fare-payers and taxpayers alike. That might sound dull, but in truth we don’t need or want railways to be political totems. We just want them to work. Apologists for extremists of left and right like to say that at least they made the trains run on time. I’m afraid that’s not true. It tends to be centrist dads like me that get the railways working.

• Andrew Adonis is a Labour peer, former transport secretary and former chairman of the National Infrastructure Commission