The bright, electric future of the railway has been dimmed: trains will now continue to run on diesel west of Cardiff to Swansea, as they will on the Midland mainline to Nottingham and Sheffield.

An announcement sneaked out on the last day before parliament went into summer recess has sparked howls of betrayal, in Wales in particular. Rail electrification, the government had once boasted, would bring quicker, greener, more reliable services, jumpstarting the economy and creating jobs with its high-voltage wires. Now it is scrapped, and ministers say at least people will not have to put up with unsightly masts.

If it is bad news for communities whose railways will never reach the improved level of service promised, it also marks a moment where the dial is turning down of investment in the UK’s rail network.

Escaping much scrutiny, the transport secretary, Chris Grayling, also last week issued 2017’s version of a document that in 2012 heralded what the Conservatives vaunted as the “biggest investment in rail since Victorian times”: a draft of Network Rail’s £38bn five-year plan and the basis for a 2015 manifesto listing a string of transport goodies for the regions – not least the Northern Powerhouse.

In that plan, electrification of 850 miles of railway was at the heart of £13bn for new infrastructure. Now, Grayling’s first “HLOS”, or high-level output statement, says tersely it “does not commit to infrastructure enhancements”. Observers are worried. Lilian Greenwood MP, Labour chair of the transport select committee, says failure to provide specific details is “deeply concerning”.

How did rail get here? At the heart of the slow-burning debacle is the ballooning cost of the electrification of the great western line, whose initial budget of £895m had reached £2.8bn before the scheme was trimmed at the edges around Bath, and now truncated at Cardiff.

The overspending was admitted in 2015, when Network Rail’s chairman was ousted and its unworkable five-year plan reviewed. “Pauses” to work became tacit admissions of vast budget holes, whose full extent is yet to become clear as projects are pushed into future, notional plans. Emergency funds have been needed after the reclassification of Network Rail meant its easy borrowing, £41bn of debt which had long disguised real costs, was ended.

Little wonder that Grayling is wary, saying: “Network Rail’s progress on improving its efficiency in recent years has fallen short of my expectations.”

But neither does Network Rail want a similar mandate again. Chief executive Mark Carne told the Observer bluntly: “I don’t want a budget for enhancements. I don’t want to go out with promised costs and expectations that can’t be met. That damages the reputation of the industry and we’re not going to do it again.”

Instead, individual projects will be costed to a high level of detail before funds are requested. A menu of TransPennine route upgrades, another fallen victim of the five-year plan, will be considered again in 2018 – but full, costly electrification is unlikely to be delivered. Necessity is the mother of spin doctors’ invention, and electrification is no longer painted as a panacea “disruptive and unsightly” in government announcements.

Despite the rumblings on the tracks, the industry’s gravy train will roll for some time yet. The man who signed off the 2014-19 plan at Network Rail, David Higgins, chair of HS2, has faced down scepticism over the project’s own £55.7bn budget – underlined by revelations last week that it spent millions in unauthorised redundancy payments to highly paid staff.

Yet with the full high-speed route now announced and £6.6bn of contracts awarded, HS2 at least looks guaranteed – even if the current era of multiple grands projets, such as Crossrail and the Thameslink programme through London, is unlikely to be replicated soon.

Transport secretary Chris Grayling says Network’s Rail’s efficiency has ‘fallen short of expectations’. Photograph: Joe Giddens/PA

And passengers will see newer trains sooner – a straw clutched by the Department for Transport last week. Instead of parking the Hitachi electric trains, more will be converted to bi-mode, to run on diesel down unelectrified tracks in South Wales and the Midlands.

More British-built trains are coming, in what has become a quietly successful turnaround. A swath of train orders for London Overground and the South Western and East Anglia franchises are underpinning thousands of highly skilled jobs at Bombardier. While trains have been built in Derby since 1840, the plant feared for its future in 2011 when big contracts such as Thameslink went to competitors abroad, soon after Hitachi had won the £5.7bn contract for the bi-modes Wales now needs.

While Hitachi’s assembly plant, opened in 2015 in County Durham, was a boost to the north-east, Derby’s renaissance means more skilled work is done in the UK rail industry. Matt Colclough, general manager of the Bombardier plant, highlights the 400 people employed just in creating the electrical train components. “This is unique to Derby. A good 50% is the electrics – it’s the nerve centre of a train, the brain really.”

While he won’t name competitors, Hitachi ships in the bodies from Japan, where much of the “high-value” design and manufacture is done. The Derby plant houses hundreds of engineers, designing and testing Bombardier’s new generation of Aventra trains.

Colclough says it has upped its game to win contracts: “We’re spending a lot more time in the design phase, making sure we’re doing things in the most efficient and productive way.”

Building trains in an “ergonomically correct” fashion, he says, means completing and testing the carriage’s constituent parts, then assembling them, rather than wiring them up afterwards – and also takes the risk away from a production line which boasts a rate of 25 carriages per week.

A £20m investment in recent years includes a test hall complete with a 25,000-volt power supply running down to the train’s pantograph, and a “Train Zero” room. Here, an Aventra train exists as a control console and all of its electronics, but with no body, allowing its systems to be tested as if it were hurtling along at 125mph.

Derby’s investment was secured in the wake of winning the Crossrail trains contract; a victory enabled, in the wake of public pressure, by the government easing its financial path.

That episode underlined the political importance of the rail industry, secured again by HS2 and train building. But questions over whether the billions invested in the industry represent value for money should make ministers hesitate before opening the Treasury vaults again.