The consultants charged with finding a way to modernise the crumbling west coast railway thought they'd hit on a magic bullet: a white-hot technology that would allow the huge task to be completed for a song. There was just one problem: it had never worked before. Ten years to the day since Railtrack took over our railways - and after a yearlong investigation - James Meek reveals the saga of incompetence, greed and delusion behind Britain's biggest public works project

On a mild, wet February morning, a work gang of 10 men moves around a stretch of railway line near the village of Goostrey, between Crewe and Manchester airport. Against the extreme green of fields and the whipped grey of rainclouds their synthetic orange work suits shine out violently, like figures in a psychedelic episode. It's hard work. Using thick, weathered, yard-long metal-capped staves, they crank jacks which raise a stretch of rail, along with its concrete sleepers, up off the bed of stone chips the railway rests on. A bulldozer on rail wheels purrs up on the other line and begins pawing at the stones. The men are to pack fresh ballast in under the rails with hand shovels.

These railway workers are engaged in the single most expensive non-military task ever undertaken by Britain alone: the modernisation of the west coast main line. A project that was supposed to cost roughly £1.5bn will, by the time it is finished in about 2008, two years late, have consumed almost £10bn, much of it from the taxpayer. That is £3bn more than the White House considers Nasa will need to send men back to the moon.

The bizarre story of those multiplying billions, reconstructed here from dozens of interviews and documents, some never before made public, ran parallel to the bloodier history of fatal accidents on the railway since privatisation, and contributed, in equal or greater measure, to the ignominious end of the Railtrack era. It is a tale of incompetence, greed and delusion, driven by the conviction that profit and share value is the only true measure of success; and that the ability to chair a meeting or read a balance sheet is always worth more than the ability to understand how machines and materials will best serve human needs.

It is a story, too, with wider implications about the kind of country that Britain has become: a country that has lost faith in its abil ity to design, make and build useful things; a country where the few who do still have that ability are underpaid, unrecognised, and unadmired.

Politics and consultants

Watching over the Goostrey work gang, dressed in cleaner, newer versions of the same orange suits, are three of their bosses, all American, from the US construction firm Bechtel, brought in two years ago to rescue the project from the incompetence of the collapsed rail company Railtrack. No British company seems now to have the skills required; nor, after Railtrack's systematic efforts to gut itself of in-house specialists, does Railtrack's quangoid successor, Network Rail.

As an extreme measure to try to bring the final bill down, the rail industry agreed to shut the entire stretch of railway we are standing on, from Crewe to Cheadle Hulme, for four months, forcing thousands of travellers on First North Western railways to use coaches while 600 orange-suited men and women work on the line 24 hours a day, seven days a week. Without such radical steps, says Tom McCarthy, the Californian who leads the Bechtel west coast team, the project could have ended up costing £13bn.

It could have cost much, much more. The project now being built is a stripped-down version of a fantasy railway Railtrack locked itself into a brutal contract to build with Richard Branson's Virgin Trains in 1997. I sat one day with Stuart Baker, the man brought in by the Strategic Rail Authority to survey the wreckage of Railtrack's plans for the west coast in 2002, and read out to him the sequence of growing price tags: £1.5bn in 1996, £5.8bn in 1999, £7.5bn in 2001...

"None of those numbers delivers the contract," he said. "You could talk about numbers six or eight times that. It is into 30 to 40 billion if you were actually to create the infrastructure necessary to fulfil [Railtrack's original] obligations."

At Goostrey, I try to encourage the Americans to talk about why the project turned out to be so much more expensive than its creators thought. Bob Brady, a Texan from Houston who is managing the reconstruction of this part of the route, begins to answer. "I think part of the issue was the politics when it was privatised -" McCarthy interrupts him. "Don't go there," he says.

Brady is right, which is why McCarthy doesn't want him to go there. The problem with the west coast project is not that it is going to cost £10bn, but that the original bosses of Railtrack thought it could be modernised at a fraction of the price, and sold this ideology-driven delusion to the public and the City.

Friday December 23 1994 dawned foggy and almost freezing in central London. The papers reported on Yeltsin bombing Chechnya, on new developments in the Guinness scandal, and on fresh Labour attacks against the millions of pounds made by directors of recently privatised utilities. On the fifth floor of an office building in Hanover Square, London, a group of 15 consultants sat in a room carrying out a strange task. They were writing numbers by hand, in red ink, in the top right-hand corner of a secret 182-page report.

The red numbers were a device to stop the report being leaked, as so many documents were in the course of rail privatisation, the most controversial of all the Conservatives' state sell-offs. That spring, Railtrack, a new organisation, had taken over British Rail's responsibility for running and maintaining tracks, bridges, signals and tunnels. It was due to be floated on the stock market. It was essential that it now convince the public and the City that it could finance and organise the modernisation of the west coast line - something BR had never been given the money to do - and the report was going to tell Railtrack how it should do it.

Five years later, when the consequences of the report's recommendations became clear but the report itself had been forgotten by the media, a subsequent generation of Railtrack bosses tried to blame it all on BR. In fact, BR had nothing to do with it. Railtrack was determined to prove that it didn't need BR's expertise, so it commissioned a consortium of consultancy firms to do the job.

That Friday, two days before Christmas, the consultants were in festive mood. They had been working on the report since March and as soon as they had finished with the red numbers, off it would go to the government and Railtrack, the job would be done, and the holidays would begin. "We had to get it finished by noon, and we finished by 11," one of the consultants recalled. "I think we all went down to the pub afterwards - the kind of jolly, end-of-project thing that happens."

On the face of it, the consultants had every reason to celebrate. They had approached a gigantic problem, and come up with an elegant solution.

150 years of decline

For a culture that has been under constant attack from the public and the media for almost two centuries, the rail industry has a surprising love of misleading jargon. The west coast main line, usually known in the business as the WCML, is a case in point. It doesn't run along the west coast, and it isn't a line. It's a 690-mile network of routes between London and Glasgow, connecting them to Liverpool, Manchester, Birmingham and scores of other large towns in the West Midlands and north-west of England.

It is almost the oldest inter-city railway in the world. It was built higgledy-piggledy over three decades, starting in 1833, by entrepreneurs and hard-drinking, red-waistcoated navigators who, if they died on the job, were sometimes buried where they fell. Its narrow tunnels, lines squeezed together, tight curves and eccentric kinks reflect not just the geography of the land but the speed of contemporary trains - 40mph max - and the reluctance of powerful landowners to have clanking, smoky iron horses hauling the proletariat across their estates. Robert Stephenson, son of George, who built the London-to-Birmingham stretch, had to make some of his measurements secretly, by night, to avoid being run off the squireocracy's property.

Between 1833 and 1994 the line was modernised, of course, but never rebuilt. Anyone who has lived in a 19th-century house will be familiar with the problem. The arcane wiring when electricity came along, the subsequent clumsy rewiring; the cheap flat conversion in the 1960s; the constant saga of patch and mend from occupants who never have the money or vision to remake the whole thing from scratch - all this, and more, was paralleled on the WCML on an enormous scale.

When the consultants came in, the WCML had been starved of investment by governments for some 20 years. The only significant reconstruction was in the 1960s, when the railway was modified to take electric trains. Repairs and the skills of Georgian and Victorian engineers enabled train speeds to increase until the mid-1980s. Then the network began to deteriorate quickly.

The greatest problem was the signalling. It had to be replaced, but to remake that system, with its thousands of miles of cables, coloured lights on poles and elderly signal boxes, would be staggeringly expensive.

A 1992 document, the Hesketh report, classified at the time but later slipped without publicity into the House of Commons library, recounts in bald language the horrific state of WCML signal boxes, on which the safety of passengers depend. They read like despatches from a war zone. Stockport signal box 1 was "installed 1896. Roof of relay room lets in rain. Cable route heavily damaged ... very few spares. Signal structures have severe corrosion. Power supplies are suspect." Brewery Sidings: "Installed 1894 ... severe structural problems ... incapable of modification ... signal structures have severe corrosion and access by staff is by special arrangement." Miles Platting: "Installed 1890 ... box and relay room have serious structural damage with propping by the civil engineer to prevent collapse."

How could Railtrack do it? How could they do what BR had not done and rebuild this tottering railway? The consultants came up with a remarkable scheme which would, besides modernising the WCML without costing the Treasury a penny, enable trains to whizz between London and Glasgow at the unprecedented - for Britain - speed of 140mph, and make the west coast line the envy of the railway world.

The miracle solution

A new idea was being discussed in rail circles in the 1990s. It was called "moving block", and it was supposed to do away with conventional signals for ever. It was based on the technology used for mobile phones. Normally, trains run on a fixed-block system. A line is divided into stretches called blocks, with signals controlling the entry and exit to each block. If a block has a train in it, the signals prevent another train entering that block and crashing into it.

Moving block abandons conventional signals in favour of computers, track-mounted radio beacons and a cellular radio network. With these, train drivers always know where they are in relation to other trains. They still have a protective block of space around them, but it moves along with the train, and shrinks or grows according to how fast it and the trains in front and behind are going.

Once Railtrack's consultants fed moving block into the equation, the miraculous happened. The numbers made sense. They wouldn't have to remake the signals; they would simply demolish them, and replace them with a few mobile-phone masts and black boxes in existing train cabs, which would be far cheaper to install and maintain. Thanks to moving block, they would be able to squeeze more trains on to the line. The trains would be able to go faster, which would not only justify charging passengers higher fares, but would mean the train operators could run more services with fewer trains. As the final cherry on the cake, the cost of upgrading the route to take 140mph expresses could be defrayed from the extra profits the express operators would make. There was only one problem with moving block, but it was a crucial one: moving block for main-line railways did not exist.

Even now, almost 10 years later, there is not a single main-line railway anywhere in the world, no matter how sophisticated, which uses moving block. It is used only on a few specially built urban transit systems, such as the Docklands Light Railway, a single metro line in Paris, and a new line in Singapore. For full-scale railways it remains where it was, on the drawing board. The consultants did not allow this detail to stop them, and nor did Railtrack.

"When I heard about it from outside, I thought: 'Wow, they must have had some amazing breakthrough which means this is now a proven bit of kit.' And it wasn't," said Chris Green, now head of Virgin Trains. "It was a wish list. To put that wish list on Europe's third busiest railway really was outrageous."

A wing and a prayer

Of the four teams of consultants that prepared the feasibility study, Booz Allen & Hamilton was responsible for coming up with the recommended signalling system. No one from the company would comment for this article, but one of the consultants involved in 1994 did talk. Of the eventual core team of eight consultants, only two had experience of British main-line railways, one at a very junior level. Four of them were American or normally based in the US. Not one had expertise in moving-block signalling on railways like the WCML; most of the experimental work on moving block was being done in mainland Europe.

The consultant asked for his real name not to be used. I've called him Arthur. He said of some of his colleagues: "My personal view was... they hadn't got the knowledge to try and do this in any sensible way. Coming from an intensely professional railway background, I was appalled by this. You can imagine. Not knowing what other railways of the world were up to, the consultants did a write-round. "We did a trawl," said Arthur. "We wrote to manufacturers and asked them for information ... We asked the Japanese what they'd got [in terms of moving block] and they said: 'Well, nothing, really.' Which amazed us." Indeed, this would, a lay person would assume, have given the consultants pause for thought. Neither the Japanese, with their 160mph bullet trains, nor the French, with their 160mph TGVs, used moving block in 1994: they still don't.

Yet despite this lack of knowledge, the fact that no railway in the world was using the system, and the fact that the equipment needed was not being made by any manufacturer, the Booz Allen team ended up recommending moving block. Why? Arthur did not say so specifically, but hinted that the remit was not to find the most practical solution and find out how much it would cost, but to find the solution that would make privatisation financially possible. The cost of conventional signalling, had they known it, would have frightened investors off and halted privatisation in its tracks.

"The thing that everybody always theorises is, why did you recommend such a highfalutin-type system for the WCML? And that's a fair question," said Arthur. "But it's not a fair question when you actually look at what the remit was. The remit was to come up with the most cost-effective solution."

The Booz Allen consultants came to another conclusion which, in retrospect, seems extraordinary. Even though they knew what the privatised railways would look like, with Railtrack owning the infrastructure, a horde of private train operators running services, and dozens of other private outfits all taking a cut and all trying not to step on each other's contracts, they based their estimates of how much the WCML modernisation would cost on British Rail history - the bargain-basement modernisation of the east coast main line, and the development of a new type of conventional signalling 20 years earlier. It was as if a western oil company based its cost forecasts in free-market Russia on the way things were done in the Soviet Union.

"What I hadn't understood," admitted Arthur, "was that the restructuring of the railway was going to bring a complexity beyond my wildest dreams." Nor could the consultants have anticipated that the bosses of Railtrack would go on to cherrypick their conclusions.

The bosses

Railtrack was led to privatisation by two men, its chief executive, John Edmonds, and its chairman, Robert Horton. Edmonds was a former senior British Rail executive, Horton the former chairman of BP. Yet it was Edmonds, the former public-sector boss, who was the private-sector firebrand. Far from being loyal to BR's way of doing things, his experience on the state railways had inspired in him a scepticism towards in-house engineers and safety experts bordering on contempt. They were, he considered, overcautious, conservative, stuck in the mud. It was this which led him, at Railtrack, to shed the nucleus of in-house expertise that left the company unable to understand what its myriad specialist contractors were up to.

Edmonds declined to be quoted for this article. A senior rail manager said of him: "He was the one who got rid of operations managers and engineers because he didn't believe in them. He thought it could all be contracted out and commercialised.

"He had a desire to break the mould and change. He was always opposed to the traditional railway. He believed there was a golden panacea in the private world where you just free people up and new technology comes in and the markets come in and it all happens. The railway doesn't work like that. You're not manufacturing baked beans."

Like Edmonds, Horton was an advocate of privatising Railtrack. But although Horton was happy to play the role of swashbuckling private-enterprise shaker-upper, he was no entrepreneur. He studied engineering in the 1950s but he was no engineering wizard, either. He was, essentially, a highly paid private bureaucrat who made his reputation in the offices of a post-imperial semi-state oil company. He did not make BP; BP made him. It was already a vast, sprawling corporation when he arrived in 1957, and it was still a vast, sprawling corporation when he was sacked in 1992, no matter how much he changed it. He did, at least, know about the oil business. But he didn't know much about trains.

"He wasn't close enough to the railway to know what was going wrong," said one rail industry source. "So he was great at privatising, great with the City, good at getting private investment into industry. He didn't understand that he'd lost all his key operators, lost all his key engineers, and was chasing technology that wouldn't work." Horton also declined to be interviewed for this article. In a brief email, he said: "I think it is important to understand that the scope of the project changed enormously over time as did the decisions on the technology to be used."

In 1994, Edmonds, Horton and the latest in a blurry succession of Conservative transport secretaries, Brian Mawhinney, all wanted to hear from the consultants that the WCML could be modernised with privately raised money - by a private railway company. Yet there was still an opportunity for someone to persuade them that the risk they were taking was unacceptably high.

Warnings missed

The key expert standing between Railtrack and the fatal decision to go for moving block at this time was Rod Muttram, the firm's new director of electrical engineering. But he knew little about the railways either: he had just been headhunted from the arms industry, where he had been involved in developing weapons systems, including a new type of artillery rocket. He believed that moving block could work, in theory. As to whether it could work in practice, on such an incredibly complex rail network, he was entirely dependent on what the consultants told him. Muttram, declined to make any public statement for this article.

Yet British Rail still existed in 1994 and 1995, and its board thought the consultants' ideas were far-fetched. John Welsby, then chairman of BR, told me: "I did have grave concerns about the attempt to integrate a type of technology that had no testing in real, active life in what, in fact, was the most complex railway in the country."

Welsby was not exaggerating his wisdom after the event. A senior Railtrack figure, one of the key men liaising between Railtrack and the consultants, recalled: "There were huge rows at the time with the British Rail board, who were completely unsupportive of the project. John Welsby, then chairman of BR, was the most vociferous opponent. He said [moving block] could never be done on a railway like this."

Welsby and his BR colleagues were unable to get the message through to their private-sector successors. "You have to remember that we were all, at that stage, unbelievably busy," he said. "All my lads had about three jobs. Firstly, they were running a railway. Secondly, they were breaking it up for sale. Thirdly, they were often preparing a management buyout at the same time. People were working seven-day weeks, 12-hour days. Our advice was passed across but then it was up to Railtrack to take that advice or not, as they wished ... of course they would have resented it anyway. The climate has to be remembered: we were big, bad BR, being broken up and done away with, because we were anathema to the government, so the new order was not necessarily going to look very favourably on us."

Another warning came almost as soon as the consultants handed in their report. It came from Europe. The consultants' report - a copy of which was recently released to the Guardian under the "open government" code - makes little mention of the European dimension. None of the consultants was from mainland Europe. But it was clear from speaking to "Arthur" that he had assumed other European railways were preparing to introduce the new technology as well, and that Britain, relying heavily on European research, would merely be the first to apply it.

Then, in January 1995, most of Europe's state-owned railways - 19 of them - came to the joint conclusion that moving block was not ready to be used in the real world, and a simpler, transitional form of new technology should be the next step. Again, Railtrack ignored the warning. In fact, Railtrack may never have heard it: at this time the firm had barely any contact with Europe. It was an extraordinary situation, of which the public was ignorant: a group of Anglo-American consultants and executives took for granted European support to develop a technology which those same Europeans openly declared to be premature.

A tale of two reports

On March 22 1995, Railtrack and the government went public with their plans for the west coast main line. Despite the warnings, they endorsed, with few reservations, the consultants' recommendations to make moving block central to the modernisation. John Watts, the Tory rail minister, spoke of an "innovative signalling and control system" which would be "at the heart of the proposals" for the WCML. (Interviewed by phone recently, Watts said he could not remember details of this period.)

The consultants' report remained secret. On March 22 Railtrack released what it said was a summary of the report. By and large, it was. But having had a chance, for the first time, to compare the two, the Guardian can reveal that there were important changes and omissions in what the public was told compared with what the consultants had said.

The consultants warned, for instance, that if moving block turned out not to work, and conventional signals had to be used, any attempt to try to get trains to run faster than 125mph - always Railtrack's intention - would incur "exceptionally high costs". This fateful warning, one of the keys to Railtrack's eventual demise, was omitted from the public summary.

At the beginning of March, signalling-company bosses had told MPs that they might - might - be able to have moving block working and installed within 10 years. In their secret report, the consultants talked, with optimism, of "a five-year programme" for the development and fitting of moving block. Yet this is what Railtrack told the public: "The development programme is anticipated to take between three and four years."

Most surprisingly, Railtrack inserted a line into the public summary that had never appeared in the consultants' report. "Most of the hardware for this train control system already exists," it read, "the technology required being relatively mature."

I read that line out to Arthur recently. He said: "Mm. That's interesting."

Did he find it misleading? Arthur paused for a long time. Eventually he said: "I don't think there's any doubt, sitting here now, that it was not as far developed as we thought it was ... I'm surprised by those words, I really am ... I am amazed at that statement. Because I don't know where they had any proof of that."

Neither Horton nor Edmonds would comment on the discrepancies. I asked another senior industry figure, with close knowledge of the subsequent attempt to make moving block work, what he thought. "To say the technology is mature - yes, I think that was a bit adventurous, certainly in 1995."

What happened? The consultants' report never mentioned it, but there was one other factor in the back of the minds of Railtrack and the Tories. 1994 had seen a painful strike by railway signalling workers. Bringing in the new technology would be another step towards ending the unions' leverage, by getting rid of thousands of signalling staff.

Some in the rail industry are inclined to give Horton, Edmonds and their colleagues the benefit of the doubt. "I think dishonesty does matter, but my suspicion is that it wasn't a question of dishonesty, it was more a question of misjudgment," a senior rail industry figure told me. "It was assumed that technology would move very quickly, but it doesn't. That was the problem. That is a bit of an indictment of the calibre of the people running the show at that time in Railtrack."

Another senior rail industry figure said: "John Edmonds was keen to get the company privatised and wanted to say things that would encourage people to believe it was bold and dynamic. It was all about creating confidence, which requires bold statements, sometimes."

Before I knew I would get to see the consultants' report, I spoke to a senior former Railtrack executive. He didn't have a copy of the report, and was trying to remember its conclusions. As I found out later, his memory didn't quite reflect the report; rather, perhaps, it reflected the real back-room conversations going on in Railtrack at the time. "The basic conclusion was that it was impossible to upgrade the west coast at any sensible cost if you went for conventional signalling," he said. "And the only way forward - whether it was feasible or not - was to bring in 21st-century signalling technology." Whether it was feasible or not: the decision was made, and Railtrack began unconsciously to weave its downfall.

The consultants weren't reckless. They never imagined that Railtrack should manage and finance the WCML modernisation itself. In some detail, they had outlined a scheme whereby Railtrack would get a big, experienced civil-engineering consortium to raise money for and manage the project, thereby assuming most of the risk. But when Norman Broadhurst, then Railtrack's finance director, studied the numbers, he thought the returns looked too juicy to be given away and brought his colleagues round. Yet one member of the board told me that he had argued against the proposal. "I said at the time Railtrack did not have the management capability to bring that in house," he said. He was proved right. No sooner had Railtrack committed itself to moving block than it began to waste time in bringing it about. In mid-1995, its rump signalling team had dwindled to the extent that it was possible for it to move office in a single taxi.The following year, a move to Birmingham caused further losses of personnel. One senior Railtrack figure at the time said: "In resources terms, two years were lost." It wasn't until March 1996, just a few months before Railtrack was privatised, that the firm picked two consortia of engineering multinationals to develop alternate prototypes of the moving-block system. Things didn't go as planned. Railtrack was beginning to suffer the consequences of Edmonds' determination to gut the firm of its in-house engineering and project-management expertise. "What Railtrack did in 1996 was quite excep tional, which was to take a really high-calibre engineering team on the BR system and destroy it," said Chris Green.

Railtrack had assumed that the two signalling consortia would develop similar types of moving-block technology. It assumed their work could then be pooled to provide the foundation for a system that actually worked. But it didn't happen that way. The consortia saw themselves as rivals.

"Not unexpectedly, their work tended to diverge rather than converge," said a senior figure in the signalling industry at the time. This would not have mattered so much, except that while the moving-block research was meandering, Railtrack made a catastrophic decision. It invited Richard Branson to hold a gun to the company's head.

· Continued in part two