It is an awkward fact for many on the left that the partial privatisation of the English National Health Service – started by the New Labour government in 2003 and enthusiastically accelerated by the current Tory-led coalition – has been such an apparent success. When Tony Blair came to power in 1997 the service was struggling, particularly in terms of elective (non-urgent) care. Like all GPs, I often saw my patients having to wait up to 18 months for routine operations.

New Labour’s initial diagnosis was of inadequate resources: public spending on health as the party returned to power was roughly 5 per cent of GDP, substantially lower than in every other developed nation. Blair’s stated ambition was to bring this percentage up to the European average; more money, it was believed, would solve the problem. Labour set about the task with gusto. Gordon Brown’s first Budget in July 1997 heralded an immediate injection of £1.2bn into the NHS, with real-terms spending to rise year on year thereafter. At the same time, Labour began to dismantle some of the previous Conservative government’s experiments with “marketisation” – ending GP fundholding (under which some family doctors operated budgets on behalf of their patients) and re-emphasising collaboration over competition between different parts of the system. For me, as for most ardent supporters of a public-service NHS, they were optimistic days.

Yet by the time of the 2001 election, New Labour was facing accusations of failure to reform. Despite the extra investment in health, service improvements had been frustratingly slow to materialise and were incremental in scale. In addition, there was a worrying new trend. Just as they were doing with the education of their children, the increasingly prosperous middle classes were opting out of state health-care provision in ever greater numbers. Over a few short years it had become noticeably more common for patients in my reasonably affluent corner of southern England to declare they had enough spare cash, or had private insurance (often included in their employment package), and would like to use it to sidestep lengthy NHS waiting times. In an era when most of us swallowed Brown’s “no more boom and bust” myth, and there was a sense the good times might just keep rolling on, we appeared to be sleepwalking towards a US-style health-care system, where those with sufficient resources could get swift access to private treatment, leaving the rest to make do with what the public service could manage to provide.

The danger, as Blair realised, was in the medium to long term: the departure of the middle class would undermine the social contract on which the very idea of a national health service depends.

These were the considerations that led to the extraordinary spectacle of a Labour government adopting a policy direction not even the Tories had dared to explore. The NHS had to become so responsive and user-friendly that there would be no incentive for anyone to go elsewhere. In short, it must be able to compete with the private sector – and the way to do that, it seemed, was to make it “compete”.

To begin with, the Blair government’s approach was to “market-make”; and so, from 2003, successive waves of independent sector treatment centres (ISTCs) were opened throughout England. Run by private companies for profit, ISTCs were contracted (often on very favourable terms) to provide solely NHS elective procedures, creating extra capacity in the system to bring down waiting lists, and at the same time forcing existing providers to polish up their act if they wanted to hang on to any of their more “profitable” work. In parallel, the best NHS hospitals were able to apply for the new foundation trust status, which freed them from public-service constraints to operate more like private businesses.

Out of this market-making grew a new logic: that as well as deliberately inserting private provision inside the NHS, the health market should be opened to external competition. In 2009, in what transpired to be its dying days, New Labour introduced the “any qualified provider” (AQP) initiative, which allowed the private sector to undertake NHS work outside the ISTC programme. It is under AQP that the vast majority of my patients who require elective procedures now choose to spurn both our local district general and the ISTC in favour of referral to the nearby private hospital run by Circle.

The coalition government seized on the inroads made by New Labour. As well as cementing competition for work on a case-by-case

basis under AQP, Section 75 of their Health and Social Care Act 2012 makes it obligatory for commissioners to put every new NHS service (above a trivial size) out to tender. Analysis of data up to 2013 shows more than £12bn of NHS contracts were awarded to private companies during the first three years of the coalition.

On the face of it, the drive to compel competition has done what it was supposed to do (albeit at vastly increased administration costs, with contracts being negotiated, invoiced and monitored by armies of bean-counters on all sides). Much elective NHS care nowadays is provided within weeks, not months or even years. This is unquestionably good for patients. And fears for the future of the social contract have receded: where is the advantage in going private when you can get your operation paid for by the NHS at the same independent hospital?

The health insurance industry has adapted to the new realities, offering cheaper products that pay out only if the NHS should be unable to provide treatment within a specified time frame. With the fall in disposable income that has accompanied austerity, it is once again relatively unusual for my patients to request private referrals. One way or another, the NHS has remained the franchise to which most people look when they have an elective health-care need.

Why then is there a renewed row over NHS privatisation in the current election campaign? It has often been said (by both Labour and the Conservatives at different times) that patients don’t really care who provides their treatment, as long as it’s convenient, of good quality and funded out of general taxation. Surely Ed Miliband should be claiming credit for Labour having been bold enough to go where no political party had ever dared tread? And why is Andy Burnham, the shadow health secretary, publicly committed to repealing the Health and Social Care Act, with its compulsion to competitive procurement?

Burnham is resurrecting the language of the past, articulating a desire to see the NHS as the “preferred provider” of most services, and labelling the 100 days of this election campaign as the last chance to save this concept. Is this simply a belated restatement of an ideology that the left is now embarrassed to have renounced during its most recent years in government? An ideology, furthermore, whose time has been and gone?

The answers to those questions lie in the nature of the problems now facing the health service and how the privatisation agenda has created barriers to tackling them. This is where things begin to get complicated, which is why politicians generally shy away from trying to air them in the media, preferring to fall back on meaningless soundbites such as X billion pounds’ additional spending, or Y thousand extra doctors and nurses. Let me take you on a whistle-stop tour.

The first thing to appreciate is that commercial competition was a response to the NHS’s historically poor performance in providing timely access to mundane, high-volume procedures: cataract removals, joint replacements, gall bladder operations and so on. These elective cases are all discrete episodes: there’s a single problem and a definable clinical activity that will close the case. There are also readily quantifiable measures by which performance can be rated: most obviously, the length of the waiting list.

Markets can work well in this sort of scenario, particularly if risk can be mitigated by excluding complex, often very elderly patients in poor general health with multiple chronic diseases, who are more likely to experience unpredictable and expensive complications. The problem is, with every passing year, there are more and more of us living to become just this kind of patient – patients the private sector doesn’t want to do elective business with at NHS tariff prices, and for whom the old NHS is therefore the default source of help.

The second issue is that these elderly patients with multiple health problems are also presenting to the NHS’s other major arm – urgent-care services – in ever greater numbers. Their health is fragile and they are prone to frequent exacerbations in underlying chronic conditions such as heart failure or lung disease. Otherwise trivial illnesses can have a devastating impact – a simple urinary infection will, in a matter of hours, render a frail and elderly patient completely “off legs” and unable to look after him or herself. Social circumstances are often precarious, patients widowed or living with an equally vulnerable spouse, with far-flung and busy families unable to provide a rapid response should the home situation suddenly deteriorate.

When a patient of this kind becomes unwell, unless significant nursing and social care can be parachuted in at a moment’s notice to shore up community treatment (and at present they can’t) he or she is heading for hospital. Once the person is an inpatient, it can take an unconscionable length of time to help them rehabilitate, and for the social-care system to reinstate or augment a package of care that will allow them to be discharged. Beds get filled; beds get “blocked”.

The third factor is the changed face of NHS urgent-care services. There are all sorts of things one could say about this but here’s the fundamental point: when someone with anything more than a completely straightforward illness becomes unwell, at some stage you are going to need an experienced clinician to decide how to manage it. When I began in practice in 1990 there were only three places you could turn to if a crisis arose: your GP (day or night), the ambulance service, or A&E. The system was understood by virtually everyone and the vast majority of contacts went through their GP first. This, crucially, introduced a highly trained professional at the earliest stage of the process. GPs are thoroughly at home managing uncertainty and negotiating complexity, and we kept a vast amount of work away from hospitals.

Nowadays there is a plethora of other entry points into the urgent-care system – the NHS 111 helpline, walk-in centres, out-of- hours (OOH) services (now mostly provided by private companies) and minor injuries units. NHS 111 and, to a variable extent, the others employ either non-clinical staff operating a risk-averse computer algorithm, or clinicians who are junior and inexperienced. The net result is that the first time many patients encounter an experienced clinician is long after they’ve been admitted to hospital. The opportunity for community management, if it existed, has been lost.

These are the principal forces behind the flurry of declared major incidents this January, which led to hospitals up and down the country closing their full-to-bursting doors. Our own district general remained open – just – but in a continual state of black alert (which is every bit as bad as the name suggests). All elective surgery was abandoned and extraordinary measures were employed to free up every scrap of capacity.

If we want to do anything other than lurch from crisis to crisis, the whole system will have to be reconfigured. Hospitals, GP surgeries, community nursing, OOH, NHS 111, the ambulance service, walk-in centres and minor injuries units are all nominally NHS bodies and should, in theory, be able to work together to ensure only patients genuinely in need of acute hospital care are admitted. The problem is, in our present-day competitive NHS, each entity is trying to protect its budget and ensure its own performance meets the benchmarks by which it will be judged next time its contract comes up for renewal. Perverse and protectionist behaviour ricochets round the system, the easiest solution often being to admit a complex patient and let their care become the responsibility of the hospital. And that’s before you try to bring social care into the mix, which is integral to the project of supporting unwell patients in their homes but which historically has been provided by local government out of a completely separate (and even more pressured) budget.

It is in this incredibly complex and messy situation that Circle – the first private company to be awarded a contract to run an NHS district general hospital, at Hinchingbrooke in Cambridgeshire – announced recently that it will walk away. It’s not that a commercial company can’t run a modern acute hospital; there are half a dozen such private facilities in London (though nowhere else in the country is affluent enough to sustain one). It’s that the kind of money the NHS is offering is woefully inadequate to mitigate the risk to the private sector of unpredictable and ever more intense surges of demand, exacerbated by perverse behaviour elsewhere in the system. Circle is going back to running its controllable elective AQP business, licking the wounds that it has sustained from its adventure into the NHS acute sector.

We made a concerted effort in our area a couple of years ago to solve the problem with urgent care. Most of the big players – our district general hospital, all local GP surgeries, the ambulance service, OOH and the walk-in centre – joined together in an effort to run the newly recommissioned service. This would have aligned the interests of all parties better and should have led to some creative solutions. However, under Section 75 regulations the procurement had to be competitive, with each of the nine eventual bidders being judged on quasi-objective grounds that were rooted largely in process and that weighed only things that could readily be measured. Such is the fear of litigation under competition law that there is simply no latitude for commissioners to use common sense or professional judgement to prefer a bid on the grounds that it is a good idea and exactly what the local area needs. Our bid narrowly lost out to a company based several hundred miles away.

As well as this structural bar to commissioning joined-up working, competitive procurement is eroding the goodwill and loyalty that the NHS has historically enjoyed from its workforce. The firm that won the contract in our area now runs the out-of-hours service and urgent-care centre adjacent to A&E. It has struggled to appoint a local clinical director (the post is still vacant a year in). Many staff who supported out-of-hours provision for years have walked away, so alienated do they feel; each week, the company has to fly or chauffeur clinicians and drivers from elsewhere in the country just to keep what is at times a skeleton service going. Turnover is high and those local staff who continue to work under the new regime are weary of the constant appeals to step into the breach to fill rota gaps.

Staff and doctors who once willingly responded to requests for assistance leave their phones unanswered when they recognise the number of the rota administrator. A rich but unquantifiable resource, which might be called the public-service ethos in the NHS, has been squandered in front of our eyes. Even at this stage it may be too late to recapture it.

The deleterious effects of a competitive marketplace have been loudly argued by opponents of privatisation throughout the past decade. Yet according to one commissioner with over 20 years’ experience of health-

service procurement, no one in government had any vision of how the competition agenda might degrade integrated systems of care for patients with multiple diseases. The

focus was unrelentingly on improving elective care – the NHS’s low-hanging fruit – with fingers crossed in the forlorn hope that the changes being made wouldn’t destabilise the rest of the service.

Of the major parties contesting the forthcoming general election, it is Labour that seems to understand the issue, and it is this that underpins Andy Burnham’s pledge to repeal the Health and Social Care Act and to legislate to exempt the NHS from EU competition legislation. Integrated care is the only game in town and it can only be delivered within projected levels of spending by well-configured public services that have been freed from the fragmentary consequences of enforced competition. That said, Labour finds itself in an embarrassing position: the party that began privatisation has to explain why that process – which has, after all, resulted in improvements in the elective-care arm of the service – is simultaneously incompatible with meeting the present-day challenges the NHS faces.

The Conservatives, by contrast, are silent; the NHS was conspicuously absent when they announced their six key manifesto areas. Having gone into the last election promising no more top-down reorganisations of the service, and having then presided over arguably the most damaging such reorganisation in the history of the service, they may quite reasonably believe that nothing they say on the subject will be trusted. They may also have calculated that the complexity of the problem defies exploration in our soundbite-dominated culture and that saying nothing will allow them to continue business as usual, should they be re-elected. If so, that would be a cynical continuation of the approach that has created the mess we are all dealing with.

Burnham is right: this election does represent a fundamental decision point as to how our NHS will develop or degrade in the future. We need to know, well in advance of the poll, where each party stands on this important matter. And having declared its approach, whichever party goes on to lead the next government must somehow be held to keep the promises on which it has been voted into power.

Dr Phil Whitaker is an award-winning novelist. He writes the New Statesman’s Health Matters column