It appears we live in desperate times for social democracy. We face a government which combines extreme policy with a rhetorical lock on the centre ground. Our movement is wrought by internal mutual fear and resentment. The Labour Party is now betting the house on circumventing conventional politics – no longer courting the mainstream media, and disregarding weighty think tank reports.

If we are to triumph, we must understand why our opponents do what they do, why it is successful and why it puts us in a difficult position. We can then plot a radical but pragmatic course back to power.

We do not just face a crisis in the UK. Social democracy is in crisis across its heartlands in northern and western Europe. It is true that there is a Socialist President in office in France, the Social Democrats are in power in Sweden and, arguably, the Italian Prime Minister, the Democrats’ Matteo Renzi is second only to Angela Merkel in being the continent’s most impressive politician. But they are chinks of light amid a sea of darkness.

Even these strongholds have severe weaknesses. President Hollande has been disastrously unpopular during his tenure, switching from being the spearhead of anti-austerity politics to pursuing liberal economic reforms with his Prime Minister Manuel Valls.

The once dominant Swedish Social Democrats came to power following their second worst result since universal suffrage in Sweden. And Renzi’s party is actually the inheritor of both Social Democratic and Christian Democratic traditions, and faces an opposition of far rightists and anti-systemic populists.

Traditional social democrats would have problems with his curtailing of trade union rights in the Jobs Act, and many liberals – who would rather tax wealth than work, consumption or investment – would disagree with his plans to abolish Italy’s tax on first properties.

Elsewhere – from Denmark to Spain to the Netherlands – social democratic parties find themselves with historically low levels of support. They are constrained by both the limits of government power – both hemmed in by supranational bodies like the European Union, in particular the demands of the Euro and global capitalism – but also by public opinion.

The European political turn to the politics of “them” and “us” with aggression directed at the “them” – Muslims and migrants – is particularly deadly for egalitarian social democrats. The puzzle on how to reconcile themselves to these realities is increasingly splitting the supporter base of these parties.

In a further uncomfortable irony, social democrats have been wrongfooted by an increasingly anti-elitist attitude among voters. Although social democrats may consider themselves as taking on the elite, their identification with the state and, in their Third Way form, accommodation with powerful traditional and private sector actors, makes them vulnerable in such a political atmosphere.

So it should not be surprising therefore that the Labour Party is struggling in the United Kingdom. In 2010 the party had its second-worst result since the advent of universal suffrage in this country. Despite five years of economic torpor and cuts to government spending, it barely made any ground in the recent elections.

Studies into the loss – from Lord Ashcroft, to Jon Cruddas MP, to, to the TUC – found that Labour appeared to be too close to the unions, immigrants and benefits claimants. As Jon Cruddas put Labour didn’t lose because it was seen as austerity-lite but because it was anti-austerity.

In fact, it sounded like the bell tolled. In one of its heartlands – Scotland – it was all but wiped off the map. In another – the former coalfield – Ukip ominously waits. Although both parties bridle at the comparison, the SNP and Ukip are both fuelled by anti-elitism. And they are both thriving in areas whose Labour support may be due mostly to a historic legacy, rather than present conditions.

Ed Miliband in conversation with Tony Blair. Photo: Getty Images

The heavy industry unions that undergirded Labour culture in these areas are all but long gone, leaving the Labour structure without foundation, waiting to be blown away by the populist wind.

The question is often asked in terms of the centre-left’s problems – what has it gone wrong? But what happens if we are looking at the issue through the wrong end of the telescope: what if the real question is why are Conservatives doing so well? The oft-given answer to this – by Conservatives and Labour centrists – is that Thatcher was basically right about economics and/or more in tune with the British people. The more Labour deviates from a basic Thatcherite frame, the less electorally successful it will be.

I think the reality is more complicated than this. We can again, turn the telescope. Maybe we can discuss the conditions under which a conservative regime does badly, to understand the conditions under which it does well.

The economist Amartya Sen, looking at famine in India under the British Raj, argued that even the most benevolent non-democratic regime was prone to famine, in the way a seemingly incompetent democratic regime was not. His argument was that in a democracy, there is conduit of information going from the smallest villages to the technocratic elite. Things can never become too truly awful, because if they are heading that way, politicians will be voted out.

That gives the politicians an incentive to instruct their civil servants to sort things out before they become took disastrous. It also means that politicians should prize correct information above anything else – as they don’t want to get to polling day to find out that in fact, people are very dissatisfied.

However, in a modern democracy, it is not just election results that give signals back to politicians as to the mood of communities throughout their nation. Freedom of speech means that newspapers reflect opinion, even if it is filtered through the prejudices of their owners and staff.

Furthermore, in the internet age, everyone can be their own publisher, instantly announcing how they feel. There is a steady stream of polling data as well as qualitative research, not just on headline voting intention figures but on voters attitudes to everything under the sun.

These it turns out are very good conditions for a conservative centre-right party to retain power. I take it that the actual purpose of the Conservative Party is to maintain, to the greatest extent that is politically possible, the concentration of wealth and power in as few hands as possible – but that "as possible" is pivotal.

Even if individual Conservative politicians do not see that mission as their life’s work, the donorbase of the Conservative Party certainly does. They fund the Conservative Party to safeguard socialism for the rich and capitalism for the poor. A lightly-taxed, lightly-regulated environment where unions are weak, but where the state is always there to support the rich and powerful through bailouts or money printing if required.

Some may argue that Conservatives have deeper principles, like defence of the union or sound money, But after employing a "Netanyahu strategy" against the Scots during the general election and repeated printing of money through rounds of quantitative easing, making uncosted spending pledges or putting the taxpayer on the hook for homes that buyers could not afford otherwise through the help to buy scheme, it is hard to make the case with a straight face.

Obviously any Conservative programme that was solely dedicating to concentrating wealth and power as much as possible would be wildly unpopular. However a steady stream of election results, media reports, polling findings and qualitative research enables Conservatives to gauge exactly how much they need to redistribute, and no further, in order to maintain power.

George Osborne. Photo: Getty Images

Such redistributive moves in times of stress may be difficult, as it threatens to split the party. However, in times of strength, this task can be applied to with gusto. If there is a defining feature of Cameron’s Conservative party from 2010 to 2015 it is how they accepted the New Labour settlement on anti-poverty measurers for pensioners including the winter fuel allowance, free buss passes and TV licenses even as the axe cut deep elsewhere.

They have gone beyond that original settlement to introducing a triple lock on the state pension, and have extended welfare for more prosperous pensioners through the so-called ‘silver bond’, where government borrowed from pensioners at a higher rate than the open market, but only if they had a minimum of £500.

After the election, emboldened by a majority, they have extended this into more controversial territory with the party faithful and donorbase, such as the increase in the minimum wage, the apprentice levy and a tax on non-domiciled residents. Cameron’s speech at conference was the culmination of this strategy. His analysis is that the "as possible" clause demands a deep, if uneven and mostly rhetorical raid, on Labour territory.

The more information that is available, the more refined the process becomes. So while in the 1950s, the Conservative party had to make great big clunking moves to the left on welfare and trade unions to be sure of retaining power, today with much more information the party can zero in on exactly how much more they have to give and remain viable, but no more. The tendency of central planning in all this by a supposed free market party, especially in the context of Messina-style data-driven politics, is almost deliciously ironic.

So while they can move dramatically to left on, for example, pensioner welfare and even childcare to support those inside their voter client groups, for those outside the tent, the Conservatives can be more extreme than ever. In fact they have to be. In order to be able to afford the extra welfare where they need to give to their voters, the cuts elsewhere have to be deeper if they are to defend the wealth piles of their donors.

Is this just a very complicated way of saying that Conservatives have moved to the centre? Perhaps - although it’s worth analysing the central process of what’s going on. And it could be argued that "moving to the centre" for Conservatives in recent years actually obscures two different processes. One is outlined above. The other is in welcoming into the establishment those businesses that rely on public sector contracts and pour millions into lobbies and think tanks, to work towards ensuring there is a constant stream of taxpayer money going their way.

But if this is just an analysis of Conservatives moving to the centre, why can’t Labour just do the engage in a similar process of moving from the left to the centre? However, it may be deadly for Labour to move to act in a similar way.

One possible frame to think about this through is Anthony Down’s Median Voter theorem. Downs famously projected that all the voters could be lined up on a right to left axis.

Therefore, in a two party systems, both parties should converge towards the centre to capture as many votes as possible. If the parties are equidistant from the centre, and voters are symetically distributed from the centre of the left-right axis, you would expect them to receive the same number of votes.

However, it may be the case that if Conservatives and Labour are equidistant from the centre, we would expect the Conservatives to win and Labour to lose. Crudely put, we can think of the Right-Left axis traditionally in Britain as based on redistribution of income and wealth – those on the right wanting as little as possible and those on the left wanting more.

For the sake of simplicity, say these can be aligned to self-interest, with those on the right generally holding wealth and income and benefiting from wealth and income being held privately and those on the left generally holding little or benefitting from it being held collectively.

If redistribution of wealth and income was carried out from what set of citizens as citizens, to another set of citizens as citizens, then a Downsian theory of political convergence, where both parties would benefit equally from such movement would make sense.

But it maybe that while we are taxed in general, we receive benefits in particular. Therefore, the Conservatives can make a pitch to be on the side of ‘taxpayers’ and be making a pitch for most of the country. As the Conservatives move towards the centre, one might expect the very rich to benefit less from their potential governance. But they still can identify with the general principle of a low tax party, and they can still be expected to be taxed less under the Conservatives than Labour.

However, we tend to receive benefits in particular. We receive support from the state because we are unemployed, or are ill, or have disabilities, or require support to access the legal justice system etc.

As Labour moves towards the centre, it could “salami slice”, taxing less and therefore having to cut similar amounts from the receipts of all beneficiates to redistribution.

But in practice, it decides to preserve the benefits of some, and reduce considerably or entirely the benefits of others. Voters in those groups with reduced or eliminated benefits have a much reduced or entirely eliminated incentive to vote Labour – it is less likely to matter to them now who gets elected.

We saw with the Conservatives that as they move towards the centre, those on their “wing” still have an incentive to vote Conservative. However, if Labour engages in a similar process voters on their wing increasingly do not.

The reality may be even worse than that. Because as the Conservatives move towards the centre ground by being willing to raise more taxes, they can redistribute that in particular, rather than in general, targeting certain voter groups who benefit from redistribution such as poor pensioners. Therefore they can “vault over” the Downsian median voter, and take some voters from Labour’s wing.

Labour could promise tax breaks for certain groups – like small business people for example, to vault similarly over the median voter, but can only do that at the expense of less revenue, which would mean reducing benefits for certain groups in particular and therefore losing voters on "their" wing.

The only way that the Labour Party – or any left of centre party can win under such circumstances, is if the Conservative wing is ‘split’. Traditionally the fault line of the Conservative party has been over free trade versus protectionism, which has been partially reborn as an argument over the European Union.

In addition, the Conservative’s wing can be split over whether the Conservative Party is competent or not. But absent such a split, or worries over Conservative competence, it is hard to see how Labour wins.

Jeremy Corbyn listens to John McDonnell deliver his conferece speech. Photo: Getty Images

This may ultimately flag up a central problem with the democratic state as a vehicle for equality, or maximising the welfare of all its citizens. If the above theory does describe how politics works, then it means that the democratic state does not work to maximise the welfare of all its citizens but a plurality of its citizens - that is a plurality of its voters.

Usually that would be a plurality of its voters needed for the party that represents the mission of leaving wealth and income as concentrated as possible to retain power.

Depressingly, there are reasons to believe that the plurality needed is smaller than it was previously. First of all, turnout in elections is one a general downward trend, meaning that the centre-right party needs to attract fewer votes than it did previously to attract a plurality of votes.

Second of all, the sheer amount of research done by the civil service, think tanks, and polling companies means that while making cuts to services, any centre-right Chancellor can carefully adjust the severity of the cut and the number of people who suffer the cut to maintain plurality support.

The result we are left with is a sort of zombie social democracy which is alive and dead yet neither alive nor dead. Centre-right politicians can use the clunking levers of state to redistribute wealth and income in the economy, but only so far. But the cuts elsewhere must be deeper to satisfy the rich and powerful that that party ultimately serves.

The tools of social democracy remain sharp and strong, ready for use, but are used in a way that entrenches inequality. And they are used in a way to safeguard the country against a social democratic party winning.

Meanwhile, the danger for the social democratic party is that it mistakes the deep cuts that must occur outside the centre-right’s party client base for a extreme free market or so-called neoliberal ideology, and campaigns against perceived extremism. However, this fails to resonate with the centre-right’s protected voter base. The alternative for Labour, to move towards the centre opens up dangers of splitting the party. The fact that the centre-right party uses progressive means for conservative ends merely adds extra torture for a social democratic party.

As long as social democrats see the state as the sole or even primary tool for achieving its goals, it will fall into this trap, because the democratic state is a plurality-serving, not a universal-serving institution.

Those who want to achieve a more equal society, where there are realms of our life not governed by market relationships, should not seek to capture the state in order to use it to promote welfare, as it tends to promote the welfare of a plurality of voters.

Instead they should seek to use the state to protect and promote those intuitions which are universal-serving, not just plurality serving. By a universal-serving institution, what I mean is one that has an incentive to expand continuously as well as enhance the welfare of its users, clients or members.

For consumers, this can take the form of co-operatives, entering markets where there is a lack of competition to keep it honest. For employees, this can take the form of unions or co-operative forms of business.

So for example, I doubt that organising cab drivers to resist Uber, through lobbying to increase government regulation will achieve much for social justice. Most straightforwardly, Conservatives will rally consumers who far outnumber cab drivers. Or they will agree to regulate for cab drivers, but having been co-opted into the Conservative voter coalition, those cab drivers will no longer be open to voting Labour.

It is better for Labour to introduce legislation to make it easier to create co-operatives. In Denver, Colorado for example, cab drivers have formed an app-based cab co-operative to challenge Uber. Once those cab drivers have a general incentive to sustain a pro-co-operative government, they may continue to support Labour even after their initial grievance has been settled.

Tony Blair on the campaign trial in 1997. Photo: Getty Images

Although such a pro-co-operative and pro-union agenda would involve strengthening such institutions, this should not be in a way that enables them into become a plurality-serving institution – or for insider groups, in the terminology of centre-right public choice theory.

So for example while it should be made dramatically easier for unions to organise – bringing in a employer neutrality rule during organising drives, as they have in Israel for example, there should be no bringing back of the closed shop. Unions should be able to investigate regulatory adherence in non-union workplaces as they do in Los Angeles, but not engage in restrictive practices in general.

Unions must also innovate. Iain Duncan Smith has floated the idea of voluntary income protection, which many on the left see as a threat to a universal welfare state. But arguably, income protection allows an avenue for unions to organise among the temporary and self-employed. This does start to blur the line between union and co-operative, but that is the way unions must head anyway. In an increasingly services-based economy, much of the means of production and distribution is merely the information held linking producer and consumer. We should be able to socialise this.

Furthermore, the more workers employed – and self-employed – are organised into unions or co-operatives where their welfare is dependent on those institutions, the harder it is for the Conservatives to minimise the number of voters required for the ultimate insider group – the number of voters required to return a conservative government. If voters are sticker – that is their welfare is bound with the welfare of other voters – then it may be impossible to support the 37 per cent of the voters the Conservatives need without supporting far more.

This emphasis on workplace protection and welfare being delivered by non-state groups that have an interest in continually expanding, obviously can only work for those of working age who are able to work a substantially proportion of hours per week. It does not give an answer for the working age with disabilities, children or pensioners.

For these groups, the answer is still likely to come through the state. However, the state can come closest to being a universal-serving institution through the idea of a basic income that it provides all citizens. This idea has had advocates across the political spectrum from Martin Luther King to Milton Friedman, who supported as the welfare system least likely to create disincentives to work and safe.

It could be argues that the government does guarantee a basic income to pensioners, and that is one reason why the group that American and British government have been most successful at reducing poverty among is the elderly, from Lyndon Baines Johnson’s Great Societies to the New Labour reforms.

However, one may want to complete the process by merging all the universal benefits, from the pension, to the free TV licence and the Winter Fuel allowance. The more pensioners’ interest in government welfare act as a single bloc – with the wealthiest benefitting as much from pro-pensioner policies and the poorest, the more likely it is any government, including a Conservative one, will help the poorest. It would make the votes of pensioners “stickier”.

Children and those who are of working age but suffer from disabilities is far more difficult. Frankly put, children can’t vote in a democracy and with some projecting that a majority of voters will be over the age of 55 at the next election, their parents do not matter as much as when Tony Blair made his famous child poverty pledge.

Those with disabilities, demonised by the Conservatives and the mainstream press to the point where many of those with disabilities believe the myths themselves also make extending government support difficult. There could be interesting policy ideas here from replicating Osborne’s pensioner bond to families with children with a much lower threshold or introducing a carer’s leave, but frustratingly, most work here would may have to involve redistribution by stealth.

Another key plank of a Labour platform at the next election should be devolution. This is because although the Conservatives may only need 37 per cent of the vote nationally to win a general election, that 37 per cent is unlikely to be distributed evenly across the country. Therefore, by introducing widespread devolution, the Conservatives cannot afford to whittle down their voter coalition too small, or else risk losing vast amounts of power across the country. This is not just an electoral point – Parties help those they need to help to retain power. If you increase the number of people who are needed for a Party to retain power, then number of people who are helped will increase.

So the basic platform for the next Labour election bid, should include the empowerment of unions and co-operatives, and widespread devolution. Elsewhere it, unfortunately be thin gruel, especially for the ultimate non-voters in a democracy, prospective immigrants.

One possible way to thicken the gruel is by hugging Cameron, rather as Cameron hugged the Blairites on academies. Labour should offer joint committees with the Conservatives on raising rages, equalities and prison reform, which, of course include civil society, business and trade unions. The Conservatives could either turn this down – in which case they look phony – or accept – in which case they further legitimate this political space, and allow Labour to run as "the real thing" in 2020.

However, much good work elsewhere in a Labour first term will have to occur by stealth. But during that first term, if unions and co-operatives can be built, and devolution delivered, then the Conservative strategy of utilising the tools of social democracy to help the minimum number of voters needed to retain power, and shutting other groups out will be blunted. Labour would be able to take more risks in its second term

It’s a harsh truth that social democracy has been reduced to a zombie. It’s a harsher truth for the millions who depend on it, than elected Labour politicians on comfortable salaries. But ignoring the facts won’t get us anywhere. It’s time to lay social democracy to rest, and rediscover solidarity.