This is the bad news gospel: the austerity agenda is succeeding. Astoundingly, neoliberalism is not dead. The chancellor of the exchequer is crowing, for good reason, and the British left is faced with its most serious crisis in decades.

How can it be that more than six years since the credit crunch, with austerity under way for more than three years, the left has barely showed signs of life, let alone scored a single significant victory? Particularly when capitalism has been in its deepest crisis in generations. We have been floored by austerity, and above all passive acceptance.

To understand how we got to this impasse, we need to radically rethink many of our core assumptions. The first is the engrained idea that a capitalist crisis necessarily leads to radicalisation. As political theorist Antonio Gramsci pointed out, it is the "traditional ruling class" rather than its opponents who are best positioned to take command of a crisis. Its control over the dominant institutions, its loyal cadres of supporters in thinktanks and the media, and its economic and political strength, all enable it to adapt and propose its own solutions. Proactively, it seeks to meet the crisis on every level on which it manifests itself by changing strategies, winning over popular layers with "demagogic promises", and pre-empting and isolating opponents.

Austerity as crisis management

Austerity is one form of doing this. However, it should not be interpreted in a narrowly economic sense. Of course, the right and even much of the centre-left argue that austerity is simple economic common sense: addressing the deficit boosts the confidence of consumers and investors and enables new cycles of growth. If this were the case, the project would have to be deemed a failure. The evidence of present and past austerity programmes, and even the view of the Office for Budget Responsibility, is that they retard growth rather than stimulate it. They certainly don't make the repayment of debts easier.

"Businessmen don't cling to doctrines without their agendas being served in some way. They would be uninterested in a simple wealth grab that doesn't promise to actually resume capitalist dynamism"

The left's response is not always much better. Either it treats austerity as a narrowly doctrinal programme, or as a sort of conspiracy of the rich to simply pilfer resources – or both. Granting that austerity has its dogmatists and mythologies, and allowing that it has intentional distributive consequences, it has to be said that the "traditional ruling class" is not stupid. Businessmen don't cling to doctrines without their agendas being served in some way. On the other hand, they also intend to survive well into the future, and would be uninterested in a simple wealth grab that doesn't promise to actually resume capitalist dynamism.

To indicate a different approach, let's take an example of an early austerity programme. In New York in the mid-1970s the city was running an increasingly unmanageable deficit, with the servicing of debt consuming about a fifth of its operating funds. Underlying the crisis was the evisceration of the city's manufacturing base in the postwar era, which drove up unemployment and thus welfare rolls. One of the main factors sustaining employment had been the growth of the city administration. The growing power of public sector employees allowed them to win better pay and conditions, and gave them a degree of political clout. The costs of the state's expansion were partly supported by federal funding, but taxes on local business and property owners made up the rest. The postwar agenda of liberal reform, especially that associated with Lydon Johnson's Great Society programme, was ideologically legitimised by the idea of America as a wealthy, growing economy in which some of the benefits should be extended to the poorest.

'From 1975 to 1978, New York was subject to an austerity regime. This involved not simply a set of policies, but an exceptional form of the state.' Photograph: Afton Almaraz/Getty Images

However, as the global economy tanked in the early 70s and the Bretton Woods system collapsed, New York began to accumulate more and more debt. Ironically, the same banks that would later complain of fiscal irresponsibility profited greatly from the debt. As it mounted they demanded that the financial system should have first line on the city's funds in the event of bankruptcy. From 1975 to 1978, therefore, New York City was subject to an austerity regime. This involved not simply a set of policies, but an exceptional form of the state – a set of special institutions with extraordinary, wide-ranging legal powers, the most important of which was the Emergency Financial Control Board. Dominated by bankers, corporate interests and the city executive, these institutions took command of the crisis by cutting services to low-income New Yorkers, attacking working conditions for the city's unionised workers, and offering incentives to its wealthy financial class.

The elite argument for austerity was simple. The city's crisis was primarily one of overspending, driven by too many services for the poor, too many bureaucrats to run them, a union-driven cost of labour increases, and a corrupt and inefficient city management driving the productive layers out of the city with burdensome taxes. The solution was to reduce the burden of the unproductive on the productive, and let wealth creators keep more of their wealth. This argument was supported by the majority of the media. The union-led opposition highlighted the tremendous wealth enjoyed by the bankers and corporations who were demanding austerity, but the business interests were canny enough to recede into the background and refuse to publicly comment on controversies in which they were deeply involved.

The unions were no match for the organised business offensive. The major corporations had their control of markets and operating capital as a considerable leverage over the city. The political clout of the unions, meanwhile, was based around bargaining mechanisms designed to swerve the very kind of confrontation they could not avoid. They accepted the dominant narrative about the crisis and its causes, accepted the need for some cutbacks, and then sought to narrowly protect their conditions within that framework.

The austerity solutions worked in the precise sense intended: by drastically reconfiguring the city's class relations, reorganising the state to marginalise popular constituencies, and winning the ideological battle for placing more authority and wealth in the hands of entrepreneurs, they started to restore profitability to capital. Soon, the austerity project would be launched nationwide, beginning with the Volcker shock in which loan rates were raised to a crippling 21%, thus driving down incomes, suppressing growth and breaking the spine of organised labour, with similar overall effects.

"Austerity reorganised the ideological terrain: the narrow focus on the fiscal crisis allowed elites to highlight overspending as the key problem"

The austerity project thus took hold of a situation in flux, a crisis, and imposed a solution in the interests of a specific class. It did so along three axes: class, state and ideology. First, it reorganised the balance of class power in such a way as to transfer wealth away from popular consumption and towards business investment. Second, it reorganised the state apparatus in a profoundly undemocratic way, linked to a wider tilt in the balance of power toward authoritarianism, marginalising and excluding popular constituencies. And third, it reorganised the ideological terrain: while the wider crisis was caused by generic dysfunctions inherent in capitalism – the decline of manufacturing, for instance, due in part to capital's drive to reduce costs and rationalise production – the narrow focus on the fiscal crisis allowed elites to highlight overspending as the key problem. On the basis that theirs was the only solution that could restore growth and general prosperity, they could link their particular class interests to the interests of the whole city. And in the process they began to displace the postwar liberal consensus in favour of a neoliberal orthodoxy that placed the emphasis on markets and competition.

The neoliberal terrain

The FTSE100 shows sharp drops on Monday 6 October 2008. 'The dominant strains of neoliberalism have always favoured an interventionist state … The British government's bailout of finance is an extension of this.' Photograph: Lefteris Pitarakis/AP

The austerity projects we face today are different in one important respect: they come after almost 40 years of neoliberal offensive. This is something which large sections of the left have totally misunderstood; a significant reason for its disarray. For most of the past three decades, neoliberalism has been chiefly analysed as a kind of free market fundamentalism, which is but a glimpsing scratch of the surface.

If neoliberalism was chiefly about free market fundamentalism, then it would be possible to understand the salience of the state as a post-credit crunch economic factor as a repudiation of that orthodoxy. Indeed, many on the left did prematurely pronounce neoliberalism deceased.

However, the dominant strains of neoliberalism have always favoured an interventionist state. It is not the volume of state activity that is the concern of neoliberals, but its character. Neoliberalism is unlike classical liberalism in that it does not assume a human propensity to truck, barter and trade as the basis for political organisation; neoliberals learned through the bitter experience of the 20th century that human behaviour could be as collective as it could be competitive. Thus, a strong state was required not merely to protect property, but to discipline its subjects and educate them in the new neoliberal dispensations.

This is evident in the expansion of penal and coercive institutions, the periodic bailouts of dominant sectors of industry and finance, and the fusion between capital and the state particular to neoliberalism, in which state functions are outsourced or semi-privatised, while being publicly funded. The British government's bailout of finance is an extension of this latter pattern.

So what has 40 years of neoliberal statecraft achieved? The political possibilities have been narrowed through serial defeats of the left, the consequent incorporation of social democratic parties into the neoliberal consensus, and the transformation of state apparatus in a less democratic direction. No governing social democratic party offers a serious alternative to the austerity remedy. The diminution of practical solidarity following on from the state-led defeats inflicted on organised labour is far-reaching. Nine out of 10 private sector workplaces have never seen a union rep, let alone a picket line; the number of days lost to strike action in recent years have been, barring a relatively small spike in 2011, at historic lows. The idea of "rank and file" organisation, let alone wildcat strike action, is something seen only on the peripheries of the labour movement. Trade unions have been effectively disciplined. This is an important reason why the labour response to austerity has been so feeble.

The traditional ruling class is not merely good at exploiting opportunities; it thinks long-term in a way the left must learn to do

Ideologically, there has been a long-term generational shift against the welfare state, and in favour of competitive behaviour. Indeed, competition has increasingly been built into the public sector ("internal markets"), and disciplinary techniques built into social security (in the form of "workfare", for example.) While older generations experienced the welfare state as part of a collective unity, younger generations have experienced it as part of a zero-sum competition for resources. This is the ground on which support for some of the most punitive aspects of austerity, such as welfare cuts, has been constructed; this is the result of a conscious political strategy. The traditional ruling class is not merely good at exploiting opportunities; it thinks long-term in a way the left must learn to do.

Today's austerity projects thus build on the successes of the past, and may continue to do so in the face of the left's perplexity until we begin anew.

Facing up to the bad new things

Bertolt Brecht. Photograph: Paul Hamann/AFP/Getty

The above analysis may be too pessimistic. And pessimism is a problem to the extent that one of the ways in which neoliberalism prevails is through consistently demoralising people. Yet what is ultimately more demoralising? To soberly face our situation and begin the hard, slow-burning, patient work of reconstruction, or continue to rally to sloganistic exhortations, thinking that each new protest or strike might radically shift the balance in our favour?

As the Brechtian maxim has it: "Don't start from the good old things, but the bad new ones."

• Richard Seymour's book Against Austerity is available via the Guardian Bookshop for £10.39