It is a rare sunny summer morning and I am on the bus from Stoke Newington to Bloomsbury in central London. In these old, slightly earnest parts of the capital, leftwing politics runs deep: from Karl Marx writing in the British Library to communes in the 70s to today's dogged socialist flyposters. This morning's bus ride does not disappoint. Seated in front of me, en route to Marxism 2009, the pre-eminent British gathering of the international radical left, are a clean-cut man and woman in their early 20s. He is wearing a crisp new T-shirt that reads "Revolución Bolivarana". She has a large rucksack. They are speaking German, but the word "socialism" recurs.

The papers today are full of the recession as usual. On the Today programme, David Cameron has been talking about emergency cuts in government spending, and a union leader has been fiercely defending the wages of public sector workers. It could almost be the heady days of the mid-70s, when capitalism seemed to struggle for breath and all political bets appeared to be off.

At Euston station, the couple get off the bus. I follow them, past the looming tower of Network Rail headquarters – once the chaotic private-sector Railtrack, until it was nationalised – and into the complex of meeting rooms hosting Marxism 2009. But the atmosphere inside comes as something of a shock. It is the final, supposedly climactic day of the conference. The speakers are reasonably intriguing and diverse – the radical playwright David Edgar, the dissident Labour MP Jeremy Corbyn, the rising young union boss Mark Serwotka. And yet, Marxism 2009 feels little different from most such leftwing summits in Britain over the last quarter century. The corridors are animated rather than feverish. Attendees greet each other as old friends and comrades rather than eager new converts. The pavement outside has moderately busy stalls for the usual causes: opposition to Israeli land occupations, opposition to the British National Party.

At one table, a weatherbeaten man sits alone selling DVDs of "activist news" and collecting names and addresses. The sky above turns overcast, then steadily darkens. It starts to pour, but he does not move. As the rain soaks his hair and jacket, he sits still and erect, impressively defiant but a bit absurd. The ink on his list of names starts to run.

The last year should have been a happy one for the left. The great global lab experiment in unfettered finance capitalism has blown up. Bankers have become pariahs. Taxes on the rich have gone up. The pages of the financial press have had a frequent air of panic. New Labour has fallen out of love with the free market. Above all, the rightwing economic and political ideas first popularised by Margaret Thatcher in the 70s have, finally, lost their air of impregnability.

"These are the best circumstances to make the left case we've known for an awful long time," says Neal Lawson, head of the leftwing pressure group Compass, "since way back before 1979, since back to the 30s." Geoff Mulgan, the former Labour strategist and a longtime observer of the left, agrees: "This is a moment that should be incredibly propitious for the left. Capitalism is collapsing. You don't get more propitious than that."

There is also the widening recognition that free-market countries have deep social as well as economic problems. Earlier this year Richard Wilkinson and Kate Pickett, at the time almost unknown outside academia, published The Spirit Level: Why More Equal Societies Almost Always Do Better. Its findings about the failings of the most fiercely capitalist countries, such as Britain and the US, in everything from obesity to violent crime to mental health, received overwhelming acclaim in both the rightwing and liberal press. Wilkinson says he is now "absolutely deluged with invitations to speak: to religious groups, to civil servants, to government". In academia he senses an intellectual tide running leftwards: "In a lot of different subjects there's a move towards a fundamental recognition of how social people are. In neurology, epidemiology, social psychology, child development, there's lots of evidence that humans do better if they're collaborative."

And yet, in Britain and most comparable countries the left is not thriving. Quite the opposite. The Brown government's mild tilt to the left has made it no more popular. At the European elections in June, left-leaning parties, whether in office or opposition, cautious or militant, were trounced across the continent. Votes went instead to mainstream conservative parties or far right and anti- immigration groups. Over the summer the broader political debate, particularly in Britain, has shifted in the same direction: "The crisis of the financial markets has become a crisis of public spending – it's incredible!" says Hilary Wainwright, editor of leftwing magazine Red Pepper. "Public servants are going to be scrutinised down to the last paperclip, while bankers are not going to be scrutinised down to the last million they have received from the government."

Has the left missed its moment? The radical American writer Rebecca Solnit fears so. "It felt like last October [the peak of the banking panic] was the golden moment to put forward an alternative vision," she says. "What's been dismaying is that there has been so little coherent response from the left since." Lawson wonders whether the sheer size of the political opportunity presented by the financial crisis has induced paralysis: "All our Christmases have come at once, but we don't know what to do about it."

At Marxism 2009, the best-attended session of the morning is "Where is the radical left going?". The main speaker is Alex Callinicos, for decades now one of the key theorists in the Socialist Workers Party (SWP), the tirelessly agitating British fringe party that has organised the conference. In the airless main hall, in front of a stage backdrop reading "Capitalism Isn't Working!", Callinicos, concise and dapper in a black shirt, delivers a strikingly downbeat speech. "The forward march of the radical left in Europe has been halted," he says. "We're in a situation that is in a sense quite problematic . . . It's not a uniform picture of stagnation or retreat. The left bloc in Portugal got 10% of the vote in the European elections . . . But the ruling classes are desperately grabbing bits of Keynesianism. So a left economic policy based on Keynesianism, when Keynesianism has entered the mainstream, isn't very powerful."

This theme – that governments everywhere have borrowed the left's traditional tools for taming capitalism to deal with the financial crisis, thus stealing the left's clothes – is repeated often at the conference. It is met with looks of resignation but also grim satisfaction from the audience. The infinite deviousness of "the ruling classes" and the immense difficulty of the left's task are a given in these halls. In 2004, Solnit published a much-praised book, Hope in the Dark: the Untold History of People Power, challenging the instinctive pessimism of many leftists. "A lot of activists," she wrote, "specialise in disappointment." She adds now: "Despair is a black leather jacket that everyone looks good in. Hope is a frilly pink dress that exposes your knees."

It is quite hard to imagine Jon Cruddas in a frilly pink dress. The prominent leftwing Labour MP for the raw suburb of Dagenham in east London is all shirtsleeves and strong handshakes when we meet in Westminster. But he is one British socialist who still sees the recession as an ongoing political opportunity. Crisis on the left or not, his own trajectory seems upward: elected as an MP in 2001, he won the most first-preference votes in the Labour deputy leadership contest only six years later (Harriet Harman won via second preferences), and is spoken of by some as a potential party leader if Labour, as is quite possible, moves truly leftward after a general election defeat.

"The 15th of September 2008, the day Lehman Brothers went bust, could be the day the world turned," he begins with characteristic confidence. "The whole politics of Blair and Cameron looks like the product of more benign times." Cruddas, unlike some on the left, supported the subsequent bank bail-outs – "you couldn't let the whole system collapse" – and does not think the apparent amelioration of the financial crisis that has followed means a return to economic and political business as usual. "This is the early knockings of this crisis. You've still got trillions of pounds of debt around. The assumption in here" – he nods impatiently towards the House of Commons – "is that we tinker with this economic system, and then go back to 60 consecutive quarters of growth. But out in the country people know different. There is no economic status quo any more. There is a hunger for political ideas. I helped do an e-book on the crisis. Cost £250 to produce, put it on the web, 50,000 copies gone – bang. There is a space for a populist left politics – around [opposition to] ID cards and Trident, around taxes, tax justice – that wasn't there a year ago."

But Cruddas says people wanting this politics to crystallise will have to be patient. Rightwing ideas have been so dominant for so long in western politics and economics that they may only slowly loosen their grip. "This is going to take years. There was a long lag between the Wall Street Crash in '29 and the New Deal [the first effective left-of-centre response to it]." In the meantime, he warns, "There could be a different new form of politics, much more populist, dangerous, fascistic, like the BNP." With only the faintest hint of ostentation, Cruddas, who has a philosophy PhD, quotes part of a famous passage by the Italian Marxist thinker of the 20s and 30s Antonio Gramsci: "The old is dying and the new cannot be born; in this interregnum a great variety of morbid symptoms appear."

To less upbeat observers than Cruddas it is the left that displays "morbid symptoms". Mulgan says: "A lot of the left literature feels like it's just words, just rhetorical. [Groups such as] Compass don't feel like they're part of a real social movement. It's very different from a generation ago."

Until well into the Thatcher era, the left in Britain was a complete and vigorous political world. It had a mass membership through the unions and the Labour party. It had credibility and charismatic figures: even establishment papers such as the Times feared and sometimes respected Tony Benn or the National Union of Mineworkers. And it had potent ideas from the likes of Gramsci and Marx and Keynes. All of these elements have decayed since the 80s; but none so damagingly, especially in the light of the financial crisis, as the left's thinking about the economy.

"The left just gave up on economics," says the economist Paul Ormerod, who retains sympathy for the cause. "Marx and Keynes cast such long shadows. There was too much of the left saying, 'It's all there in the old masters.'" Marx died in 1883 and Keynes in 1946; by the 80s – some would say much earlier – the world economy had changed sufficiently to invalidate some of their ideas. Yet the left was more interested by then, Ormerod argues, in other issues such as race and gender and sexuality. Lawson agrees: "We've had a hollowed-out generation of economic thinkers."

Since the 80s, Ormerod says, rightwing economists "have taken over in treasuries and central banks all over the world". Western universities, too, have become production lines for rightwing economics graduates – and for graduates who do not even consider a complete faith in the free market to be a political position at all. Meanwhile, the left has suffered a broader crisis of confidence: as Lawson puts it, "We've had the intellectual stuffing knocked out of us – the fall of communism, the fall of postwar social democracy."

By the early 21st century, even fresh and successful leftwing books such as Naomi Klein's The Shock Doctrine or Empire by Michael Hardt and Antonio Negri assumed that global capitalism was immensely strong, "in the midst of conquering its final frontiers" in Klein's words. Most of the left, just like most other political schools of thought, did not see the great financial collapse of 2008 coming. Since the recession set in, the left has not been able to play what should have been its electoral trump card: "We told you so."

Solnit considers this picture of universal leftwing retreat too bleak. She sees signs of radicalism in Barack Obama's administration, for example on green issues. She points out that anti-globalisation and left-leaning environmental groups across the west remain energetic and creative, and that some have paid attention to economics. "I do feel like there are a lot of small alternatives out there: community agriculture, people living by barter, people living off the grid. That revolution is slow and incremental. It's been going on since the 60s. That continues." In Hope in the Dark, she criticises those who "expect . . . a punctual reaction" from the left to big political or economic events "and regard the lack of one as a failure". The way politics works, she writes, "is more complicated than cause and effect".

At Marxism 2009 there is the occasional reminder that leftwing politics still has potential. In the conference bookshop, for the most part a well-visited mausoleum of nostalgic volumes – Glorious Summer: Class Struggle in Britain 1972 – there is a brief, more forward-looking pamphlet on sale for £1. Visteon: How Workers Occupied and Won is an SWP account of the factory occupations in Northern Ireland and England this spring at the car component manufacturer Visteon. It is written in the usual overdone party style – "Now we have the template for resistance" – but suggests that the left's response to the global slump may not be completely toothless.

In March, with the motor industry in free-fall, Visteon, a spin-off of Ford, abruptly closed its UK plants and sacked all its workers. Staff reportedly received "six minutes" to clear their lockers, and redundancy terms far inferior to those they had been promised when Ford created the company nine years earlier. Kevin Nolan, a Unite union official at the Visteon factory in Enfield in north London, was one of those fired.

"I've always been a middle-of-the-road working man," he says. "I always voted Labour but I wouldn't say I was too leftwing." Yet the mass sackings radicalised him almost instantly. "I started thinking, we've got to come up with something. This was a corporation which had decided to use the recession to walk away. The initial plan was to ram a car through the main gates. Then we found a gate round the back of the factory open – no one knows the plant better than the worker – and we could just walk in." Once inside, Nolan and between a third and two-thirds of the Enfield workforce (accounts vary) blocked up the entrances to the plant with plastic crates, climbed on to the roof and fire escape, and announced that they would occupy the premises until they were offered satisfactory redundancy terms.

Nolan and many of his colleagues had never been on strike, but they made beds out of cardboard on the chilly shopfloor and dug in. Local people, some with no connection to the plant, brought them food and blankets. Members of the SWP arrived. "I said to them, 'I used to think you were a bunch of nutcases,'" says Nolan. "But they were very, very helpful." The Enfield occupation acquired a revolutionary tinge: "Don't Need Politicians, Don't Need Bosses, Workers Take Control," read one placard prominent in the TV and web coverage.

The Visteon sit-ins led to pickets of Ford dealerships and the threat of walkouts at Ford factories. In May, after less than five weeks' campaigning, the Visteon workers were granted redundancy payments close to what they had originally demanded.

Other British factory occupations have followed, most recently at the Vestas wind turbine plant on the Isle of Wight. But the ability of such well-publicised local episodes to restore a lasting momentum to the left is far from obvious. Over the last 20 years, there have been intermittent waves of leftwing militancy – the huge and vivid anti-globalisation protests of the 90s, for example – while the underlying political assumptions of Britain and similar western countries have continued to move rightwards. The modern left, its internal critics say, has become too fragmented, too utopian and divorced from how most people live. Wainwright asks: "What is the underlying social force that's going to be the basis of the left? In the mid-20th century it was the factory worker and the union member. There are far fewer of them now." Solnit says: "I don't see the networks in which great ideas circulate."

Other people think the left has just run out of ideas. "The feeling is still around that the left doesn't have any solutions," says Wilkinson. "Actually, our society is full of alternative ways of organising things" – he cites the success of the Co-operative Bank, built on ethical investments – "but the left desperately needs a developed ideology . . . an analysis of society." When capitalism had its last great crisis of confidence in the 70s, the British right had a set of remedies and a whole alternative worldview – later called Thatcherism – ready and waiting, decades in the making. Neal Lawson refers provocatively but also enviously to the early Thatcherites' political and intellectual "brilliance".

This time, perhaps the real challenge to the tottering status quo is not from the left at all. "The greens share a lot of the ideas of the left," says Mulgan, "but they are not in coalition with it, they are suspicious of it." Climate change is almost certain to make environmentalism more powerful. "The dominant sectors of the economy in 10 or 20 years' time," Mulgan predicts, will not be banking and property but "environmental services, health, education. This will be good for the left."

Maybe. Yet the left used to aim to change society rather than wait for society to change in its favour. For the bankers, who seemed to be facing near-extinction less than a year ago, the prospect of much more slowly losing their dominance over western economies to Mulgan's caring capitalists may not seem such a bad deal.

At the closing rally of Marxism 2009, with all the seats eagerly taken but the air stale as ever in the main hall, the SWP's national organiser Martin Smith interrupts his speech to read a short poem by the radical American writer Langston Hughes called Dream Deferred. It is an odd but stirring interlude, at least at first. The hall goes completely quiet; the heavyset, middle-aged Smith switches from bare-fisted rhetoric to the ambiguity – half defeatism, half defiance – of Hughes' verse: "What happens to a dream deferred?/. . . Maybe it just sags/Like a heavy load/Or does it explode?" But Smith rushes too quickly through the words and the moment is gone.