Never believe that polls don't matter. They are like the lighting and soundtrack for Westminster drama. They set the mood, throw the shadows and change the meaning of words. And once more that atmosphere is changing. The Tories are up again. Labour people are despairing again. The prime minister is struggling to maintain authority and status.

In a small way, the change is reflected in the number of commentators who are once more discussing the prospect of a spring poll, perhaps coinciding with the English local and European elections. Mostly talk of a snap election reflects ministerial confidence. This time it reflects the opposite. It is being discussed because the alternative, hanging on until spring 2010, through a year of corporate failures, mass bankruptcies, fast-rising unemployment, a shrinking pound and growing political protest looks too grim to contemplate. Better amputation now than certain death later.

These days, anything's possible. Certainly the public mood is swinging from scared to vengeful. The bankers may have said sorry but it doesn't wash. After watching their Commons evidence sessions, I heard a story that sums up their problem. They did indeed have media coaching before the MPs' grilling. One coach asked one of them how much he earned. "I'm afraid I don't know," came the reply. Oh dear.

But I also watched Gordon Brown's evidence to the liaison committee this week and found myself squirming as he refused, again and again, to apologise for any failings on his part. This has become embarrassing. Yes, I know, it's the game being played by Cameron - get him to say sorry, then call on him to go. But among the many who failed, the chancellor, now prime minister, cannot step aside and blame someone else.

He was there throughout. He took credit for the good times. He made friends with the same Scottish bankers now in disgrace. The failed regulatory system was the one he set up. The Tories also went along for the ride. So did much of the media. So did many more, here and around the world. But an honest reflection on what he got wrong, and why, would perhaps be the only thing that could restore some of his reputation right now. People are very angry. They want plain speaking.

This assumes, of course, that Brown is going to stay on. But let me pass on a story doing the rounds among a few senior Labour people at Westminster. I can't say it will happen and I am sure Downing Street would deny it, with a hearty, but not necessarily convincing, laugh. But it comes from quite close to the inner core. It's worth bearing in mind.

It starts with the 2 April London G20 summit. This will be an important moment, with lots of red carpet, as leaders queue to be photographed with President Obama. But as the world stares at full-blown depression, with countries such as China and Germany under huge pressure to do more to revive the global economy, it's a lot more important than that. What will actually come out of it? Well, there's one near-certainty: agreement about the need for a new global financial regulator, whether based inside or outside the IMF.

I'm told the German chancellor, Angela Merkel, has a favourite candidate to head this new body - Gordon Brown. She is said to be quietly pushing the idea behind the scenes and getting quite a good reaction from other leaders. Obama can be won over, says my source, and even Sarkozy would be pleased to see the man he's been tussling with off the European stage.

My immediate reaction was that this was daft, not least because Brown would want to soldier on and take his case through an election, even if he thought he would be beaten. There is an almost illogical doggedness about the man and he has spent his life waiting for this time in No 10. Won't his every instinct say: soldier on and hope for vindication? Yes, goes the argument, but he isn't stupid. He reads the polls. He knows he faces a catastrophic defeat. Now it's only a matter of when. If there is a lifeboat - jump.

My second reaction was: what about Labour? Doesn't the party want Brown to take them through the election, go down fighting - if that's what happens - and then start again with a new leader? Not necessarily. If Brown stepped aside and was replaced by, say, Alan Johnson, then Labour might do better in that election. The one quality Johnson does have is authenticity - and that is what is needed right now. Labour people aren't saying they would actually win it, but think that they could limit a Tory majority, or hold them to a hung parliament. Other cabinet ministers are again taking very private soundings about their own standing, and what needs to be done.

This brings us to my third reaction - and perhaps biggest objection. After the events of the past few weeks, culminating in the Lloyds-HBOS fiasco, Brown's CV as financial regulator is - how shall we put it? - looking a little tarnished. It was meant to be a brilliant coup, keeping both banks in the private sector, limiting taxpayer exposure and showing Brown's deft use of private contacts. That isn't how it looks now.

It all adds a layer of extra sensitivity to the government's battle to defend its reputation. Brown and his ministers still insist that the economic crisis was mainly an imported problem and that, though the bail-outs have been expensive, they have so far saved the banks and prevented complete melt-down. Lessons are being learned, with changes to bank regulation, but after all, nobody else spotted the dangers either.

Up to now Labour has had the benefit of the doubt. Perhaps that should be benefit of the ignorance, because the vast majority of people struggle to understand the financial system, and therefore struggle to apportion blame. But this may be changing. There seems to be a quite dramatic slide downwards in the opinion polls and for the first time David Cameron's message that it's the fault of Brown and Labour, for not regulating better, is getting through. The personal involvement of Brown in the Lloyds-HBOS deal - and if it had to be fully rescued by the taxpayer we would be talking about one of the greatest corporate disasters in British history - comes at the worst possible time.

So there we are - a possible lurch in the plot in the final act of the drama. I stress, only possible. But Brown must be feeling dreadfully battered. Even his reserves of self-belief aren't limitless. The Labour party is asking whether it would save a few score seats with a different leader. And remember, the lighting and mood music have changed. It's a wild thought. But these are wild times.

jackie.ashley@theguardian.com