Frank was standing on the Wetherspoons steps watching the scene in the square by Portsmouth town hall. The big red Brexit facts bus was nearing the end of a round-Britain tour that was scheduled to sweep into more than 30 towns and cities across England, Wales and Scotland – from the remain borough of Camden in London (75% in) to Brexit Middlesbrough (65% out).

What did Frank think of the bus campaign? He observed its red message: “Brexit to cost £2,000m a week, says government’s own report. Is it worth it?” There were speeches, leaflets warning of Brexit penalties, and hecklers shouting “Disgrace!” and “You’re undemocratic!”

Polls, focus groups and citizens’ assemblies suggest people like Frank may decide the future. A long-retired army sergeant, he voted Brexit but says: “People would vote to stay in if there was a vote now … We were naive. Nothing’s going to change, is it? Immigrants won’t go home, will they? No one will spend more on the NHS and housing. They never told us it would be so bloody difficult.”

Frank probably follows the tortuous trade-offs between jobs, borders and sovereignty more closely than many voters, and he’s with the YouGov poll showing that 64% think it’s going badly. For the last nine months polls show opinion has switched to 52% for remain (from 48% at the referendum), says YouGov’s Peter Kellner.

Frank is no convert to the glorious EU idea – just a grudging realist, and angry as hell. If leaving doesn’t happen as promised, or if a needs-must vote keeps us in, there will be many like him, and the aftermath will be a wretchedly disillusioned and dispirited country.

The “Is it worth it?” bus has done its best to lift spirits. Crowdfunded from 661 small donors, this small, unaffiliated, unpaid group taking facts and argument to every corner of Britain warns that leaving will cost 10 times the £200m a week we pay Brussels. But local press and TV are more interested in the stories told at each stop along the way: local nurses and doctors tell of vacancies because of EU staff fleeing Brexit; a Liverpool businessman, another in Durham, and a High Wycombe farmer tell how Brexit threatens them. Real-life stories have more influence than project fear stats.

What swings opinion? Not much yet, say pollsters: not the welter of Brexit speeches last week from the prime minister and two of her No 10 predecessors, or the opposition leader’s policy shift (though polling hints Labour would get a strong swing if it opposed Brexit). The clash of two Tory nearly-PMs had William Hague warning his party rebels that “[John] McDonnell would do incalculably more damage to the British economy than Brexit could ever do”, while Michael Heseltine on Radio 4 said the opposite: he praised “very brave” rebels who like him “feel so strongly about the European issue that they’d rather risk the short-term damage of a Corbyn government – and let’s not underestimate that – than see Britain make this calamitous mistake of leaving Europe”. Lord Tebbit and the Bow group call for Heseltine’s expulsion. The Tory civil war might be enjoyable, but the outcome matters too much.

A local Brexit bankruptcy or the flight of a company, with jobs shifting to the EU, is, say pollsters, most likely to cause voters to change their mind. But the muted voice of business during and since the referendum has been shameful: too many as ever, timidly reluctant to offend their government. Broadcasters despair of finding captains of industry ready to put their heads above the parapet, however appalled most companies are at the consequences of Brexit for their business. James Dyson, and the leavers whose own enterprises are less affected by EU trade, make too much of the running. But one barnstorming speech last week that went largely unreported ought to rouse business from its torpor.

The likes of Frank in Portsmouth may be more influenced by the direct hit on employees than anything from Westminster

At Mansion House – a day before Theresa May – the CBI president, Paul Drechsler, gave the assembled business leaders a mighty walloping. He listed hair-raising examples that he had been told of privately by distraught businesses, of their risk of leaving, failing or moving jobs to the EU: one case of 600 jobs to Poland; a Newcastle tech firm heading to the European mainland; and in all some “tens of thousands of jobs at risk”. He exhorted chief executives to start telling Britain the truth. “You’re leaders. You can make a difference. It’s time to speak up. Tell your story. The real risk is to say nothing – and reap the blame later for our silence now.”

His message to politicians was: “Listen to the facts, be guided by the evidence, and don’t sacrifice young people’s futures on the altar of ideology.” In decades to come people would “remember what business did when they needed us to act”. Here was a call to arms business needs to heed – for its own and everyone’s sake. Drechsler had a roll of honour of the few who do dare speak out – and by implication there will be a roll of shame for all who stay silent. We shall see who lets fear of Jeremy Corbyn – and his tax-take from their pockets – outweigh their duty to their companies and country.

Trust in politicians to get us out of this calamity is at a low ebb. Business has its own well-earned trust problems. But the likes of Frank in Portsmouth may be more influenced by what companies say of the direct hit on employees than they are by anything that comes from the Westminster cacophony.

• Polly Toynbee is a Guardian columnist