On every day that parliament has sat since September last year, Steven Bray has positioned himself across the road from Downing Street or the Palace of Westminster and shouted “STOP BREXIT!” whenever a suitable target presents itself: usually a black ministerial car ferrying a government politician between one place and the other. His voice is loud and commanding, and sometimes he stretches the syllable in the first word, “STAAWWP BREXIT!”, as he runs to the pavement’s edge waving a hybrid flag that combines the union jack with the 12 stars of Europe.

Steve is perhaps the most interesting human spectacle in Whitehall after the Queen’s Life Guard. He wears a blue top hat and a blue T-shirt decorated with the words “Stand Of Defiance European Movement”, at first sight a clumsy title until you realise that the initials make SODEM, which is how Steve feels about the Brexiteers. The surprise is that he isn’t one of them, as a 48-year-old numismatist from Port Talbot in south Wales who had an army childhood and deals in English/British coins minted since the reign of Edward VI

– and whose get-up, let’s face it, isn’t all that different to John Bull’s.

Of course, he isn’t. During a demonstration outside parliament this week, I asked him why Wales had voted to leave. “After austerity, nobody was going to vote for the government,” he said. “Every deprived area wanted to leave. They wanted hope. The injustice is that those areas are going to pay the biggest price.” As proof of the argument, he compared prosperous Cardiff (which voted 60% remain) to depressed Swansea (51.5% leave).

His life is now entirely given over to protest. Sometimes he stands shouting alone. More often he is the noisy centrepiece of a small group of anti-Brexit activists – people who, like him, devote their days to making their cause visible and theatrical. How much effect they have is hard to know – they are certainly small beer compared with organisations such as Open Britain, which has 550,000 Facebook followers, and three months ago moved into a shared office with seven other anti-Brexit groups at Millbank Tower, which is only a lobbyist’s stroll away from the Commons; or Best for Britain, to which George Soros has given £800,000. But what these little pavement groups lack in numbers, they make up for in intensity and belief, as though they were trying to forestall the question “What did you do in the war, Mummy?” as well as the United Kingdom’s departure from the EU.

Their commitment is shaming, and perhaps also slightly alarming. Earlier this month I spent a late afternoon with Gareth Steel, a retired British civil servant who spent many years in Germany and Brussels, seconded to the Foreign Office as specialist in agriculture and trade. His enthusiasm is Marcel Proust, the novelist to whom he devoted his doctoral thesis and continues to write scholarly papers on, and as we sat in the cafe in St James’s Park he tried to explain his theories about Proust’s use of chronology and time – unsuccessfully so far as I was concerned, because I have never read Proust. It struck me that our little scene perfectly caricatured the Daily Mail’s idea of the “liberal metropolitan elite”: two milksops discuss an effete French novelist over tea in central London. But then, as we walked through the arch from Horse Guards Parade to Whitehall, where Steel was joining the evening vigil, a great and thrilling change came over him. He unfurled a flag and hurried across the road, and in no time at all was jeering with the best of them at any car leaving Downing Street. Since the referendum, protest has taken up far more of his time than Proust.

Like all political movements, Steel’s group (No10Vigil) trades in souvenirs: flags, badges, T-shirts for the Proms (“Thank EU for the Music”) and blue and yellow berets for the big London march on 23 June. I liked the persuasive 50-guinea notes issued by the Imperial Bank of Brexit, which have a portrait of a top-hatted Jacob Rees-Mogg with some cod Latin beside it (Superium Pompisi Prattus) and the usual pledge to the bearer replaced with the words: “I promise to pay myself more than you.”

All were available on Tuesday at a Pies Not Lies demo outside parliament, together with jam tarts made by the Proust scholar’s wife and other baked goods. There was a decent crowd, and passing tourist buses and cars regularly honked their support. I met a couple of women from Bath – it was Bath for Europe that commissioned the berets and the Rees-Mogg notes – and a serious man from the east Midlands, who pointed out that changing the Brexit mindset in his part of the world, which includes the fiercely Europhobe towns of Boston and Spalding, was a more daunting proposition than it might be in, say, Bath. Other than a young man promoting a stunt involving dogs – a “wooferendum” – people under 30 were hard to find. This was generally noted as a worrying augury for turnout on the big march.

The gothic stonework of Augustus Pugin and Charles Barry loomed over us. Inside, only a few hours earlier, the businessman Arron Banks and his sidekick Andy Wigmore had given their evidence to the digital, culture, media and sport select committee; or rather they had produced a tour de force of condescension to an underprepared group of MPs who couldn’t work out a way of not being condescended to. The committee’s too-eager-to-please chair, Damian Collins, had succumbed from the start to their smug jocularity, which said: we are all men of the world, but we are bigger men of the world than you are.

The mood became infectious. “Fellas, hang on!” said committee member Christian Matheson at one point, before correcting himself: “Gentlemen – sorry.” “No, fellas is all right,” said Banks. So there could be no reprimand when Banks said: “I pay a shedload of tax, probably more than this entire committee put together … I’m not gonna be lectured about my business interests.” Or when he asked of the wrong person: “Are you the MP who got drunk in the House of Commons [at a karaoke evening] and harassed a woman?” Or when he put it to the committee, with no demurral on its part, that he knew they were “all remainers – all remainers, right?” who had a vested interest in trying to discredit the Brexit campaign.

Banks funded Ukip and the Leave.EU campaign fronted by Nigel Farage that focused on immigration, which in the opinion of Banks and many others was the single biggest factor in Brexit’s success. It is hard to know what he believes in, other than that politicians are owed no deference, and you may as well have a laugh.

The next day the Scottish National party staged their walkout from the chamber. Westminster seems beleaguered. The protests in the streets around it are part of a long tradition that, in their case, understands the meaning of respect. But the real parliamentary crisis – the crisis of legitimacy – started from within.

• Ian Jack is a Guardian columnist