The conference hall may be one-third full, but the real action at the Conservative party conference is on the fringes. While Boris Johnson is yet to arrive, there were queues half an hour long to get in to see Jacob Rees-Mogg discuss Brexit, and elsewhere, delegates crammed into meetings to hear Priti Patel denounce her colleagues in government.

Rees-Mogg told an audience largely composed of his supporters in a packed morning fringe event at the ICC in Birmingham that Brexiters had to get away from the “Ukipisation” of the vote to leave the EU in order to win back the support of young people, and tried to reframe the issue as a question of sovereignty.

“I think there’s lots of failures of propaganda really that we have allowed Brexit to be about immigration or putting up barriers or not liking going on holiday in Europe. It’s none of that. It’s about who runs your government, do you or somebody else,” he said at an event where dozens of frustrated delegates found they could not get in if they did not arrive well in advance.

The chairman of the European Research Group was also not afraid to tell those who had made it into the room that Theresa May was struggling to connect with voters on other domestic concerns – a sign that even if the prime minister steers Brexit through parliament, her leadership position will remain precarious. “What resonates with people is when you say ‘we will make your lives a little bit better’, and frankly we are not saying that at the moment,” Rees-Mogg said.

At a separate TaxPayers’ Alliance event, Patel, the former international development secretary, attacked her erstwhile cabinet colleagues for failing to apply the capitalist values of Margaret Thatcher to government as Britain leaves the EU – a reference that was always going to please the faithful in the room.

“I am really bored of Conservative politicians [speaking] about their love of the free markets, particularly those who are in government who have done nothing to change government policy,” she said.

“The last time when the Conservative party was truly on the side of business was when we had exceptional leadership under Margaret Thatcher. If you have the privilege of being in government, you step up and apply Conservative values.”

Other hard Brexit events were less successful, however. The highlight of a chaotic and poorly attended Leave Means Leave lunchtime event in the city centre was a standoff between the pro-Brexit campaign group’s co-chairman, Richard Tice, and a noisy group of remain-supporting demonstrators wearing blue T-shirts bearing the EU flag.

Polly Ernest from the Stand of Defiance European Movement (Sdem) claimed Tice’s hands were shaking when challenged and said he accused them of trying to support “a losers’ vote” in the form of a second referendum.

Only a handful of journalists attended, despite the offer of free alcohol from the “Wetherspoons non-EU drinks menu” and promise of a short bus tour around Birmingham, although that was not helped by the fact the PR agency involved only came onboard at the last minute. There was no sign of Tim Martin, the JD Wetherspoon chief executive, who had billed as attending but was not able to turn up until 4pm.

JD Wetherspoon’s Tim Martin has been a strong supporter of Britain leaving the EU. Photograph: Ben Stansall/AFP/Getty Images

Despite the setback, Leave Means Leave plans to hold a string of rallies around the country in the coming weeks, including one on Saturday in Torquay, where Nigel Farage and Rees-Mogg, despite the earlier warning of Ukipisation, will share a stage. Organisers hope 1,800 people will attend. Tice said it amounted to a “save Brexit tour” in opposition to May’s Chequers proposals, “to focus on the message that no deal is no problem”.

Meanwhile, the small number of Conservative second referendum campaigners held their own fringe event across the road from Leave Means Leave. It was an event the party had refused to list in its official fringe guide and the organisers of which – from Best for Britain – it had denied conference passes to, but dozens attended to hear the party’s remainer wing express itself.

The former minister Phillip Lee told the meeting of Conservatives for a People’s Vote that Tory MPs were under “huge pressure” from activists not to speak openly of their concerns about EU withdrawal. He claimed three other serving ministers – who he did not name – privately agreed with him.

Justine Greening, a former cabinet minister, told those present that none of the suggested Brexit options – Theresa May’s Chequers plan, Boris Johnson’s Canada-style trade agreement, or no deal – could win a majority in the House of Commons.

“We are going to reach stalemate … I think there is a growing band of Conservative MPs who have thought their way through this and reached the logical conclusion that a referendum is the only way we can unblock parliament’s stalemate,” Greening said.