On the handsome Surrey borders of south-west London, Esher is the kind of place that comes attached to the adjective “leafy”. It sports a large village green, a Farrow & Ball paint shop and a Waitrose on its plush high street, and has more golf courses than you can shake a six-iron at.

Once known as the stockbroker belt, this verdant corridor between the city suburbs and the M25 sometimes goes under the estate agent tag of superbia, suggesting a higher class of semi-detached lifestyle.

It’s also, unsurprisingly, a Tory heartland, but after Theresa May’s proposed Brexit agreement, that heart appears in need of some urgent CPR. Esher’s MP is Dominic Raab, the former Brexit secretary, who resigned following May’s announcement last week – his predecessor, David Davis, having resigned in July.

To lose one Brexit secretary may be regarded as a misfortune, but as Lady Bracknell might have put it, to lose two looks like carelessness. “It’s a total mess,” says David Nutting, at the Moore Place golf club just outside the town. “We weren’t told the facts at the beginning, and we haven’t been told the facts all the way through. And now they’ve got themselves in a real state.”

Nutting, who used to run his own plumbing business, plays golf with a group of his fellow retirees two or three times a week. They are evenly split between Remainers and Brexiters, which reflects Esher’s divided loyalties. The town forms part of Elmbridge, the Surrey borough that voted in the 2016 European Union referendum to remain. But Raab – who’s been mentioned as a possible leadership contender, should May lose a vote of confidence – is a staunch Brexiter.

Yet for all their differing gripes, both sides at the golf club were agreed on one thing: the deal that Theresa May presented last week was not what they thought they were voting for, or against, in the referendum. If it was a choice between the current deal and staying in the EU, then Nutting, who voted to leave, would now opt for remaining.

Cliff Taylor, far right, with fellow golfers at the Moore Place golf club, Esher. Photograph: Sophia Evans/The Observer

“I feel very uncomfortable,” says Charles Warren, who is also retired but used to work in insurance. “I don’t really know what their proposals are. It just seems to be a mess.” Given that the published draft withdrawal agreement runs to 585 pages, it can’t be said that May has neglected to explain her proposals. Nonetheless it’s what’s left out – the grey areas about what the Northern Ireland “backstop” measures will involve – that concerns the golfers. “We don’t know what they are,” says Cliff Taylor, a former jockey.

Sandown Park racecourse sits on the edge of Esher, a huge expanse of parkland with towering stand that has seen some epic races down the years. But it’s a safe bet that no horse has ever been as much as a gamble as Brexit is turning out to be, particularly for May.

When Raab retained his seat at the 2017 general election with a large but reduced 23,000 majority, he rejected the idea that there was growing resentment towards the Tories. As he put it at the time: “The truth is in modern politics you hear a lot from the fringes and what gets neglected is the silent mainstream majority.”

No doubt he had his constituents in mind when he used that phrase. And at the golf club and elsewhere around town, a distinct gap does seem to be opening up between the grassroots and the leadership.

Aware of the problem, May went over the heads of her MPs direct to Conservative association chairs last Friday afternoon, which may explain why Esher’s chairman, Tim Grey, wasn’t taking my calls.

But if Raab and his fellow contenders hope that popular disappointment with May’s proposal will rebound on her, the message coming from Esher is far from encouraging. For all their frustration, few Tories at the golf club appear to want May to go. “There is no alternative,” says Warren.

When I ask what the feeling is about Boris Johnson, the clubhouse explodes in a chorus of “Oh, come on!”

Across town at the Wheatsheaf pub, the mixed message remains the same. “I’ve really lost faith in the Tory leadership,” says Sue Hill, a former detective chief superintendent with the Met. But she believed that May should stick to the job. “She has to get a backbone and get into some of these men and tell them who’s in charge. Be a bit more like Margaret. I’d be taking them one by one into a room.”

Not backing Boris: former police officers Janice Runacres and Sue Hill. Photograph: Sophia Evans/The Observer

Her friend Janice Runacres, a former forensics officer, thinks that May “needs to go”, but doesn’t believe that there’s a suitable candidate for her job. She and Hill think their local MP is too inexperienced. And again both dismiss Johnson out of hand.

“A bumbling idiot!” says Runacres.

Jacob Rees-Mogg? “Lower than a snake’s belly,” says Hill.

Both women voted to leave the EU, and neither is happy with the deal. Asked if she would have voted the same way, knowing what she knows now, Runacres says: “I’m not sure that I would.”

Hill doesn’t believe the electorate really knew what it was voting for in the referendum. “We all voted on a whim,” she says.

In her own case she says it was for personal reasons to do with border security. “There was a little girl I dealt with that was murdered by a Latvian guy that was over here and nobody knew he was here. Our borders aren’t safe. But had I known the impact that it was going to have on me, my kids, property prices – looking back, the government didn’t give us enough information.”

In reality, there was a great deal of information. It was just that a lot of it was false or hard to understand. What’s striking is that both women say they were misinformed, both feel worried about the impact of Brexit, and yet both want to get out of the EU “as soon as possible”. And neither wants a second referendum.

“This country has voted,” says Hill definitively, although she says it didn’t know what it was voting for. “You’ll destroy all faith in voting in the future. We have to go through with it, whether we made the right decision or not.” Just as oddly, given her concern about the economic implications of Brexit, she’d prefer a no-deal option to what’s on the table.

After talking at length to Raab’s “neglected silent mainstream majority”, I began to appreciate why they might be neglected. What is most notable are the glaring contradictions: May is not up to the job, but she should stick it out. Brexit has been a disaster – and we should get out of Europe as soon as possible. And my personal favourite, that being in the EU has meant, as Hill claims, that the younger generation, including her children, can’t get on the property ladder, followed by the complaint that “property is dropping like a stone round here”.

The cognitive dissonance seems endemic. However, in politics the voters are always right. And while the criticism and analysis heard in Esher may lack a certain intellectual coherence, there is an emotional discontent that is undoubtedly authentic.

Hill describes herself as a Conservative through and through. But, she says: “I’m beginning to think that our politics are trash, we’ve got no leadership and they’re all backstabbers with no integrity.”

That’s something an enemy might say. When it’s coming from a diehard supporter in the sleepy comforts of superbia, the Tories should know they are in trouble.