What was it the Queen said? “As we look for new answers in the modern age, I for one prefer the tried and tested recipes, like speaking well of each other and respecting different points of view; coming together to seek out the common ground; and never losing sight of the bigger picture.” She spoke those words last month at the centenary of the Sandringham Women’s Institute (humble occasions are no enemies of wisdom), and they are understood to refer to the social and political division caused by the Brexit vote. I remembered the gist of them last weekend as our train trundled towards Boston in Lincolnshire, a town on the other side of the Wash from the monarch’s Norfolk estate.

The division between migrants and natives long predated Brexit, and the vote was a consequence rather than a cause of it

A green flatness reached to the horizon. Roads crossed the line via level crossings, some of them still tended by handsome signal cabins. The station at Sleaford had awnings big enough to shelter bank holiday crowds changing trains for the beach at Skegness. The further we went down this branch line to the coast, the more the sense grew that the past had been greater and busier than the present – a misleading impression because Boston’s population has grown by nearly a fifth since the beginning of the century and the fields around it produce more food than they ever did. The Boston Stump, the church tower that has soared above these fields for 500 years, marked out our destination long before we arrived at it. Then a short walk took us to the market square, which must be among the most attractive in England and should be better known. Like many other pleasant aspects of Boston, it has been eclipsed by a recent political fact.

My companions were two members of a Lambeth group called More in Common, who were travelling to meet an organisation of the same name in Boston. The two places represent the furthest extremities of the 2016 referendum: Lambeth, in south London, recorded the highest remain vote (78.6%) while Boston had the highest proportion (75.6 %) of voters who supported leave. The contrast appealed to the media. The More in Common groups owe their name to the late Jo Cox MP and her words, “We are far more united and have far more in common than that which divides us”, but it was a Radio 4 programme, Two Rooms, in 2016 that inspired the idea of people from politically opposite places meeting each other and, just possibly, finding ways to heal their differences.

Since then, Lambeth has been to Boston and vice versa, and a modest scheme has been overtaken by the far more ambitious notion, so far unrealised, of a “national conversation” leading to national reconciliation. Writing in the Guardian recently, the therapist Susie Orbach pleaded for conversations that addressed what she called the country’s “mutual disturbance” – the transfer of angry feelings from leavers to remainers with the message: “You will no longer have it your way. You are going to feel threatened as we have felt threatened. You can lose your hope as we lost ours.” Another therapist, Gabrielle Rifkind, has written for Open Democracy about the need for “a concerted effort to hold a dialogue at all levels that seeks to promote greater understanding between our vastly polarised communities”. In a more despairing and divided Britain than anyone can remember, who would disagree? What remains less clear is where and how these conversations would happen. Orbach mentioned precedents that crossed socio-political boundaries in Northern Ireland and Central America; Rifkind talked of meetings in town halls and community centres where people who were trusted by their communities (“not the usual great and good”) would “elicit a collaborative dialogue to break down the posturing [that is] pervasive in too many of our current sterile conversations … ”

‘Does Boston feel it has won? Perhaps. But there is always the question of who will do the work in the fields.’ Photograph: Sean Smith/The Guardian

In Boston, that means you need to talk about the migrants, mainly from eastern Europe, who work the new system of food production introduced by polytunnels, longer growing seasons, packing plants, just-in-time deliveries and the changing demands of supermarkets. Some Bostonians will remember a time when farmers employed locally born people as seasonal workers and say, “We did it then, and when this lot go we can do it again.” But that misunderstands the nature of modern agriculture and the transformation of farms into the domestic equivalent of the colonial plantation, demanding a constantly available supply of labour prepared to work long, physically taxing shifts in bad weather, and therefore usually recruited via gangmasters and their scouts from poorer and more vulnerable populations elsewhere.

Migration accounts for Boston’s swollen population, its high rents and house prices relative to other parts of Lincolnshire, and, for a time, it led to overcrowded doctors’ surgeries and schools. Post-referendum, it seems surprising that governments cared so little about these effects. In 2010, an early act of the Cameron-Clegg coalition was to scrap a £50m fund set up under Gordon Brown’s administration to ease the impact of migration on public services, on the aspirational grounds (how laughable to read them now) that controlling immigration was the better answer, “which is why the government will reduce the level of net migration back down to … tens of thousands each year, not hundreds of thousands”.

Today, according to people I talked to in Boston, the main complaints are about the kind of street behaviour that often accompanies groups of youngish single males: public drinking followed by public urinating, discarded burger boxes, violence between themselves. A Polish woman remembered her fury when she heard a Bostonian excuse street urination as an example of cultural difference. “I said to him, ‘Do you think they teach us how to pee in the streets in Poland?’” Another woman, from Lithuania, said it would be right if the offenders were sent home. There was general agreement, among both natives and migrants, that it was important “to see a person as a person and not as a nation”. But to set against this individuation several people spoke of limits to migration. “We just couldn’t do with any more of them coming here … with HMO [house in multiple occupation] licences, you suddenly find 10 foreigners are living next door to you.”

As to dialogue, my impression after a short weekend in Boston was that More in Common had promoted understanding between the various migrant groups – Lithuanian, Latvian, Polish, Romanian, Bulgarian – as well as between them and the host community. But the division and distrust between migrants and natives long predated Brexit, and the Brexit vote in Boston was a consequence rather than a cause of it. The difference in outlook between Lambeth and Boston, between remain and leave, seemed less easily bridged. Lambeth now understood that Boston’s leave attitudes hadn’t come out of stubborn racism, but it wasn’t clear how Lambeth’s remainer views had changed Boston, if at all.

Does Boston feel it has won? Perhaps. But there is always the question of who will do the work in the fields. Are there regrets? A Polish woman suddenly blurted out that her husband, an Englishman, had voted for Brexit. “I couldn’t believe it. I said, ‘You voted Brexit and a big part of your business is trading car parts with eastern Europe. How could you do such a stupid thing?’” She blamed the influence of his parents, who’d migrated to Australia and didn’t know how things stood back home.

Her eyes moistened. I felt we were on the same side.

• Ian Jack is a Guardian columnist