The Lithuanian man I meet on my second day in Wisbech doesn't want to tell me his name. He has lived in this Cambridgeshire market town – the "capital of the Fens" – for nearly eight years, but his English remains staccato and stilted. He has the leathery complexion of someone who has spent long hours outdoors, and what looks like a black eye. He tells me he spent the previous night in a shelter for the homeless; today, he has stopped by the Rosmini Centre – more of which in a moment – to try to get some advice, and eat.

When he first came here, he says, he worked in the fields, picking and cutting daffodils for piece rates: £5.50 for each box of 100 bundles, which could sometimes be put together in half an hour. He lived with his girlfriend in a house shared with other people, and sent money home to his daughter; when work on local farms drew to a close, local agencies then sent him to food-processing factories. And so life went on until, six months ago, he was beaten up – by other Lithuanians, he says – and left with four broken ribs.

"They were drunk," he tells me. Unaware that he was eligible for benefits, he then lived on handouts from friends, but soon fell through the gaping cracks that often open up in migrant workers' lives. He and his girlfriend split up, and he got used to the daily rituals required to find room in the night shelter.

For many of those who arrived here in the first few years after their countries joined the EU, he says, things have got harder. "Many changes," he says. "Less jobs. Lots and lots of new guys. Now, they do English test, and agencies have choice."

Does he ever feel minded to go back to his home country? "I have nothing to do in Lithuania. I would like to stay."

Within minutes of my arrival in Wisbech, "insecurity" ceases to be any kind of abstract concept. Neither does it feel like something that can be captured solely in terms of wages and employment conditions. It is a deep condition that blurs over into relationships, families, and mental health – as well as the delicate stuff of identity and belonging. Sometimes, it manifests itself in anger and hatred that bubbles away on social media and occasionally flares into ugly life in the real world; you can also sense it in a meek, heads-down sensibility among many of those who have recently come here.

In its own way, Wisbech is a fascinating place: a once-wealthy river port whose most ambitious architecture suggests a relocated slice of Holland, where traditional English shops now sit among an expanding share owned by people from abroad, as well as the standard signs – value outlets, mainly – of lives lived in precarious circumstances. Since the EU expanded in 2004, the town's population has hugely increased and of its 30,000 people, about a third are now reckoned to be from eastern Europe – Poland and Lithuania, mainly, with a rising share from Latvia (there is also a less visible Portuguese community, more spread out across East Anglia).

Since the EU expanded in 2004, Wisbech's population has hugely increased and of its 30,000 people, about a third are now reckoned to be from eastern Europe. Photograph: Graham Turner for the Guardian

As in nearby Boston, in Lincolnshire, most of these people work in a part of the local economy that embodies a kind of frontier capitalism, rooted in agriculture and food processing, which pays base-level wages and makes exacting demands. Local recruitment agencies advertise jobs based on 12-hour shifts, "4 days on, 4 days off" – as well as extensively using zero-hours contracts. Tangled up in all this is illicit stuff that teeters into outright criminality: illegal gangmasters who recruit people in eastern Europe; the unauthorised subletting whereby houses are crammed with people living two or three – or even more – to a room; an allegedly thriving trade in bootleg alcohol and cigarettes. There are some grim aspects of recent local history: between 2010 and 2012, for example, reports of tensions between Latvians and Lithuanians, five murders of eastern European people, and headlines about an alleged "Baltic mafia".

Plenty of locals say that zero-hours, minimum-wage arrangements inevitably exclude anyone with a family to feed, and rent or a mortgage to be paid – and that they are therefore stranded in a part of the economy where the arrival of Burger King is deemed important enough to make the front page of the Fenland Citizen. Here, the recession had a particularly difficult subtext: as the local economy shrank, say some people, the effects of the downturn were often blamed on new arrivals who, in time, actually revived the town centre.

There are still concerns about the pressures that a huge expansion in population has put on schools and doctors' surgeries. I also hear whispers and worries I have picked up in other places: some people claim they have seen shops and pubs with "No English" signs; a few say they are now too scared to leave their houses after dark.

There are some signs of burgeoning integration, not least when people talk about their children, but most adults still seem to live on either side of a deep divide, embodied in some things that have to be pointed out to an outsider. The large satellite dishes affixed to some houses, for example, are to pick up eastern European TV – perhaps a sign of how technology makes separateness easier. Moreover, Wisbech's social and economic divides have now been vividly politicised: Ukip took all three local seats from the Tories in last year's county council elections, and the town now forms one of the many key parts of the party's eastern English heartland.

The Rosmini Centre sits in the south of the town, next to a Catholic church. Founded in 2007 to teach English to recently arrived Polish people, and funded by fundraising and donations, it now offers a range of advice and help to people from abroad, as well as laying on activities for the whole town. Its manager is London-born Anita Grodkiewicz, who talks me through the centre's work, and the issues with which she and her colleagues deal, on a seemingly hourly basis. "Compared to when I started here, the problems have intensified," she says. "They're more complex. And it's not that numbers are increasing; it's that people are settling and bringing their families."

Among the staff is 29-year-old Oksana Straseviciene. She grew up in a Soviet-era Lithuanian town called Elektrenai, before spending a year in Northern Ireland, returning to Lithuania, and then moving to Wisbech four years ago.

How, I wonder, does she find local people's attitudes to those who have come here? "A lot of people are still rude. They're still not very happy with what's happened … I think Wisbech really becomes [sic] a small Latvia or Lithuania, or something like that."

Oksana Straseviciene, at the Rosmini Centre in Wisbech, says she has direct experience of hostility from locals: 'They have really given our children abuse. That is the worst thing.' Photograph: Graham Turner for the Guardian

She says she understands why some are either anxious or actively hostile – and, just to underline the fact that what people say about migration often defies easy categorisation, lays some blame on people from eastern Europe. "Some of them are quite annoying," she says. "They refuse to work on purpose. They know they can work for just 24 hours [a week] and get tax credits." She puts the proportion of migrants who have that mindset at 25%. "I don't want to say that our people are bad," she says. "But some of us are."

Not surprisingly, she has direct experience of hostility from locals. "I have heard it: 'Fucking immigrants – go back to your country.' They have really given our children abuse. That is the worst thing. A lot of things happen in the park: they try and push them from the swings. They think their child is privileged because they're English. It's really upsetting.

"Some of them say we are taking all the jobs," she says. "It doesn't bother me, really. Because I think some English people are lazy." Does she have any English friends? "No."

Does she think people in the town essentially lead separate lives? A long pause. "It's really, really hard to answer this question. I would like to get English friends, but I can't go around and say, 'Oh, be my friends' … Around us, it's mostly Polish and Lithuanians."

In May last year, a large crowd of people – estimates varied from 100 to 300 – gathered in Wisbech's main park, for a "static protest" modelled on similar recent events in Boston and nearby Spalding. It had no official title, though a smattering of placards – identical to ones seen at the earlier demonstrations - highlighted what had drawn people there. One said, "Controlled immigration not open door invasion"; another read, "Save our country, save our heritage, save our jobs". By way of an alternative, the Rosmini Centre organised its own event which, its staff say, attracted 500 people from all the town's nationalities.

The chief organiser of the first protest was Sharon Jardine, a native of Sussex and a Ukip member who has established a Facebook group called Immigration Issues In Wisbech. She has lived in the Wisbech area for 20 years. Along with her fellow campaigner Karen Goude, she meets me at a town-centre hotel, where the two of them explain their view of what has happened here, making repeated reference to the eastern European people they call "EEs".

When seasonal work was done by a mixture of travellers and students from abroad, says Jardine, "they'd keep themselves to themselves; you wouldn't really see them outside work." Now, though, migrant workers are a more enduring and visible presence and, they claim, fond of public drinking, violence and urinating in the street (this is all much argued about locally: the eastern European people I speak to say that the prevalence of such behaviour is no worse than among the English population). Though other people credit eastern Europeans with keeping the town centre alive, they also reckon that the recession highlighted a deep economic imbalance: "Somehow, the EE immigrants have not suffered, and they're opening up shops."

Both are at pains to talk about their activities in terms of pragmatism and the absence of prejudice. Jardine says she has "no issue whatsoever with people coming over here who want to do better for themselves, want to raise a family, and input into our system – [but] that means no benefits, obviously."

Their local detractors point to two problems, centred on the great swaths of ugliness to be found on social media, now such an ingrained part of stories about Ukip and immigration that finding that they are the hot local topic comes as no surprise at all.

First, there is the company Jardine and her local allies keep. Among their Facebook group's administrators and the 2013 protest's organisers, for example, is a resident of Boston called Dean Everitt. He was recently revealed by the anti-racist group Hope Not Hate to have put up an array of online posts, including endorsements for the BNP and a grim response to the deaths of five Lithuanian men killed in an explosion at an illegal vodka distillery in Lincolnshire: "Well today's off to a happy note with the news there's five foreigners less in the town boom boom."

"I haven't seen that," says Jardine. "That's not my stance at all. I'll go and have a look now."

The second big issue relates to other content that Jardine seems to have endorsed, pointed out to me by someone who has access to the Immigration Issues in Wisbech page after my meeting with Jardine and Goude (the group is "closed", meaning permission has to be given for anyone to see its posts). In response to one local woman's recent allegations of antisocial behaviour by two eastern European men, another member of the group wrote: "Dirty foreign bastards all of 'em!! They all need sending back to were [sic] they belong!! A pit in the ground" – a post that Jardine not only did not delete, but responded to with a 'Like', as well as a positive reply.

Her emailed response to this centres on the woman who put up the original complaint: Jardine says this woman recently lost a son to cancer and added: "My comment liking the [subsequent] comment was purely against those that had been outside [her] home."

The task of pouring oil on Wisbech's sometimes troubled waters often falls to its established politicians – people such as Simon King, a Conservative district councillor who lost his seat on Cambridgeshire county council when Ukip so spectacularly surged last year. King's Conservatism is vicar-ish, and dutiful: he is a leading member of an organisation called We Are Wisbech, founded by people from other parties, the trade unions, and community groups, to enter the local conversation about immigration and "present the other side of the coin". He says there are many optimistic tales to tell – migrant families, he says, are helping to drive up standards in local schools – but such stories tend to get lost in an online world that has precious little interest in them. "Social media is a very new phenomenon," he says, "and it's part of the changing face of politics. Sadly, it lends itself to people who are unhappy. It's very, very hard to get positive messages across."

Simon King, a leading member of We Are Wisbech, an organisation founded to enter the local conversation about immigration and 'present the other side of the coin'. Photograph: Graham Turner for the Guardian

Two hours later, at the end of the High Street, I meet a 45-year-old man called John: born and raised in Wisbech, he says. Having worked for 25 years in the local construction trade – mainly as a bricklayer, though he could turn his hand to plenty of other stuff – he hit huge problems around five years ago: his weekly earnings tumbled from £500 to £250, and he was soon unable to find work. His marriage foundered, and he left the family home; he was prescribed antidepressants.

He is an apparently modest and level-headed man, quietly going off his head. "I can't even get agency work on the land," he says. "I tried, and they said, 'Are you fluent in Polish, Latvian, Lithuanian or Russian?' I said, 'Well, no – I'm English.' They said, 'Well, you're no good to us.' You get the same in the factories: 'You need to understand another language to get on with people.'"

Plenty of people – here and elsewhere – claim such issues are real, but endlessly denied. One insider in the agency worker business, by contrast, tells me they amount to "utter rubbish", and sticks to the same line you hear from all the people involved in his trade: that most British people are simply not interested in the kind of work done by eastern Europeans (this man has 200 people on his books; four are from the UK). John, though, is having none of that. "We've got to learn their language instead of them learning ours," he says. "It's not 'do as the Romans do', is it?" In the 1990s, he says, he voted Labour before switching to the Tories; this year, he voted for Ukip.

As we speak, the heavens open, and the streets empty. After what I've heard for two days, it feels like a merciful pause, as this most unlikely of frontier towns collectively figures out the more practical aspects of politics and economics, and how to live at the outer edge of just about everything.