Everything was, very briefly, going OK for the Labour party, and then it brought out its immigration mug. It was a simple red mug, with the words “Controls on Immigration”; it is one of the party’s five pledges. The others are either infuriatingly vague (“An NHS with time to care”) or pointlessly catch-all (“A country where the next generation can do better than the last”). But the immigration pledge is the worst, having neither any foundation in policy (bar some very mild notions of benefit-parsimony), nor any basis in politics. Plus, who would drink from that? The party doesn’t even understand what a mug is for.

This is the question that has plagued Labour for as long as I can remember: why can’t it talk about immigration? Half the time, it would have us believe that the free movement of human beings is an inherently racist topic, which no true progressive could ever discuss. The other half, the party bigwigs roll up their sleeves and bruise in, weaklings following Ukip thugs. The truth is, they stopped being able to talk about immigration the moment they stopped wanting to talk about solidarity, nationalism, compatriotism, human rights and, indeed, the fact that humanity has a value beyond money; without those things, it is really hard to talk about people. You end up talking about the rebalancing of the global labour market, while pretending that is about people. Here are the five questions that Labour needs to answer for itself before it will be able to convince anyone about freedom of movement.

What is a citizen?

This defence of immigration comes up a lot: immigrants aren’t a drain on any service or benefit, being on average fitter and younger than the indigenous population. Regarding hospitals, Adam Steventon, of the Nuffield Trust, found that, “admission rates were around half that of English-born people of the same age and sex”. Not only are they less ill, they are less encumbered, less needy – 60% less likely to claim benefits than British people – and better educated. A European Union Commission report last year underlined, also, that migrants were less likely to claim disability benefits. A 2013 University College London study [pdf] said migrants from the European Economic Area contributed £8.8bn more than they gained between 1995 and 2011 (“gained” – I can’t stand that word: it means “lived a full life, with dependants, had ups and downs”). Nobody disputes these figures; well, probably MigrationWatch does. I don’t. I respect and salute the people who compile them etc.

Not only are immigrants less likely to be ill, but they contribute far more than they ‘gain’. Photograph: Christopher Furlong/Getty Images

However, the argument as a whole tries to sell us people as units whose value can be counted in economic productivity. Simultaneously, we see the native population bagged up and measured by who is healthy and strong enough to work, and who takes which benefits – so that shadow ministers say things such as “Labour isn’t the party for people on benefits”, and the governing parties use the language of parasites and dregs to talk about people who have simply been – in work or health – unfortunate.

Citizenship, in modern British rhetoric, is conditional upon the money you bring in. The moment you are not economically productive, you are not just a non-citizen but a drain on other citizens, an anti-citizen. The truth is, it is impossible to be born and to die in a country without spending probably 40 of those years needing something, “gaining” more than you “contribute”, being economically unproductive. So pretty well all indigenous people are going to cost more than people who arrive in a country as adults.

Consider, now, that 21% of people need housing benefit (most of them in work, some not) and that one fifth of working-age people are described by the political narrative in which they live as a kind of sub-species, whom nobody in the main parties wants to be seen to represent. In these conditions, to tell people to welcome immigration because migrants are healthier and better at making money is just … well, it’s just really dumb.

Some loyalty to one another – beyond “what can I sell to you?” and “what service can you provide to me?” – is a precondition to freedom of movement, because people just won’t put up with newcomers unless they feel valued and protected. The loyalty isn’t ethnic, but it is based on place: you are allowed to care what happens to your neighbour. You are allowed to care more about what happens in Doncaster than what happens in Toulouse. The borders of your country are the borders of your democratic agency; you are allowed to want to build a society within it. You can believe, urgently, in freedom of movement between Doncaster and Toulouse, in the name of freedom and agency and fulfilment and the cross-pollination of ideas: but not on the basis that people from Toulouse are younger and fitter than people from Doncaster.

What is a nation?

“Far from being a burden on the NHS, immigrants are actually propping it up,” someone always says to rebut the “too many foreigners” argument. Migrants constitute 11% of health service staff, and 26% of doctors; add in auxiliary staff, and the health service is 40% staffed by migrants.

This is not an accident: in nursing, for example, there have been active recruitment drives for immigrant labour, partly undertaken directly by the government, partly brokered through agencies.

“It’s fairly well publicised,” Howard Catton, head of policy and international affairs at the Royal College of Nursing says, “that the NHS hasn’t planned the number of nurses particularly well over the years. We boom, we bust, we feast, we famine, we peak, we trough.” One government fails to fund nursing places, the next introduces a nursing target. So there is an urgent need for nurses, who have to be got from abroad.

I am going to assume that nobody here cares what colour their nurse is, nor what accent he or she has. This is, nevertheless, a senseless argument for immigration. Not because it denudes the nursing population of other countries (though it does) but because it does nothing to create a good society. We pay through the nose for agencies to recruit nurses from abroad and then complain about a “low-skilled” British workforce, which, in reality, is a lot of people (83% of them women) in their 30s and 40s, earning the minimum wage as carers, because there was no money to train them as more highly skilled nurses in 1995. What kind of a society comprehends looking after its ill people, yet cannot wrap its collective head around looking after its young?

I was on Pienaar’s Politics on Radio 5 Live with Nigel Farage. It was a Sunday morning in January, right after Amjad Bashir defected from Ukip to the Conservatives, so I thought he might look terrible, or be crying. He was utterly unperturbed. He said his policy would be to make nursing and medical degrees free for British students. On the radio, thank God, nobody can see you nod. But I do agree with him: the difference between us is that I think all higher education, not just that with an obvious social purpose, should be free. This is the question: do you want to educate the nation’s young at the nation’s expense? Do you conceive the nation’s wealth to be its people? If so, that’s nationalism. A party that’s afraid of nationalism finds it pretty well impossible to make a coherent case for free movement.

We’re told migrants do the work British people won’t do, but how fair is that on British workers? Photograph: Christopher Furlong/Getty Images

What is a fair wage?

Immigrants do not bring down wages but behind that broad truth is a more complicated effect on the labour market. They are good for the economy because they create a need for goods and services. Studies admit a small effect – a 0.5% drop in wages for every 1% rise in the share of migrants – in low-wage sectors. In a sector with a large share of migrant labour – care, for instance, with 20% – that “small effect” will be pretty big.

Time and again, in industries where there is significant foreign labour, reports tell us that this is work “British people won’t do”, as the Migration Advisory Committee concluded in a 2013 report that British people would not do live-in seasonal agricultural work [pdf], and the Low Pay Commission said that British people wouldn’t do domiciliary care in London [pdf]. Whenever anyone ascribes some inherent characteristic – of sloth or unwillingness – to an entire race, even if it is your own, you should smell a rat. British people will do anything; but people are often reluctant to do work if they cannot live on the wages, and this drives the wages up, unless there is someone prepared to do it for less to escape a harsher economic reality elsewhere. In other words, anybody who can say in one breath “immigration doesn’t bring down wages” and, in the next, “this is work British people won’t do” is refusing to connect their own dots.

The insightful economist Jonathan Portes once compared immigration to coal mining. It was good for the country’s economy that the mines were closed down, even if it destroyed the communities that relied upon the mines. Well, OK, great: but you have to go back for those communities. There is no future for the country that doesn’t include them.

What is a voter?

That brings us neatly to the classic line that “anti-immigrant feeling is strongest where there aren’t many immigrants”. This is meant to be the killer blow, because it means that xenophobia is just born of ignorance, and not of anything real. The south-west is famous for this kind of thing – high Ukip support, low numbers of immigrants, very poor data collection. The Migration Observatory once pointed out that east Devon had an estimated 4,000 immigrants, a figure with a 4,000 margin of error. So it may have had 8,000, or none.

I talked a few times to Su Maddock, the prospective Labour candidate for Torbay. She was always out leafleting. I found it comforting, talking to her on the phone, with her always breaking off to tell someone: “Number 53 has a dog!” This is politics as I know it from my childhood, the constant dissemination of colourful bits of paper that nobody will read, like the doomed US plan for Pyongyang.

“Most of the dissatisfaction is with Whitehall,” she said. “People are very disenchanted with politicians generally. The people in the grammar schools are highly stressed because they’re all trying to get out [of the region]. The people who aren’t go into very low-waged seasonal work, or into care, so it’s a real low-wage economy with no strategy to do anything about it.”

Seaside towns tend to share these traits because many are remote and un-commutable, and they all feel left behind by London. Ridiculing these voters because they have the temerity to support an anti-immigration party, yet don’t have any immigration, is a wilful misunderstanding of what that support is really about. If your plan for growth is all housing bubble and financial sector, and leaves behind half the country, that isn’t a plan: that’s a slow-motion democratic catastrophe.

The treatment of asylum seekers at Yarl’s Wood shames a nation that proudly signed up to the Refugee Convention in 1951. Photograph: David Levene for the Guardian

What is the point of human rights?

Austerity as we know it – where basic human decency is sacrificed to solutions that purport to be cheaper but turn out not to be – actually started before the austerity narrative. It started with New Labour’s treatment of refugees. Over the past 20 years, this has become more and more hostile; the women at Yarl’s Wood are now treated as prisoners, except without the physical protection and decency for which the prison service is, rightly, known. “Nothing in my 30 years’ experience as a children’s doctor prepared me for Yarl’s Wood,” wrote Professor Sir Al Aynsley-Green in this paper, going on to describe situations that sound as if they come from a time before the Human Rights Act, outside a democracy: kids who had been in the UK since they were babies, grabbed on their way to school, sitting in cells in their Asda school uniforms months later, worrying that they hadn’t said goodbye to their friends. Some of this sheer, blank cruelty is simply what happens when you hand over the duties of the state to private companies; but at the root of it is this instrumental view of a human being which determines the way we talk about immigration. How much are you worth? Rich people who come in on an investor visa are effectively paid to be here. People who come with nothing are treated as criminals without rights, which is to say, less than criminals; less than human.

We aren’t a soft-touch for asylum seekers. In fact, we take just over half – 0.47 per thousand – the European average of 0.91 per thousand. We take asylum seekers in tiny numbers; in 2013, it was just over 20,000. But it is nothing to be proud of.

When we signed the Refugee Convention in 1951, we did so willingly. Nobody had a gun to our heads. We did it because that was the country we wanted to be, a haven, somewhere generous and stable. There is a peculiar pridelessness in the current Britain, a listless indifference to the morals it holds and represents. Its progressive wing fears the conversation about “British values”, when it should be grabbing it with both hands, saying, yes, those were great. It was brilliant when we had values.

Far from being the thing the left can’t talk about, immigration should be the thing the left wants to talk about the most.

Get It Together: Why We Deserve Better Politics by Zoe Williams is published by Hutchinson on 2 April at £14.99 rrp. Buy it for £11.99, including free UK p&p, at bookshop.theguardian.com

Join Zoe Williams at Foyles, London WC2, on 7 April for a discussion about the state of British politics. Tickets from foyles.co.uk

• This article was amended on 8 April 2015. An earlier version said a 2013 University College London study said migrants contributed £8.8bn more than they gained between 1995 and 2011. That related only to immigrants from the European Economic Area