Last weekend 700,000 people marched through London demanding a “people’s vote” on Theresa May’s Brexit deal – many hoping for a remain option on any future ballot paper. It was reportedly the second largest demonstration this century after the Stop the War mobilisation against the invasion of Iraq in February 2003. The chaos of the Brexit negotiations, loss of inclusion in the single market and the threat to workers’ rights guaranteed by the European Union were all lamented – but freedom of movement featured highly as a principle that should be preserved at all costs.

Freedom of movement has become something of a shibboleth where progressive, liberal politics are concerned. More than just the right of a middle-class family to travel to the Dordogne unimpeded, the defence of it forms an integral part of the liberal fightback against the increasingly racist, xenophobic and violent rhetoric employed by the right. Yet what people are fighting for is a rose-tinted and narrow manifestation of a freedom of movement that only ever really existed for (predominately) white, affluent Europeans. This was not a desire to truly open up the world to all. This was a demand to protect their rights, while ignoring the very real problems with EU immigration and asylum policy.

Migrants to Europe 'need to go home', says Czech prime minister Read more

The citizen’s rights directive (2004/38/EC) encapsulates the right to freedom of movement within the EU. It created the notion of an EU citizenship. More than just a simple right to reside based on economic status, it guaranteed the rights of EU citizens to take up residency in any state within the union for up to three months, or indefinitely if they are employed, self-employed, a student, economically self-sufficient, or a “legitimate” jobseeker.

Yet in the case of Gunars Gureckis last year, the UK government tried to deport three rough sleepers from the European Economic Area on the grounds that their lack of housing breached their treaty rights, despite at least one of them being employed. The tenacity and determination of the Home Office extended to sending deportation notices addressed and delivered to the claimants at central London toilet blocks. The government may have failed in its attempts at deportation, but the case bears testament to the precarious nature of the rights of less fortunate EU citizens.

Play Video 1:19 Migrants stranded at sea face long journey to Spanish port – video report

Beyond the well-documented horrors of the hostile environment, allowed to exist almost unimpeded under EU law, the EU as a whole has shown increasing hostility to those attempting to find refuge and solace within it. In the first two months of 2016, the number of migrants who arrived on Lesbos, a Greek island close to the Turkish coast, was double its population. Camps designed to house 2,000 refugees now hold three times that. Countries to the north of Greece closed their borders, while the EU’s response to the crisis was to negotiate a deal with Turkey to return undocumented peoples there. In essence, moving an unsightly problem beyond its borders.

In September Jean-Claude Juncker, the president of the European commission, promised 10,000 more border guards for Frontex, the European border and coastguard agency. The agency currently employs 1,800 guards. The proposal means that for the first time the EU will be spending more on its borders than on development programmes in Africa – from which many attempting to find safety within the EU originate.

Swedish student who grounded deportation flight faces prosecution Read more

In 2015, 1,015,078 people arrived in the EU. A further 3,770 people died trying to complete the journey, which equates to roughly one in every 270. So far in 2018 around 20,000 people have arrived in the EU, with 1,600 dying – around one in every 18. The EU plans to increase spending on border controls to €34.9bn (£30.9bn) in 2021-2027, up from €13bn in this period. Though the number of people seeking sanctuary in the EU is decreasing, the journey is becoming ever more perilous.

Since the Windrush scandal, it is increasingly accepted that Theresa May’s hostile environment is brutal, inhumane and wrong. But it is naive at best to hold the EU up as some kind of liberal paragon. That those campaigning for a people’s vote would unequivocally champion the EU as a bastion of progressive values is an insult to the thousands lying motionless at the bottom of the Mediterranean.

We should be under no illusion what freedom of movement means within the EU, and who it really benefits. If the progressive voice is the one fighting for retaining membership of the EU, it should also be fighting, with as much ferocity, for the deconstruction and disinvestment of Fortress Europe. Without it, the movement risks tacitly endorsing the fate of those many escaping from wars, torment, and economic and social unrest that Britain had a hand in creating.

• Ben Smoke is a freelance journalist and activist