In the Netherlands, an "orange alert" is issued when the country's rivers rise to alarming levels. The time has come to issue another kind of orange alert – one that warns about some of the negative consequences of the free movement of workers within the European Union. We need to watch out: in some places the dykes are in danger of bursting.

Most of us benefit from the free movement of workers within the EU. It is important to our economies, especially in professional occupations where one can see the outline of a European labour market emerging, and the principle is rightly seen as part of the European ideal. We do not want to see this pillar damaged through dwindling popular support. That is why we, especially on the European centre-left, must think harder about how to make it work in the interests of all our citizens, not just well educated professionals.

The right to live and work in other EU countries is one of the founding ideas in the 1957 Treaty of Rome. But until the mid-2000s it was rarely taken advantage of; in the year 2000 only about 0.1 per cent of EU citizens moved to another EU country.

Download the new Independent Premium app Sharing the full story, not just the headlines

That changed in 2004 when the UK, Sweden and Ireland waived the seven-year transitional period and allowed immediate access to their labour markets for the new member states in central and eastern Europe. The effect, especially in the UK, was rather dramatic with about 1.5m people arriving in the UK from those countries in the following six years. Since 2011, all the other EU states have opened up too, with further significant flows from central and eastern Europe into countries including Germany and the Netherlands.

In retrospect, not enough thought was given to the scale of the flows. Up until the mid-2000s very few people took advantage of free movement because the economic levels of different EU countries were similar. Yet with the accession of the central and eastern European countries in the mid-2000s, a bloc of countries joined the EU (combined population around 80m) with income per head of only around a quarter of the richer EU states.

This has created a big incentive to move, at least temporarily, especially for those in lower skill jobs. And this has had a disruptive effect on some of our poorer and less well educated citizens in the richer EU states like the UK and the Netherlands. They are competing against people with much lower wage expectations.

In the UK, about 20 per cent of all low-skill workers are born outside the country and certain low-wage sectors such as hospitality and food manufacturing are heavily dominated by people from poorer EU countries. In the Netherlands, workers from central and eastern Europe make up 12 per cent of all employees in agriculture and horticulture.

We need a new settlement which is fair both to the people of the sending countries and the receiving ones. And we need to stamp out abuse. Workers from poorer EU countries are sometimes taken advantage of by unscrupulous employers who win a competitive advantage over those who play by the rules. Too often workers receive low wages, work long hours and sometimes pay high rents for terrible accommodation.

The Netherlands is already taking a tougher line, imposing higher fines on unscrupulous companies and appointing inspectors who target fraud and rogue employment agencies. But we need to do this together, within the EU.

Even when the system is not being overtly abused there is some displacement and competition that is considered unfair, especially when unemployment is high. Some of our weakest citizens are losing out in the labour market to better equipped outsiders. It is important to think about how we can protect the labour market situation of these vulnerable groups.

It is wrong to dismiss the complaints of those affected as the usual gripes about "foreigners". Even if such complaints are often exaggerated, we must nevertheless take them seriously; if we don't, they will fuel xenophobia.

In continental Europe, countries warn each other when their river levels rise. For the Netherlands, that's very reassuring. We're able to take timely safety measures and avoid undue disruption.

That is the thinking behind this Anglo-Dutch warning too. So while free movement is a cornerstone of the EU, as our experience of it grows we must be alert to the side effects and ready to be flexible in our response. We must not be blind to the fact that the EU approaching 2014 is different from the EU of years past.

There is a lack of urgency in Brussels on this question, which is why we strongly urge our European colleagues to put the downsides of the free movement of workers high on the agenda and tackle this issue together. If we wish to keep enjoying the benefits of free movement, we must be prepared to combat its negative side effects. This is in the interest of every EU citizen.

Correction: The original version of this article incorrectly stated that “in the year 2000 only about 0.1 per cent of EU citizens lived and worked in another EU country”. It has now been changed to: “in the year 2000 only about 0.1 per cent of EU citizens moved to another EU country”.

David Goodhart is director of Demos. Lodewijk Asscher is Deputy Prime Minister of the Netherlands and Social Affairs and Employment Minister. He represents the Labour Party in coalition