A tide of humanity — people on the run from Africa, the Middle East and Asia — is washing over Europe’s boundaries, and many don’t survive the journey.

The plight of migrants only grabs the media spotlight intermittently. It usually takes a disaster involving death in large numbers to drag this story into the public eye, although in fact it is a story that never ends.

And the inevitable deaths are ongoing regardless of the attention or inattention they receive.

For example, in early February 2014, at least 15 migrants drowned as hundreds tried to reach the Spanish enclave of Ceuta from Morocco. And one of the most wretched recent incidents, in October 2013, involved the death of over 360 people off the coast of the Italian island of Lampedusa.

Tragedy on this scale demands, and generally gets, the attention of the broad public and political leaders. During his visit to Lampedusa after the October 2013 disaster, José Manuel Barroso, President of the European Commission, proclaimed, “The European Union cannot accept that thousands of people die at its borders.” Malta's Prime Minister Joseph Muscat also issued a statement after the tragedy, warning that the Mediterranean risked turning into a “cemetery” for desperate migrants.

These well-publicised incidents, and the bursts of engagement from politicians and officials that follow, highlight the conflict at the heart of Europe's asylum and migration policies: on the one hand, the authorities acknowledge the humanitarian imperative of saving lives at Europe’s borders while at the same time they vigorously apply politically driven policies that restrict migration. In the end, the dynamics that put so many lives at risk remain unaltered.

Deaths and disappearances: the known and the unknown

Catastrophes grab the headlines; smaller incidents often go unreported. Any reliable overview of migrants’ deaths has been difficult if not impossible to achieve, until now. Thanks to The Migrants' Files, compiled by a consortium of more than 10 European journalists, the most comprehensive and rigorous database on migrant fatalities ever assembled is now up and running. Their assessment is shocking: since the beginning of the century more than 23,000 people have died or vanished attempting to enter Europe.

Their approach was to use “open-source intelligence” (OSINT), a method originated by the intelligence services, acquiring data from publicly available resources such as news media, public data or grey literature. This material has now been collected, screened, cross-checked, analysed and registered in one database.

The main data sources for The Migrants' Files are United for Intercultural Action, a non-profit which coordinates a network of over 550 organisations across Europe, as well as Fortress Europe, founded by the journalist and author Gabriele Del Grande, which monitors the deaths and disappearances of migrants to Europe. The Migrants' Files' database also uses data from Puls, a project run by the University of Helsinki, Finland and commissioned by the Joint Research Center of the European Commission.

The findings of The Migrants' Files data mining have been eye-opening. The number of fatalities suffered by migrants on their journey to Europe turns out to be much higher than previously believed. Earlier estimates had ranged from 17,000 to 19,000 victims since the early 1990s. However, the estimated number including carefully assessed undocumented cases is indisputably higher.

Shifting routes make data more difficult to collect

The Migrants' Files data indicates that migration flows vary between sea and land routes according to season, flaring or ebbing depending on local conflicts and war zones, as well as the preferences of human traffickers. In recent years, the European Union has concluded a series of bilateral agreements with various north and west African countries and has undertaken several measures to tighten border security along Europe’s borders. As these measures came on stream, the routes taken by migrants shifted from Spain, to Italy and then Greece, underscoring the dynamic, adaptable nature of migrant flows.

Frontex is the European Union’s border security coordination system. Its Operation Poseidon aims at tightening border controls between Greece and Turkey. In addition, on its own, Greece added some 1800 police officers to patrol its border with Turkey in response to the rising flood of migrants attempting to cross that border in the period from 2008 to 2012.

The number of migrants observed travelling overland from Turkey into Greece subsequently fell from more than 55,500 in 2011 to just over 12,000 in 2013. Meanwhile, the sea route between these two eastern Mediterranean neighbours saw an increase in detected emigrants from less than 1,500 to over 11,000 in the same period, an eightfold increase.

At the moment, a growing number of migrants are making their way to Europe via the Greek islands and, once again, Italy. Since the land route from the Horn of Africa through Sinai into Israel has been cut, the sea journey between Libya and Lampedusa has lately returned to favour among the traffickers, who claim to arrange passage for paying migrants.

As a jumping-off point, Libya is currently one of the main hubs for migrants wishing to enter Europe. Libya’s lack of effective law enforcement, reflecting the country’s growing power vacuum, only makes the situation of migrants more precarious.

Data from Frontex backs those findings but Frontex does not track dead or missing migrants, nor does Eurosur, the European Border Surveillance System or the International Organisation for Migration (IOM). Data on dead and missing migrants risks being lost. Using open-source intelligence and carefully sifting other relevant information sources, The Migrants' Files is committed to supplying this vital information.

The Migrants' Files is a project by datajournalism agencies Journalism++ SAS, Journalism++ Stockholm and Dataninja ; media outlets Neue Zürcher Zeitung, El Confidencial, Sydsvenskan and Radiobubble as well as freelance journalists Jean-Marc Manach and Jacopo Ottaviani. The project is partially financed by JournalismFund.eu.