Here they come: swarming and noisy, dirty and barely civilised – stories about the Roma. Talk-radio gobmeisters, custodians of mass-market morality belching out bellicose headlines and editorials, on-the-make politicos eager to show they're in touch with what the public is "really" thinking: in recent months they've all united in a chorus of hate aimed at Romanians or Bulgarians who have the temerity to step foot in Britain. Their accusations are as horrendous as they are hackneyed: the Roma are pickpockets, baby thieves, spongers, beggars, tax-shirkers, doorstep-defecators. Worst of all, they bring down house values.

On the periphery even of the liberal imagination, Romani people have long oscillated between being invisibles and folk devils. Hardly homogeneous (those living in present-day Britain are a mixed bunch, including Irish Travellers, Hungarian Romani who arrived after the failed uprising in 1956, and the Kale of north Wales), they have been labelled and relabelled ever since they migrated from India in the middle ages: Saracens, Bohemians, Egyptians, all terms signifying vagabond otherness, dangerous mobility. As early as 1496, the legislative council of the holy Roman empire, meeting in Bavaria, declared them "enemy agents".

Yaron Matras's I Met Lucky People, a historical and linguistic survey of the Roma, tells how many European territories soon followed suit. In 1530, King Henry VIII passed an Act expelling "outlandish people" prone to palm-reading and fortune-telling. In 1541 they were blamed for fires in Prague and kicked out of Czech lands. In 1548, the German diet in Augsburg ruled that their murder was not a prosecutable offence. By the early 18th century, those living in parts of Austria might have their backs branded.

Time and again over the last millennium, Roma movement has been more than purely cultural; it is as likely to be political, a series of displacements and exoduses triggered by prejudice and social panics. Like the Jews, another group repeatedly castigated and cast out through history, the Roma were targeted by Nazis. They were sterilised and, during a period they refer to as the "Great Devouring", at least 100,000 (some estimates claim 1.5 million) were exterminated. Matras calls this "genocide", but also points out that not one German implicated in it has been brought to justice.

Relatively little scholarship on these people exist. No doubt this has something to do with stories often being passed down orally rather than textually. But given how consistently harassed and persecuted the Roma have been, that they've been described as "the first blacks in Europe", and to this day use words such as "pani" (water) and "sap" (snake) that are identical to similar words in Hindi and Punjabi, it's surprising that they're so little studied.

Perhaps it's because, in some eyes, they can pass for white, and therefore aren't the "right" kind of minority. Certainly, they don't easily fit within the schema of postcolonial analysis; their history precedes that of European imperialism, and the polemical claim "We're here, because you were there," often used by pro-immigrant activists, has little force. Besides, many Roma, if they're not living in rural landscapes, are found in edgelands that are terra incognita to many scholars.

I Met Lucky People is expository, describing its subjects in almost anthropological prose. At its best, though, it homes in on the possibility that the Roma problem has nothing to do with the Roma, but with the "paradigmatic dilemma" they raise: "The identity of the Roms as a people is difficult to conceptualise if one's understanding of nationhood is bound to territory, national sovereignty and formal institutions. We tend to think of a nation as having a land, a state, an official language and a recorded history."

Matras rightly debunks the idea that the Roma are in constant motion; some settlements go back decades, while it's often pushy local authorities, or grand projects such as the 2012 Olympics, that perturb and unanchor inhabitants. No doubt he'd agree that it's odd, if not downright hypocritical, for them to be so routinely talked about in the language of pandemics or military conquest ("swarms", "invasions"), or that they should be damned for their perceived nomadism when non-Roma society fetishises speed and circulation, weekend breaks and EasyJet getaways.

No less strange is how European society discusses the Roma relationship to work. Often it's assumed there is no relation: they're portrayed as being indolent and like "chavs" (a Romani word). Specific histories – such as the number of Roma who served as snipers for Britain during the first world war or who laboured in munitions factories during the second world war – are conveniently overlooked. At the same time, the extravagant and voyeuristic attention given to the tiny number of Roma families who have lavish mansions feeds suspicions that "they" are getting rich at "our" expense.

Matras points out that most Roma live below the poverty line. This may be the price they pay for cleaving, in his words, "to independence and flexibility in earning their livelihood". There's not a vast amount of money to be made in hawking carpets, shifting deckchairs, repairing roofs, selling trinkets. Working seasonally rather than for steady wages is tough, but it's neither illogical nor reprehensible. It may even be preferable to the forms of psychic servitude and clocked-on torment that capitalism inflicts on so many people.

Perhaps on one level we suspect that already. Matras draws attention to the many novels (including ones by DH Lawrence and Virginia Woolf) that celebrate the Roma for their perceived transgressions, as well as the hundreds of songs – by jazzbos, metal bands and folk chanteuses – that, however hokey and sentimental, fantasise about exercising Gypsy freedoms.

He might also have mentioned Ewan MacColl who, when speaking of his experiences recording The Travelling People (1964), the final episode in his widely revered Radio Ballads series, recalled: "There were occasions when it was easy to imagine that one had slipped through a time-warp and was in a place where automobiles and television and supermarket were still in the future. To sit in the dark in a bow-tent made by throwing a tarpaulin over bent sycamore saplings and to listen to a disembodied voice telling the story of the firebird was to experience a special kind of enchantment."

To some that kind of reverie is as worthy of scorn as situationist Guy Debord championing the Roma as fork‑tongued rebels, rhetorical refuseniks whose dialects helped them avoid capture by straight society: "The Gypsies rightly contend that one is never obliged to speak the truth except in one's own language; in the enemy's language the lie must reign."

But both MacColl and Debord are, in their different ways, expressing their shared belief that what gets called civilisation all too often feels like enclosure. Perhaps today's hatred of the Roma springs mostly from hatred of ourselves – our complicity in a deformed social system that few of us can imagine opting out of.