Somewhere in the warm plains of Thrace, where humanity’s first vines were planted, is a spot equidistant from the Aegean, the Black and the Marmara seas. It is off the beaten track and may not trip off the tongue, but Kapitan Andreevo-Kapikule is the world’s busiest land border crossing. Here, the west supposedly becomes the east, the European Union officially ends and something else begins that is – well, still Europe, geographically at least. Borders are ambiguous places. The post-communist wreck of a silk factory on one side is mirrored by a shiny mosque on the other. When I was growing up in Bulgaria in the 1970s and 80s, this entire region was heavily militarised and out of bounds for civilians, except for those who lived here. If you approached the border zone without a special permit, you could be shot, no questions asked.

Breezing through with my EU passport, I glimpsed a scene that was typical in its simplicity. Only the roles – trespasser, gatekeeper, intermediary, witness – had been reshuffled by history. Once an inmate, I was now a witness. The direction of trouble had changed too: once you were in trouble when heading south (out of the Soviet bloc), now when heading north (into the EU). But one thing hadn’t changed – the double standard of the border: once soft for some and hard for others, now hard for some and soft for others.

Using sensors to detect human beings inside lorries, Bulgarian border police had intercepted a group of Kurdish refugees. They looked as doomed as the Turkish lorry driver who was protesting his (possibly real) ignorance, although his face was drained of hope.

“Borders,” said my travelling companion, as we drove on to Turkish soil. “Have you heard of a more pointless invention? They only make it harder for people to do the things they need to do. And costlier.” The Orthodox crucifix hanging from his dashboard mingled with a blue Turkish evil eye, because Ventsi is a man of the border – that is, a borderless being. These days he is the owner of a hotel in the last Bulgarian town before the border, where once mulberry trees, sesame oil and watermelons fed the economy, and now casinos feed the addictions of the three border nations. But as a young conscript, he’d had to serve this border with his body. Ventsi and I are the last generation of Europeans to have come of age behind a hard border.

With the Brexit negotiations under way, there is one view of borders that encourages us to regress to the 20th century. It is an inward-looking view, intent on stopping the barbarians at the gates. It is understandable that borders are creeping into fashion again, lubricated by the passions of various new nationalisms that aren’t at all new – in the same way that, once in the system, viruses aren’t new, just dormant or rampant depending on the general health of the patient. It is understandable because there is an overwhelming sense of fear in many – and a border appears to be “a kind of solution” (the Greek poet CP Cavafy’s “Waiting for the Barbarians” , from which this quote comes, is perennially fresh). But it may be useful to recall that, until 1990, for half of Europe’s people, borders were a trauma. The iron curtain was more than a figure of speech. It cut into flesh, into families, into the lives of the unborn.

Facebook Twitter Pinterest ‘It cut into flesh, into families, into the lives of the unborn’ … masons constructing the Berlin Wall in December 1961. Photograph: Volkmar/K Wentzel/National Geographic/Getty Images

These two species of border knowledge – the soft cuddly one that western democracies have enjoyed since the end of the second world war and the hard, barbed one that some of the rest of the world knows – are indivisible, even if they rarely meet. Borders have always been with us, they have always served a purpose, and they have always failed, ultimately, though their failures have been so varied, so profound and so complex as to be confused, at times, with efficiency. With their yearning for absolute certainty, for black and white, us and them, here and there, hard borders end up casting a long shadow of doubt and duality, paradox and paranoia. If you live long enough in the corridor of distorted mirrors that is a border zone, you end up seeing your likeness in the image of your neighbour. Sooner or later, you end up meeting yourself. Which makes it all the harder to tell exactly where the barbarians are: pushing at the gates, already among us, or inside our heads.

There’s a story I heard about a Turkish man from a border village in 1970. He went to the border river to water his horse. He saw a shepherd on the Bulgarian side, and waved. The shepherd waved back, silently. It was a fine day, the man with the horse was in good spirits, he shouted “Merhaba!”, “Hello!” A Turkish border patrol overheard him. He was taken away in an army truck and sentenced to 14 years in jail, for espionage. By his own side. The same, and much worse, happened on the Bulgarian side. Thousands of eastern Europeans attempted to cross the electrified wall of this border between 1961 and 1989. Some survived. Some even succeeded. It was their stories, and the stories of those who didn’t, that I came to unearth.

In no man’s land, not far from where the iron curtain of our childhood had arrested the forest, rises the new wall between Bulgaria and Turkey. It is identical to the wall between Greece and Turkey, Turkey and Syria, and other walls in Europe, current, past and future. Surrounded by lush hills full of ancient ruins, buried secrets, unmarked graves and poisonous snakes, the new wall struck me as a perfect trope for that most timeless of things, the collective unconscious. Though devoid of conscious memory, the wall is its own foretold story.

How unjust borders feel when you are on the hard side, and how surprisingly small when they crumble

“When an inner situation is not made conscious, it appears outside as fate,” said Carl Jung of the psyche. This is the principle of hauntings, time warps and tragedies. In this remotest of border mountains, a poignant form of tourism is practised by the three border nations: ancestral tourism. More than 100 years after the Balkan wars of 1912 to 13 and the politely phrased and brutally executed “exchange of populations” that followed, the Greek, Turkish and Bulgarian grandchildren of the displaced still travel to their ancestors’ villages in Thrace, to the ruined houses, the blackened kitchens where pots and pans were abandoned as people ran for their lives across new borders. It is here that the locals have, for generations, claimed to see a mysterious ball of fire. It may be a freakish phenomenon of light, but it is richly imagined in legends of flying dragons. It appears in liminal spaces – at the entrance of old mines, over the border river, near curative springs – and always after dark, at the witching hour, the hour of the border and its inevitable transgression.

All my life, I have been haunted by borders – how unjust they feel when you are on the hard side, how alluring when you’re on the soft side, and how surprisingly small when they crumble. I was driven back to this border by an almost fateful urge for exorcism. Once among the border ghosts, I went down into the depths of their hunger and I listened to their stories.

Is it unavoidable that we would enter an era of building hard borders, again? No – it is only desperately unwise. The reason why new borders haunt us is because we haven’t listened well enough to the stories of the old ones. It is because the barbarians are here, not just among us but inside our heads, tirelessly tweeting hatred.

New borders will fail just as old borders failed. In the wretched meantime, they will not make our world freerer or fairer. Only harder, costlier, and more haunted.

• Kapka Kassabova is the author of Border: A Journey to the Edge of Europe, published this week by Granta. To order a copy for £11.24 (RRP £14.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.