I’m trying to remember the first refugee I ever met. Our next-door neighbours when I was a child were Jewish, from Germany, and they complained with a righteous horror when my dad invited Leni Riefenstahl to talk at the BFI.

My father was conflicted about this. In 1946 he’d been part of the occupation of Germany sent to flattened Cologne; it was an experience that coloured the rest of his life. “Millions of people just walking across the blasted continent, going to some distant home, not knowing if there was anything or anyone still there. You have no idea, no idea.” But he also believed that what he’d been fighting for was civilisation, and that art transcended and healed all. In my teens I had a few years of getting on very badly with him; he was frustrated and disappointed with me, and I was frustrated and resented his disappointment. We had a family friend, a Hungarian film director, who took me aside to mediate. With a sulky insouciance, I asked him how he’d got on with his father when he was 16. “I was in Auschwitz,” he said. “Cherish your father.”

Today, my neighbours are a Russian by way of China, a Vietnamese, Spanish, an American, an Armenian and a Colombian. I hadn’t thought about that until just now, because I know where they’re from: they’re from next door. The amazing thing about refugees is not their difference but how quickly they’re folded in to become us – hyphenated, perhaps, for a couple of generations, but how many Huguenots do you know? How many identifiable descendants of Cold War Hungary, how many Depression-era Italians or Salazar-defying Portuguese?

The sanguine truth is that in 20 years the fear about Middle Eastern immigration will be thought of, if thought about at all, as absurd Little English xenophobia, the phantom pain of a long-lost imperial limb. The first story I ever did on refugees was in Peshawar at the beginning of Shock and Awe. Thousands of Afghans ran from the bombing of Kabul to Pakistan, many of them Taliban or Taliban supporters.

Peshawar was vivid and febrile. I was walking through a covered alley in a market when an Afghan with his pakul cap pulled down, hunched in a blanket, came the other way. His pale eyes fixed me with a ferocious purpose. I knew that he meant me harm – for the bombing of his home; because I was an invading infidel; a historic enemy. The path was so narrow, there was no avoiding him and with one swift movement he grabbed me and threw me up against a wall. I tensed for the blade, too frightened to make a sound. His face was inches from mine.

And then I saw the boss of a great cartwheel pass where my head had just been. He’d saved my life. He stepped back, touched his heart with his hand and gave me the merest flicker of a smile. I’m sorry, you may have heard this story before: I tell it a lot. It was a lesson; those we think have come to harm us may have come to save us.

A A Gill’s “Pour Me: a Life” will be published by Weidenfeld & Nicolson on 12 November