Harry Borden’s father didn’t much care about being Jewish, so it was left to Harry’s grandmother to make him feel his heritage was special. And when his marriage ended, it proved invaluable

My dad, Charlie Borden, was born in New York in 1929, the only son of Jewish immigrants who had moved to the US as children. His father was from Ukraine and his mother from Romania. Having escaped the pogroms of eastern Europe, they met and married, built their own house in Upstate New York and had my dad.

Dad has always been an atheist. He has no truck with any religion. If he was evangelical about anything, it was being an American. In that more innocent era, the US was a welcoming place, allowing everybody a chance to get on. During the second world war, he lied about his age and joined the US Marines, just missing active service. After that, he went into advertising. I learned so much about him from watching Mad Men.

He has never really spoken about being Jewish, but he is a typical Norman Mailer-type wise-cracking Jewish guy. He has the humour, the putdowns. As a boy, he boxed, like his dad and his uncle, who was a promotor with three world champions in his stable of boxers. Boxing was very much part of Jewish life in New York then – and Dad’s humour was like a verbal fight to be the funniest guy in the room. He is macho, competitive, pugnacious and troubled – maybe the product of generations hardened by random violence. His mentality is fighting his corner.

By the time he came to England to work as the art director of a fledgling agency, he had a wife and baby son – me. My mum was half-Irish, half-English, quite posh, the daughter and granddaughter of admirals. We settled in Fulham, south-west London, and my brother and sister were born. I think Dad felt pressure in an industry that is constantly looking for newer, shinier people. He wanted to be his own boss, so he sold the house and bought a pig farm in Devon – 30 acres and a mass of concrete buildings on a cold, north-facing hill.

Facebook Twitter Pinterest Harry Borden as a child.

Growing up in Devon, I knew that part of Dad’s identity was “other”. I looked up to him – he seemed quite handsome, completely different and much cooler than the other dads. He was not communicative – especially with his kids. I remember travelling with him to deliver pigs – that was my “day out”, a rusty old van and squealing pigs. I would try making conversation but he wasn’t interested. He always seemed to be somewhere else.

The only reference he would make to his Jewishness – in a refrain I heard often – was that the Nazis would have killed people like us. I used to find it shocking. I was acutely aware that we came from “over the hills”. There were a couple of instances where it came up. I remember Dad taking me and my friend to the Devon county show, and my friend saying the Mini Metro was “a real Jew car” – meaning it used hardly any petrol. There was an awful, embarrassing pause and Dad later took me aside and told me that what he had said was offensive – though I instinctively knew. Another time, on a school trip to Germany, some of the people I was with began making jokes about “dirty Jews”. I remember feeling a bit ashamed that I didn’t say anything.

Facebook Twitter Pinterest Ella Prince, a Holocaust survivor: ‘A long road … from the gutters of life in my early youth to the beauty and freedom of Australia, at my sunset …’ Photograph: Harry Borden

For Dad, Jewishness was only a burden. Then his mother, Lillian, who had been widowed, moved from the US to live with us. Her attitude to being Jewish was completely different. She would talk about it and point out people who were Jewish, including, she claimed, Charlie Chaplin. She made it seem a bit special – like a really great club to be in.

When you are a teenager, you secretly want to find out you are a prince, or were adopted. In Devon, my Jewish heritage fed into that narrative. As I got into photography, I began to realise that all the astonishing photographers I adored were Jewish: Diane Arbus, Irving Penn, Richard Avedon – to me, the holy trinity! Then I discovered Woody Allen, Mel Brooks, and, later, Seinfeld. Why wouldn’t you want to accentuate that part of you?

I was 40 when I began photographing Holocaust survivors. I wanted to use photography to an intelligent end, with something that would stand the test of time. Dad’s ambivalence towards Jewishness was another motivation. He gets no spiritual sustenance from being Jewish, nothing from the traditions, no community or social network – and yet it is still clearly a part of his identity.

I took photographs first in the UK and then Australia, and next I had planned a trip to Israel. Just days before I was due to go, my wife ended our 14-year marriage. It was shocking, like a trapdoor opening, and I tried to cancel the trip. Then I thought, “Actually, this might be the best thing for me.”

Facebook Twitter Pinterest Leon Rosenzweig: ‘The best time of my life is when I am with my family.’ Photograph: Harry Borden

It was. I arrived in an emotionally raw state, smoking constantly, and I lost 15kg (2st 5lb). I remember driving around Jerusalem, in tears, completely lost. But it was amazing. Talking to survivors gives you a sense of perspective about your own life, your own concerns. You realise that what you are dealing with is of very little consequence. People were so kind. I went to my first Shabbat – which for me was about friends and family coming together. It was a very healing trip and, by the end, I was in a much better place. When I needed it most, I got a sustenance that my dad had never found.

Surviving the Holocaust: ‘I didn’t allow any hatred to grow. But I don’t blame those who did' Read more

Lots of photographers have taken pictures of Holocaust survivors, but I photographed 200 over three years. I just kept going. It was an exploration of my identity. There was this personal investment.

Though I have suggested to Dad a few trips – to Israel, or to his father’s homeland in Ukraine – he is not interested. He will say something like, “Why? They still hate Jews!” And that will be it. When the book was finished, I took it over to show him. He is not that interested really. He thought the publishers had done a nice job. “That’s as good as you’re going to get.”

As told to Anna Moore

• Survivor: A Portrait of the Survivors of the Holocaust by Harry Borden is published by Cassell Octopus, £30. To order a copy for £25, go to bookshop.theguardian.com or call the Guardian Bookshop on 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.