I have been working off and on for a couple of years with an architect friend, Declan Hill, on an attempt to map the wall or rampart that in the 17th century surrounded the town of Belfast – or, at least, may have surrounded it. There is some debate about whether maps of the time show an existing structure or sketches for work never carried out. Declan and I meet occasionally to look at places where we think the old town plan intersects with, and occasionally contradicts, the modern city grid. One day a column in Argos on Corn Market, the next, a staircase in the Linen Hall Library.

Our thesis is that, even when all physical trace is gone, walls persist. Not altogether original, you might say, but in Belfast, where peace walls remain an all too visible fact, it is pertinent to Declan’s mind and mine. Our hope is that we can collaborate with an architectural institute in Berlin. Not altogether surprising, you might also say, but Berlin is a city to which we both feel bound by something more than bricks and mortar.

On my own first trip to Berlin, in the spring of 1990, I paid a visit to the Reichstag, whose rear wall was just feet from the other wall – the Berliner Mauer – and whose sole purpose seemed to be the housing of a “permanent” exhibition, Fragen an die Deutsche Geschichte (Questions on German History). The sixth of its seven rooms covered the period of the Third Reich, and there I encountered a photograph that showed a man kneeling, a soldier behind him with a pistol, and a crowd of onlookers. It was clear what was about to happen, yet the expression on the victim’s face was not one of terror, but rather of disgust that this thing was being done to him, that these people were capable not just of doing it, but of watching it being done.

I say I encountered this photograph. Actually, I was stricken before it.

Long after I had returned home, I was turning over the components in my head – the victim, the executioner, the audience – haunted by the thought that, as much as I wanted to imagine I could only ever find myself in the place of the kneeling man, there was nothing in my DNA, my Glenn Patterson-ness, that absolutely guaranteed I could not end up playing either of the other roles. The only safeguard that I could see was vigilance, against any ideology that reduced human beings to the one word for which they could be murdered (Jew, Commie, Prod, Taig, Brit, Mick … or fascist, come to that); and against yourself, that you didn’t just shrug such language off. By the time the onlookers have gathered, the victim has been made to kneel and the soldier is pointing his gun, it is far too late to ask how you got here.

It became such a mantra that I cited the photo repeatedly over the years. Always I talked of the near-uniform age of all those depicted, and always I talked about the expression on the victim’s face as he looked at those looking at him about to be shot. I would sometimes talk too about the intimacy of mass murder: how it was far from unusual for the victims to be known to their killers, or to those who aided and abetted them.

Eventually I began to think there was a book there I should write: identify as many of the people in the photograph as possible, work back from that moment of the shot about to be fired, see how they all arrived at that lethal configuration. I wrote to a friend in Berlin, describing the photograph and its significance to me and asking if she knew what had become of the Questions on German History exhibition once the Reichstag had got its Fostered glass dome and become the Bundestag. My friend wrote back that the exhibition had been broken up and the images dispersed to the various libraries and collections from which they had been drawn. In fact, she had already spoken to people in two of the most important archives in Berlin. Neither held the photograph I was after, but it was early days and she would keep looking. A couple of weeks later I had another email. She had now been in touch with all the archives. Not only had she not found the photograph, she had not found anyone who thought her description of it sounded at all familiar.

“There are not many photographs,” she wrote, “with just one guy being shot.”

She sent me a scan of the only one she had turned up. I opened the attachment and closed it almost instantly. It was totally wrong: for a start, the victim was not facing his audience, who in turn were not baiting him. Most tellingly of all, the execution was taking place on the edge of a mass grave, whereas I had specifically described a city street.

An hour or two later I opened the attachment again. There was certainly something familiar in the cast of the victim’s eyes. I was looking at a scan of a copy of the original, but the expression was definitely not one of terror. The casual stance of the soldier with the gun was not unfamiliar either. And though the onlookers were not as gleeful as I had remembered (and were also to a man soldiers, not civilians), there was something disturbing in the simple fact of their looking on, even in one case leaning to the side to see around a comrade who had cut across the sightlines.

For the first time it occurred to me that with the passing of years I might have tidied up the image that had so paralysed me when I first saw it in the Reichstag, and made it more neatly allegorical, geometrical, even: a triangle of morality.

It is in many ways more appalling than I had remembered, as is the legend – in the hand of a German soldier – on the back: “The last Jew in Vinnitsa.” I realise that my search for it must seem incredibly dated now: today I could, if I wanted to, click on any one of several thousand websites and view that and other atrocity photographs that survive from that time. It might surprise some people that so few do; although that’s another thing that has changed, since the second world war, certainly, but even in the last decade. Where once perpetrators went to enormous lengths to cover up their massacres and atrocities, now they post them online.

The Vinnitsa photo came to represent something else, too – a reminder of how warily you must tread when you try to enlist the past, however good your intentions, and indeed however modest your enterprise.

Maybe that is one of the reasons Declan Hill and I have proceeded so slowly with our map: we are pretty sure we are on to something, but we are reluctant to force the issue, to see what we want to see rather than what we can.

Every so often, though, in large ways and small, standing before a photograph in the Reichstag, or looking down at an inexplicable kink in the staircase of the Linen Hall, history seems to buttonhole you and ask, “Have you seen? Have you understood?” And then, “What are you going to do about it?”

• Glenn Patterson’s novel The Rest Just Follows is published by Faber. Buy it for £11.04 at bookshop.theguardian.com