No one wants to live surrounded by death. It's understandable that people who now live on the spot that was once the Kovno ghetto, where close to 35,000 Jews were herded, starved and eventually led to their deaths, would not want to be constantly reminded of the fact. So I was not too surprised this week to watch fathers pushing baby buggies and mothers carrying groceries on Linkuvos Street, a residential road in modern Kaunas, Lithuania, with just one small obelisk – barely visible amid the traffic at a junction – marking the site where the gates to the ghetto once stood. The wording, in Hebrew and Lithuanian, is brief: no death toll, no mention of the unspeakable suffering that happened within.

I understand, too, why there are no special road signs directing visitors to make the short drive to the Ninth Fort, the place where the Nazis and their Lithuanian collaborators dug deep, vast pits – into which they shot almost 10,000 Jews, including 4,273 children, on a single day in October 1941, the so-called Great Action. I can see why the people of Kaunas would prefer the Ninth Fort to be seen only by those people who come looking for it.

Memory and history never belong solely in the past; they are contested in the here and now, as freighted with politics as any other aspect of the present. So it is in Lithuania, which, along with neighbouring Latvia and Poland, had a walk-on part in British politics last year, when David Cameron came under fire for partnering his MEPs with assorted ultra-nationalist fringe parties from eastern Europe. This week, searching along with my father for the roots of our family – one branch of which once lived in the Lithuanian village of Baisogala – I had a chance to examine what had once been a faraway Westminster battle on the ground and up close.

I have now seen for myself, for example, that the Ninth Fort includes not only a massive, Soviet-era socialist-realist memorial to the dead buried in those pits, but a newer exhibition hall, covering the oppression of the Soviet years – even though the connection between subject and location is tenuous at best. Of course, I can see why Lithuanians want to remember the era of the gulag and forced exile to Siberia. It was more recent than the second world war; it lasted longer; and it affected families still living in Lithuania. Besides, for four postwar decades to speak of that pain was forbidden, leaving a yearning for commemoration and recognition.

Pushing myself hard, I could almost empathise with the "double genocide" approach, officially endorsed in Lithuania and other former Soviet lands, which holds that nazism and communism were twin evils of the 20th century and ought to be remembered alongside each other – an approach embodied by the Ninth Fort, with its double museums, one recording the horrors of Hitler, the other counting the crimes of Stalin.

After all, this is not a competition – and if it is, it's not one any Jew would want to win. Jews don't want or need a monopoly on grief. Tears are not in finite supply: there are more than enough to go around.

But, no matter how great an effort of empathy I make, I cannot go along with the "double genocide", especially not now that I've seen how it plays out in practice rather than in theory. For one thing, the equation of Nazi and communist crimes rarely entails an honest account of the former. The plaque at the Ninth Fort, for instance, identifies the killers only as "Nazis and their assistants". It does not spell out that those assistants were Lithuanian volunteers, enthusiastically murdering their fellow Lithuanians. In my travels, visiting a whole clutch of sites, I did not encounter one that gave a direct, explicit account of this bald, harsh truth: that Lithuania's Jews were victims of one of the highest killing rates in Nazi Europe, more than 90%, chiefly because the local population smoothed the Germans' path. Indeed, they began killing Jews on June 22 1941, before Hitler's men had even arrived.

Second, even if the theoretical intention is to remember a "double genocide", it rarely stays double for very long. Take the Museum of Genocide Victims, off Vilnius's central Gedimino Boulevard. You would think such a place would feature the genocide of which Vilnius was close to the centre, namely the slaughter of the Jews. But you'd be wrong. The Holocaust is not mentioned. The focus is entirely on the suffering inflicted by the KGB. Outside, there are two prominent stone memorials for Moscow's victims. If you wish to remember Lithuania's 200,000 slain Jews, you have to wander far from the main drag, up a side street, to the tiny Green House – which is anyway closed for renovation and whose director, under pressure from state officials, is fighting for her job.

It's the same story with a 2008 change in the law that, in the name of equivalence, banned not just Nazi symbols but Soviet ones too. As if that were not bad enough – banning a veteran of the anti-Hitler resistance from parading his medals – in May, a Lithuanian court held that the swastika was not a Nazi symbol after all, but part of "Baltic culture" and therefore could be displayed in public.

Even if the authorities were rigorous in maintaining a balance, and telling both stories honestly, I would still reject this "double genocide". For the symmetry here is false. No one wants to top the persecution league table, but nor can one accept that those who were "arrested, interrogated and imprisoned" – to quote the Vilnius museum – suffered the same fate as those Jews who were murdered, despite the exhibit's attempt to equalise them under the bland umbrella term "losses". The oppression of the Soviet years was terrible, but it was not genocide: to be arrested is not to be shot into a pit. They are different and to say otherwise is to rob "genocide", a very specific term, of all meaning.

Finally, there is a sinister undertone to all this equivalence talk. Professor Egidijus Aleksandravicius of Vytautas Magnus University in Kaunas told me that many Lithuanians like to imagine that if their forebears killed Jews it was only as "revenge" for all that communists (for which read Jews) had inflicted on them. On this logic – warped because Soviet rule hit Jews as hard as anyone else – the "double genocide" in effect says: you hurt us, we hurt you, now we're even.

Why has this poisonous idea taken such deep root? Dovid Katz, who taught Yiddish at Vilnius University until his contract was not renewed this year, suspects geopolitics: "It supplies a massive stick with which to beat today's Russia," he says. Lithuania wants its European Union partners to see Moscow as a genocidal regime that has not made restitution.

He detects another motive too: the nationalist desire for Lithuanians to see themselves as a pristine people, free of stains on their record. Admitting the truth of the wartime past threatens that; insistence on victim status preserves it.

This may inform the action the rest of the world should take. Professor Aleksandravicius calls for a "soft hand", for outsiders to understand how psychologically difficult it is for people to realise that victims can be perpetrators too, to accept that having suffered in the first Soviet rule of 1940-41, "Lithuanians turned on the weakest people of all, the Jews".

I respect that approach: memory is a sensitive business. But governments will have to speak more forcefully. Lithuania is in the EU and Nato: its partners in those bodies have a duty to tell Vilnius plainly that it needs to reckon with its past truthfully, no matter how painful that may be. Only then will the haunting spirits of the past let it rest.

• In Search of A Shtetl, presented by Michael and Jonathan Freedland, will be broadcast on Radio 4, October 18, 8pm