On the last night of February 1986, the Swedish prime minister Olof Palme and his wife, Lisbet, were strolling home through downtown Stockholm. They had taken an impromptu trip to the cinema and decided, as they often did, not to bring bodyguards. Palme made a point of living as much as possible like an ordinary person; he did not want the fact that he was running the country to come between him and his countrymen. “You saw him in the streets all the time,” says the Swedish ethnologist Jonas Engman. “You could speak to him. There was an intimacy to it.”

At 11.21pm, as the couple walked down Sveavägen, one of Stockholm’s busiest streets, a tall man in a dark coat walked up behind them. The man put one hand on Palme’s shoulder, and with his other hand fired a single round from a gun into the prime minister’s back. He grazed Lisbet with a second bullet before fleeing up a flight of 89 steps that links the main street with a parallel road above.

It was a Friday, and Sveavägen was packed with people ambling between bars and restaurants. Bystanders rushed to try to revive Palme, who now lay on the pavement in an expanding pool of blood. Six minutes later, he was taken to the nearest hospital, where, shortly after midnight, he was officially declared dead. It was later determined that the bullet had severed his spinal cord and that he had died before hitting the ground.

Although more than 20 witnesses saw the gunman, these facts are still more or less everything that the public knows for certain about the killing of the most controversial leader in Sweden’s modern history.

To his fellow countrymen, Palme was more than a politician. For more than 16 years, he had led Sweden’s leftwing Social Democratic party, which was in power for much of the 20th century. The party was responsible for many of the policies that people typically associate with Sweden, including high taxes and a robust social welfare system. Palme had come to embody not only the party, but these values, too.

For this, Palme was loved by many – his predecessor Tage Erlander called him “the greatest political talent Sweden has seen this century” – and despised by others. He was distrusted by some on the left for being from aristocratic stock, and distrusted by aristocrats for being a class traitor. Paranoid corners of the Swedish right made wild allegations that he was a Soviet spy. Contra, a popular conservative magazine, sold dartboards featuring a caricature of his face. On the night of the killing, when word of Palme’s death reached Claes Löfgren, a journalist for the Swedish national broadcaster SVT, he was in a restaurant. When they heard the news, Löfgren told me, some people in the restaurant cheered and toasted. To Swedes of all political persuasions, the symbolism of Palme’s assassination was clear: it was as if the killer wanted to destroy the idea of modern Sweden itself.

Palme addressing crowds in 1986. Photograph: Sipa/Rex

Following Palme’s death, the country was cast first into turmoil and then into confusion. Over the past three decades, one chief investigator after another has failed to solve the case, and today the official inquiry remains open. In 2010, Sweden removed the statute of limitations on murders, specifically so that investigators could continue their search for Palme’s killer for as long as it takes. More than 10,000 people have been questioned in the case, whose files now take up more than 250 metres of shelf space in Sweden’s national police headquarters. It is the largest active murder investigation archive in the world.

The mystery of Palme’s death has become a national obsession. “One of my earliest memories is of my parents discussing who killed Palme,” a friend I met while living in Sweden for the past couple of years told me. “I can’t describe to you how deep this is in the Swedish soul.” The murder has inspired films, plays and music, and has even been cited as a factor in the worldwide explosion of Scandinavian crime fiction. A number of Swedish amateur detectives have devoted much of their lives to solving the case. Investigating it has led some of them to break the law and driven others to something approaching madness.

Some Swedes call this Palmessjukdom – “Palme sickness”. More than 130 people have falsely confessed to the crime. “Swedes are breastfed with the idea of this horrible trauma,” Måns Månsson, a director who made a film about the murder, said. “It’s genuinely hard to let go.”

Every unsolved crime creates a vacuum that people fill with their own theories, but there are few cases in the world for which this has been more true than the killing of Palme. He had many enemies, and his death lends itself easily to conspiracy theories. Palme so divided opinion that the detective Lennart Gustafsson, who worked on the case from 1986 to 2016, told a reporter in 2012 that “you could suspect half the Swedish population”.

One of Palme’s sons, Joakim, who is now a political scientist at Uppsala university, told me that many of the conspiracy theories are far from baseless. “You can come up with not only one or two but a handful of different, more or less credible scenarios for an organised assassination,” he said. When he died in 2004, the crime novelist Stieg Larsson was working on a theory of the case involving an international conspiracy that is now being seriously investigated by Swedish police.

Although there have been false dawns before, there is reason to be hopeful that the case may soon reach its conclusion. In February of this year, the latest lead investigator appeared on Swedish television and, with astonishing confidence, promised to give the public a solution within the next few years. Caution is advisable; this possibility has been teased many times before. But police have confirmed that they are interviewing new subjects and testing new physical evidence for the first time in many years. After 33 years, modern Sweden’s defining drama may be coming to an end.

From the moment when the first emergency call was made after the shooting, Sweden was thrown into chaos. Recordings of early conversations between police headquarters, officers at the scene of Palme’s shooting and staff at the hospital are mostly expressions of disbelief. “What? No! Our prime minister?” one police officer asks. “There’s total confusion here,” another person says. “Is it really Olof Palme who’s been shot?”

Leif Brännström was reporting for the national newspaper Expressen that night. He remembers calling the head of the city police for confirmation of rumours that Palme had died. “He screamed something at me and immediately hung up,” Brännström told me.

On Sveavägen, where the shooting occurred, shock seemed to have taken over. Police failed to cordon off the crime scene properly, covering too small an area. One of the bullets was not found until two days later, when it was picked up from the pavement by a passerby. Mourners arriving in the hours after Palme’s death slipped past the tape to place flowers near the pool of blood; by trampling the crime scene, they rendered future searches for the killer’s footprints useless. Key witnesses were allowed to leave the scene without being interviewed. Löfgren, the broadcast journalist, was out in the area that night and hailed a cab to take him home. The driver had witnessed the killing but had not been questioned, Löfgren recalled with disbelief. “I phoned the police and said: ‘This guy here claims that he was a witness to the murder, and he’s still out driving a cab?!’”

Other protocols were ignored or forgotten. The Stockholm police have a system for searching the inner city street by street, but it was never deployed. Squads of police tore around looking for the gunman, but had almost no information about what he might look like. Trains, ferries and flights continued as normal, while the roads and bridges out of the city remained open for hours after the murder. At that stage, it seemed as if nobody was really in charge. It was “sports week”, a holiday when many Stockholmers head for the mountains. Hans Holmér, the chief constable of Stockholm county police, was skiing in the north country with his mistress.

Holmér had never conducted a murder inquiry before, but when he got the news early the following morning, he rushed back to the city and took charge of the investigation. He looked the part, at least, with a craggy face, hard-boiled demeanour and a strong line in leather jackets. From his first television appearance, he played the role of the hero that the horrified nation could rely on. The public soon flooded his office with bouquets and chocolates, and newspapers hailed him as Sweden’s Clint Eastwood.

The site of Palme’s murder in Stockholm in 1986. Photograph: Sipa/Rex

In the days and weeks that followed, the country struggled to come to terms with what had happened. “It was almost literally incredible, it was something that completely couldn’t happen in nice, safe, controlled Sweden,” the journalist Andrew Brown, who lived in Sweden for many years, told me. “You might as well have asked them to defend the prime minister against pigs falling from balconies.” The last murder of a Swedish government official was in 1792, when King Gustav III was shot by assassins at a masked ball. When the on-air DJ at the national radio broadcaster found out about Palme’s death, he was at a loss for what to do, the author Jan Bondeson writes in his book Blood on the Snow: The Killing of Olof Palme. The only emergency protocol was to break open a small glass box in the studio marked KD for “the King is Dead”. It contained a cassette tape of sombre classical music.

Despite his popularity, Holmér seemed out of his depth, too. One of his first steps was to release a composite image of a suspect who had been seen running near the site of the murder. But gaps between supposed sightings of the fleeing killer meant no one could be sure the man in the sketch was in fact the killer. Nevertheless, the image of the suspect, a Nordic-looking man with a long nose and thin lips, who the Swedish press dubbed “the Phantom”, ran in every newspaper in the country. Over the next few days, the telephone exchange at Stockholm’s police headquarters was brought down by the volume of calls it received. More than 8,000 tips flooded in about neighbours and acquaintances who resembled the Phantom.

Seventeen days after the murder, the first suspect was brought into custody. He had links to rightwing groups that believed Palme was an undercover KGB agent, but was quickly released due to lack of evidence. By that point, Holmér had begun to fixate on an alternative theory: that members of the militant Kurdistan Workers’ party, or PKK, had assassinated the prime minister. The group, which had an outpost in Stockholm, had recently been declared a terrorist organisation by Palme’s government. But the only other thing to recommend the theory seemed to be that the PKK was easy to scapegoat. There was virtually no evidence against the group.

Holmér was so caught up in the theory that he didn’t seem to care about the lack of evidence. In early 1987, after investigating the PKK for almost a year, police raided a Stockholm bookshop that served as a base for the group and arrested 50 of its members. But the raid produced nothing of value to the investigation. Newspapers turned on Holmér for this failure, running headlines such as “Holmér must go” and comparing him to Inspector Clouseau from the Pink Panther. On 5 March, Holmér resigned in disgrace. An official inquiry into the first year of the investigation later concluded that it was “characterised by alarming aimlessness and confusion”.

Things only got more bizarre from there. In secret, Holmér continued to pursue the PKK theory as a private citizen until 1988, when he and a journalist called Ebbe Carlsson were caught trying to smuggle illegal wiretapping equipment into Sweden in order to keep surveilling members of the group. It was a national scandal – the country’s erstwhile hero breaking the law to investigate a discredited theory of the prime minister’s assassination – even before it was discovered that the minister for justice at the time, Anna-Greta Leijon, was also in on the scheme.

In 1988, the Swedish police appointed a new chief investigator who went back to a lead that had been dropped by Holmér during the Kurdish debacle: a man behaving suspiciously on the night of the murder near the cinema where the Palmes had been. There was also a potential suspect, Christer Pettersson, who matched this man’s description. Three acquaintances had come forward to say that Pettersson, who often hung around Sveavägen, was certainly capable of murder: he had spent time in prison for randomly stabbing someone to death with a bayonet.

If Holmér had looked like a storybook hero, Pettersson was a storybook villain, with a strong brow, downturned mouth and wild eyes. Although he had no evident motive, police took him into custody on 14 December 1988. That same night, Lisbet Palme watched a video line-up that included Pettersson and identified him as her husband’s killer. Pettersson maintained his innocence and there was no forensic evidence linking him to the crime scene. But on the strength of Lisbet’s identification, in July 1989, after a seven-week trial, he was convicted and sentenced to life in prison. For a moment, it seemed that this national trauma was over.

But Pettersson’s lawyers immediately appealed against the decision. They demonstrated that police had told Lisbet before the line-up that the suspect was a heavy drinker. She had identified Pettersson by saying “No 8 matches my description” and then, crucially, “you can see who is the alcoholic”. Pettersson was released in October 1989. A now iconic photograph shows him returning to his apartment, clutching bottles of vodka and Bailey’s Irish cream, as if to show the public how he intended to spend his freedom, and the £38,000 he had been awarded in compensation for his wrongful arrest. A cocktail of Bailey’s and vodka subsequently became popular in Stockholm bars. It was called The Killer.

Pettersson didn’t disappear from public life. In the years after his release, he charged newspapers and TV stations large sums for interviews in which he was baited and bribed for a confession. He hinted at the possibility he was guilty, but never confessed. He died in 2004. If he did kill the prime minister – which many Swedes continued to believe – he took the secret to his grave.

By the start of the 1990s, so much time and money had been spent fruitlessly pursuing Pettersson and the PKK that basic questions about the night of the murder remained unanswered. Where was the murder weapon, which was believed to be a Smith & Wesson .357 magnum revolver? Why were witness reports of men with walkie talkies near the site of the killing not taken seriously? Was the police incompetence too extreme to be accidental?

Over the next two decades, the official investigation floundered. Although at least four different lead investigators took over the case between 1988 and 2013, not a single credible suspect was taken into custody. When I asked Måns Månsson, the film director, how the official Palme investigation is perceived these days, he laughed and said: “A horrible, miserable failure.”

Into the void opened up by the lack of official progress flowed a stream of amateur investigators pursuing their own solutions to the case. No detail about Palme’s life or death, no coincidence or inconsistency proved too small a foundation on which to build one conspiracy theory or another.

No theory was too outlandish: Palme’s wife killed him because of his serial infidelities. The people who killed Palme are the same ones who killed JFK. It was feminists in cahoots with Scientologists. It was a planned suicide and the trigger was pulled by Palme’s son Mårten. No, the entire murder was staged. Or maybe the gunman was the Russian film-maker Andrei Tarkovsky, who had met with Palme in 1984 to ask for help getting his own son out of the USSR: Tarkovsky’s 1986 film The Sacrifice contains a scene shot at the corner where Palme was gunned down. “These are mostly made up by sick people,” Mårten told the New York Times in 1998, referring to the conspiracy theorists fixated on his father’s death. “I’ve had some of them call and apologise to me after they received medication and got better.”

Hans Holmér, who headed the investigation into the Palme’s assassination. Photograph: Hakan Roden/TT/Press Association Images

Some theories, including several that appear more credible, came from committed Palme obsessives known as privatspanarna, or private detectives. These are people who have devoted significant chunks of their lives to finding a solution to the case. The first privatspanarna emerged in the wake of Holmér’s failed raid on the PKK, and were sometimes accused of harassing witnesses and interfering with the official investigation. Many used live re-enactments at the scene of the murder as an investigative tool. Månsson recently described these reenactments to a reporter as a “mix between seance and ceremony … one of the strangest things I’ve ever experienced.” At one meeting of privatspanarna in 1998, Christer Pettersson reportedly showed up. He drank several rum and Cokes, claimed Palme was killed by a group of rightwing Italian freemasons with links to the mafia, told sex jokes and left.

Over the years, privatspanarna have ranged from serious investigative journalists to crackpots. Some have become professional conspiracy theorists, fuelling a cottage industry of Palme mania. The well-known journalist Sven Anér published five books about the case between 1988 and his death in 2018. The privatspanarna are also predominantly men. Perhaps it’s “a male thing to think you alone can solve the case given the biggest police resources in Swedish history”, Stephanie Thögersen, a policy officer at the Swedish Women’s Lobby, told me.

One reason the murder still has such a hold on Swedes is that Palme was the nation’s first real political celebrity. “He was quite unusual for a Swedish politician at the time because he was outspoken, he was polemical, a hard hitter,” his biographer Henrik Berggren told me. “He sought conflict.” Henry Kissinger, whose political views could hardly have diverged further from Palme’s, once said “many political leaders are really boring except for the office they hold. That was not true of Palme.”

The past decade has seen a boom in Palmology, especially among people who weren’t yet born in 1986, or were too young to follow the investigation’s early years. This is partly because of 25th and 30th anniversary coverage in the Swedish media, and partly because new investigators were appointed in 2013 and 2016. “It goes in waves,” says Jan Stocklassa, a journalist and author of a 2018 book on the Palme murder, The Man Who Played with Fire, which will be published in English in October. “And I would say that now we’re at a sort of a peak.”

One of the most popular new venues for this obsession is Palmemordet (“The Palme Murder”), a podcast exploring the ins and outs of the case, which runs to 173 episodes and counting. The host, Dan Hörning, also organises Palme walks on the aniversary of the killing. He and his listeners walk Palme’s last route, arriving at the murder site at the exact minute when Palme was killed. Schiaffino Musarra, who is making a television series about trying to solve the Palme murder, has been on a couple of these walks. “It’s fun,” he told me. “We go to a Mongolian barbecue place, too.”

Many privatspanarna are still hard at work. Some of the more serious ones have recently founded a group called Sanningskommission, or The Truth Commission. These people have been independently investigating the case for years, and began to pool their resources in 2016. “Our primary mission is to collect and pass on anonymous tips – either to the police or to journalists,” the group’s chairman, Sven-Åke Österberg, told me. “Some people might not be inclined to speak with the police, then we are a secure alternative.”

Solving the case is still of paramount importance to Sweden, Österberg said. A Sanningskommission press release rather bombastically claims that “this murder must be settled for the credibility and survival of our democratic society”. To other private investigators, however, the unsolved case is less of a pressing threat to Sweden’s democracy and more of an intellectual challenge. Louise Drangel, a long-time privatspanare, likens it to a complex Agatha Christie puzzle. “And you have to be sharp to get all the pieces together,” she said.

Recently, the work of some journalists and privatspanarna has begun to yield theories that are influencing the official police investigation. Last summer, a Swedish magazine called Filter published the results of a 12-year investigation that claimed the assassin was a witness in the case named Stig Engström. Engström is better known in Sweden as the “Skandia man”, because he worked for the Swedish insurance giant, which had offices next to the murder site.

Politically, Engström was what is known in Sweden as a moderat – firmly to the right of Palme. Thomas Pettersson, the journalist who led the magazine investigation, discovered that Engström had previously served in the Swedish military. Pettersson alleges that Engström would have had weapons training and access, through a friend with a large firearms collection, to the sort of .357 magnum with which Palme was apparently shot. Records from the Skandia office show that Engström left the building at 11.19pm, two minutes before Palme was killed.

Engström killed himself in 2000. His wife, who he divorced the previous year, believes he was too much of a coward to assassinate Palme. But she recently told the Expressen newspaper that police questioned her twice in 2017. Expressen also reported that Thomas Pettersson, the journalist behind the investigation, has been questioned by police as an expert in the case.

Police are exploring a far more unsettling theory, too, which was developed in part by the most famous person to become a privatspanare, Stieg Larsson. In addition to writing the bestselling Millennium trilogy, Larsson had a long career as an investigative journalist. The theory he was working on when he died of a heart attack in 2004 is that the murder was carried out by an international conspiracy consisting of two groups with different motives but a shared belief that Palme should die.

The first group was made up of pro-apartheid members of South Africa’s security and intelligence services. Palme was an outspoken opponent of the apartheid regime, and his government had given millions in humanitarian aid to Nelson Mandela’s African National Congress. Theories about South African involvement in his murder have circulated since the earliest days of the case. They became particularly popular in 1996, when a former commander of a South African police hit squad alleged that Palme’s killing was part of Operation Long Reach, a top-secret programme to neutralise opposition to the apartheid government at home and abroad. In 1982, members of this operation had killed the anti-apartheid activist Ruth First in Mozambique and bombed the ANC’s London office.

Christer Pettersson, returning home after being acquitted on appeal for Palme’s murder. Photograph: AFP/Getty

The second group Larsson identified consisted of rightwing extremists within Sweden, whose networks Larsson had been investigating even before the Palme killing. One of the men Larsson believed was involved in the assassination plot was a Swedish mercenary, Bertil Wedin, who had allegedly worked for the South African spy in charge of Operation Long Reach. Larsson claimed that Wedin helped to recruit Palme’s assassin, a Swedish extremist. Wedin denies any involvement in the case and has never been charged. “I have nothing to lose from the truth coming out since, luckily enough, I am not the murderer and had nothing to do with it all,” Wedin said in an interview published in the Swedish newspaper Svenska Dagbladet in 2014. Swedish police investigators visited South Africa in October 1996 and said they were unable to uncover evidence of this conspiracy. Jan Stocklassa, who carried on Larsson’s investigation after Larsson’s death, and published his study of the case in Swedish late last year, told me that he believes the assassin is still alive and is presently under investigation by Swedish police.

Both of these lines of inquiry – the Skandia man and the so-called “South Africa track” – were around in the earliest days of the investigation. So why is information previously unknown to the police still being discovered by journalists? Here, too, there are multiple theories. In the first 15 years after the murder, at least four official inquests were launched into Hans Holmér’s original investigation. The incompetence of that investigation gave rise to the popular theory that Palme was murdered by rightwing extremists within the police force, of which there were a good number. Holmér himself was in charge of one particularly fearsome group of plainclothes police officers who had a reputation for brutality and expressing support for Nazi ideology.

Others believe the problem was that Holmér and subsequent investigators were under enormous, although unofficial, pressure from the Social Democratic government to find a solution that would be easy for society to accept. First, the Kurdish separatists seemed best. Then Christer Pettersson, an alcoholic with a violent history – a figure, as the author Jan Bondeson puts it, “whom no one would have missed”.

This is the view held by Gunnar Wall, an investigative journalist who has written several books about the Palme murder. He told me that a reluctance to look at the uglier sides of life in their country is a long Swedish tradition. Historians in Sweden took a long time to address the fact that ostensibly neutral Sweden had traded with the Nazis during the second world war, for instance. This same reluctance, some think, explains why many in Sweden were so surprised by the recent surge in the electoral success of the far-right Sweden Democrats party, which took nearly 18% of the vote in the 2018 parliamentary election.

As with almost everything in this case, this view – that the investigators were looking for a palatable solution – is contested. According to the privatspanare Drangel, the fundamental problem with the investigation was that Holmér did not know what he was doing. A few years after resigning from the police force, Holmér took to writing moderately successful crime fiction about a Stockholm policeman nicknamed “The Flea”. At the time of his death in 2002, he was working on a non-fiction book about the Palme murder. “He was an administrator, but he wanted to be the hero of the country,” Drangel said. “The simple answer is that he was not competent.”

If the unofficial investigations into Palme’s death have gathered pace over time, it is only in the past couple of years that the official Palme investigation has taken on new life. Unbelievably, new physical evidence has surfaced after 30 years. This April, investigators received a walkie-talkie that was allegedly found in the vicinity of the murder two days after Palme was killed. It is unclear why the person who found the walkie-talkie, who investigators publicly refer to using a pseudonym, held on to it for so long, or why this person decided to hand it in now. If the walkie-talkie is connected to Palme’s murder, that fact would suggest that more than one person was involved in a plot to kill the prime minister. News reports claim that the walkie-talkie’s mouthpiece is being tested for DNA.

The investigators are secretive, Gunnar Wall told me. The new lead investigator, who took over in 2016, is a man rather improbably called Krister Petersson. He did not respond to my requests for comment, but when he appeared on the weekly crime show Veckans Brott in February, he said that his team of investigators is receiving new tips every week. “That’s why I’m optimistic and that’s why we’ll be able to solve this crime,” he told the host. “We will be able to tell the Swedish people what happened – that I am sure of.”

Mourners lay flowers near a flame burning in Palme’s memory soon after his murder. Photograph: AFP/Getty

Whether or not a solution will be delivered soon depends on whom you ask. Most of the people I spoke to gave a resigned no. “Various quacks and charlatans have brought forward fantasy solutions to the murder, but nothing valuable whatsoever has happened since I wrote my book [in 2005],” said Jan Bondeson. Others say too many key witnesses are now dead, while others have proven unreliable, and that there is too little forensic evidence.

In contrast, Jan Stocklassa and Sven-Åke Österberg, the head of the Sanningskommission, are optimistic. “But it won’t come by itself from the Swedish police,” Stocklassa said. “I believe that the pressure to solve the murder will come from young people who won’t accept an unsolved murder.” For its part, the Sanningskommission has pledged not to give up until a solution is found. And yet, as Stocklassa writes in his recent book, in some ways it feels that we know less now, 30 years on, than we did in the first year.

Last May, Sweden’s current prime minister, Stefan Löfven, called the case “an open wound in Swedish society” and said: “It is extremely important that this is solved.” The only thing more terrible than having the figurehead of modern Sweden murdered would be for the country never to know what it meant. Arne Ruth, the former editor-in-chief of the newspaper Dagens Nyheter, said in 1998 that it was the aftermath of resignations and scandals, rather than the assassination itself, that haunted the country: “The total failure of the judicial system to handle the case was in a way an even worse disaster for Sweden.”

“We know at this point that there is less than a 10% chance to solve the murder,” an anonymous member of Palme’s staff told the New York Times less than a year after the killing. “That is not our problem. Our problem is what the people of Sweden believe happened and how they deal with that.” Three decades on, without a perpetrator, without a motive and without a conviction, that problem continues to fester.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.