On 29 May it was announced that the Japanese and North Korean government had made an important agreement. North Korea promised to create an investigation committee to look into the fate of Japanese citizens who were kidnapped by North Korean agents in the 1970s and 1980s. In exchange, the Japanese government promised to ease sanctions on North Korea.

This agreement - long anticipated by certain insiders - is a major twist in a long and painful story that sometimes resembles a bizarre thriller.

Gone forever

Our story begins in the late 1970s when people began to mysteriously disappear from coastal areas of Japan. With few exceptions, victims were young men and women, usually in their 20s. None of the missing persons had any significant political or financial connections, and most of them were just ordinary people - a nurse, a noodle chef, a college student, a carpenter and the like.

From the early 1980s, there were persistent rumours that these strange disappearances were somehow related to North Korea. In some cases, suspicious men or boats were spotted nearby, and once or twice, a postcard from the disappeared person mysteriously appeared, telling the family that the person was living in Pyongyang.

These suspicions initially were seldom taken seriously. However, from the late 1980s credible evidence began to emerge: A number of the arrested North Korean spies admitted that they had once received language and practical training from people who closely resembled the missing Japanese. However, the North Korean authorities also denied the accusations - and many people believed them. After all, it was bizarre to assume that secret agents would abduct randomly chosen civilians from a neighbouring country.

The admission of the dear leader

So, the world was surprised when in September 2002, during an official visit of Japanese Prime Minister Koizumi to Pyongyang, the North Korean leader Marshal Kim Jong Il admitted that the North Korean intelligence services had indeed kidnapped 13 Japanese citizens.

It seems that the North Korean leadership hoped that the admission would be seen as a sign of their goodwill, thus making possible a dramatic improvement of Pyongyang’s relations with Japan. At the time, North Korea badly needed Japanese economic cooperation, and hoped to secure compensation for the damage inflicted during the colonial period of 1910-1945.

The admission was seen as the first step towards better relations with Tokyo. However, the attempt backfired, and the North Korean government was punished for its rare attempt to be honest: The Japanese public was outraged by the revelations. In weeks, the abductee problem became the major preoccupation of the Japanese public. They demanded the immediate return of all abducted people.

It is not known for sure how many Japanese citizens have been abducted. Some groups talk about 100 abductees, while the Japanese government list includes 17 names, but the North Koreans have admitted responsibility for 13 cases.

In 2002, the Japanese were also told that of the 13 abductees, only five were alive at the time. In 2004, these five people were allowed to go to Japan with their children. With regard to the remaining eight, the North Korean official position has been steadfast - until last week, at least. Pyongyang kept saying that all these people had died in North Korea of natural causes. Given that in 2002 all but one of the abductees should be in their 40s or late 30s, such statements do not look plausible.

Since then, the situation has developed into a stalemate. Few people in Japan believe that the abductees are dead, and the Japanese public continues to demand their immediate return. Their fate evoke great sympathy: They were, literally, men and women from the street whose lives were broken forever because of whims of a bizarre foreign dictator.

The North Korean government’s refusal to further discuss the issue is not as irrational as it might appear at first glance. Pyongyang policy planners have learned from their sad experience that honesty does not necessarily make the best foreign policy. There are at least two reasons that make concessions difficult or impossible - even if we assume that all abductees are indeed alive.

First, it is clear that many or even most people in Japan will not be satisfied by the return of the officially recognised "missing eight", and will insist that many more people are still secretly held in North Korea. Even if the North Koreans return all alive Japanese, it will be impossible to persuade the Japanese public that nobody else is still secretly forced to stay in North Korea.

Second, the five abductees who were allowed to return in 2004, were not involved in the training of spies while at least some of "missing eight" are known to have done this type of job. Sending these people back to Japan would be tantamount to exposing all North Korean clandestine operations there. As the events of 2002 have demonstrated, the Japanese public is unlikely to see this as a conciliatory gesture. On the contrary, another burst of popular anti-Pyongyang sentiment is likely.

A new attempt

Over the years, the North Korean government has come to understand that without at least partial concessions on the abduction issue, it will be impossible to resume normal trade with Japan, let alone win the diplomatic recognition of Tokyo. North Korea badly needs Japan as a trade partner and aid donor. Before 2002, Japan provided much humanitarian aid to North Korea, and there is a probability that Japan will pay to Pyongyang a significant compensation for the colonial era misrule. However, the unsolved abduction issue has frozen all contacts between the two countries.

On the other hand, the Japanese government also found itself cornered. The strictest set of sanction, introduced after 2002, means that virtually all interactions between Tokyo and Pyongyang have stopped. In essence, Japan has excluded itself from all meaningful negotiations with North Korea, and ceased to be a player in the diplomatic games that take place in and around the Korean Peninsula. This tough stance might placate the Japanese public, but it does nothing to advance Japanese diplomatic interests.

The need for a change of policy has been understood in Japan for years, but now it is a prime minister who can afford such a U-turn without being accused of softness. Prime Minister Shinzo Abe has the iron-clad nationalist credentials, so he can make concessions on an issue so dear to the hearts of Japanese nationalists.

Over the last two years there have been signs that Japan and North Korea are looking for a solution to the stalemate. The recent declaration confirms this. North Koreans have agreed to reconsider the issue - a rather nebulous obligation, to be sure, but still far more than Pyongyang has been willing to consider for a decade. In exchange, the Japanese side will ease sanctions, allowing, for example, regular trips of North Korean ferries to Japanese ports.

Will more steps follow? Will Kim Jong Un decide to do what his father once did with little success - to admit more abductions and return some of the supposedly dead victims to their families, with the expectation that Japan will reciprocate and shower North Korea with aid? And, if so, will such a risky gamble pay off this time? Only time will tell…

Andrei Lankov is professor of Korean Studies at Kookmin University, Seoul. He is the author of "The Real North Korea: Life and Politics in the Failed Stalinist Utopia".

Follow him on Twitter: @andreilankov