With the flow of information from North Korea fiercely controlled, outsiders have long relied on defector testimonies to gain an understanding of what goes on inside the secretive state.

But relying on the anecdotes of individuals – all with different views and experiences – can also be risky.

In a report released last year, the UN accused the North Korean leader, Kim Jong-un, of crimes against humanity, and called for the case to be referred to the international criminal court.

UN investigators had been denied access to the country, so the organisation had instead carried out 240 confidential interviews with North Korean refugees living in South Korea, Japan, the UK and the US, including Shin Dong-hyuk, whose story was told in the bestselling Escape from Camp 14.

In January, the DPRK government released a video claiming to show Shin’s father denouncing his son’s stories as fake. When questioned, Shin confessed that parts of his account were also inaccurate, including sections on his time in Camp 14, the infamous labour camp for political prisoners, and the age at which he was tortured.

Shin is not alone. Another North Korean, Lee Soon-ok, offered testimony to the US House of Representatives in 2004, describing torture and the killing of Christians in hot iron liquid in a North Korean political prison.

There are questions to be asked about how heavily outsiders should rely on defectors’ testimonies as credible evidence Jiyoung Song

But Lee’s testimony was challenged by Chang In-suk, then head of the North Korean Defectors’ Association in Seoul, who claimed to know first hand that Lee had never been a political prisoner. Many former DPRK citizens on the website NKnet agreed Lee’s accounts were unlikely to be true.

Similarly, Kwon Hyuk told the US Congress that he was an intelligence officer at the DPRK embassy in Beijing and had witnessed human experiments in political prisons – a critical factor in the US decision to pass the North Korea Human Rights Act in 2004.

Kwon’s account, retold in a BBC documentary back in 2004, was later questioned by South Korea’s Yonhap news agency, which argued that he never had access to such information. Many years later, Kwon has since disappeared from the public eye.

While there is no doubt the North Korean regime has committed serious human rights abuses, there are questions to be asked about how heavily outsiders should rely on defectors’ testimonies as credible evidence.

North Korea refugee and human rights activist Shin Dong-hyuk at a rally outside the White House in 2012. Photograph: Chip Somodevilla/Getty Images

I have been interviewing North Koreans as a DPRK watcher and human rights researcher since 1999. What I’ve found suggests there are serious ethical dilemmas in the way we gather information.

What is behind the inconsistencies?

Cash payments in return for interviews with North Korean refugees have been standard practice in the field for years.

Initially, the payment was to cover the cost of meals and local transport, which was approximately $30 in the late 1990s when I first began interviewing in China and South Korea. However, the fees had risen to $200 per hour by the time I attempted to interview people from North Korea in May 2014.

A government official from the South Korean ministry of unification told me the range of fees could vary wildly, from $50-500 per hour, depending on the quality of information.

But this practice raises a difficulty: how does the payment change the relation between a researcher and an interviewee, and what effect will it have on the story itself?

North Korean refugees are ​well aware of what the interviewer wants to hear

This practice also drives the demand for “saleable stories”: the more exclusive, shocking or emotional, the higher the fee.

Shin, the son of a political prisoner, doubtlessly had a horrific time inside the country, and is thought to be the only prisoner to have successfully escaped from the “total control zone” internment camp.

After his escape, he met former US president George W Bush, and his story was made into a documentary. Though parts of his story have been challenged by counter-testimony and contradictions, the power of the west’s curiosity and the media’s desire for a good story cannot be overestimated.

North Korean refugees have become well aware of what the interviewer wants to hear. Whether speaking to the UN, US Congress or western media, the questions are the same every time: why did you leave North Korea, and how terrible is it?

With the number of defectors reaching 20,000 in 2010, first-person testimonies have become the norm, and have increasingly come to involve younger victims with more tragic, dramatic, visual and emotional accounts.

This raises another issue, because in my experience one-on-one interviews tend to generate more exaggerated stories. Although there are ways to confirm information through cross-examination and by consulting multiple sources, these methods are highly time-consuming, while a significant amount of the information disclosed by a single source is simply unverifiable given the fiercely secretive nature of the DPRK.

A woman and child stand in a Pyongyang subway station. Photograph: Ed Jones/AFP/Getty Images

Pressure

In my 16 years studying North Korean refugees, I have experienced numerous inconsistent stories, some intentional omission and occasionally, some lies. In one case the breach of trust was so significant that I could not continue my research – not just because it affected my professional capacity to analyse and deliver credible stories in an ethical way, but because it also had a deep impact on the personal trust I had invested in subjects I had sincerely cared about.

But many refugees say they feel pressured for defector stories. Ahn Myung-chol, a former prison guard at Camp 22, said people liked shocking stories and these so-called “defector-activists” were merely responding to this desire. Chong Kwang-il, a former prisoner at Camp 15, said the fame brought by media exposure trapped them, forcing them to reproduce a certain narrative.

Choi Sung-chol, from the Korean Nationality Residents Association, said the line between small and large inconsistencies was often hard to draw: “Most North Koreans do not worry about small factual mistakes as long as the big picture that North Korea violates human rights is right.”

Choi added: “We, North Koreans, know what is true and what is fake, but at the same time we do not want to ruin the bigger political moves like the UN committee of investigation or the US Human Rights Act.”

Versions of this article originally appeared on Asia & the Pacific Policy Society’s Policy Forum and on Song Jiyoung’s blog. A Korean version of this article is available at NK News Korean