But Mr. Yu’s sister later testified in court that officials had beaten and coerced her into making false confessions against her brother while they held her without legal representation in solitary confinement at the agency’s interrogation center for 179 days. The agency denied using coercion at the center, south of Seoul, where it screens fresh arrivals from the North for up to six months to ferret out spies.

Photos that the agency presented as evidence in court, saying they had been taken while Mr. Yu was secretly visiting North Korea in 2012, turned out to have been taken in China. The Chinese immigration documents that the agency said had recorded Mr. Yu’s border crossings into North Korea were found to have been faked. A Korean-Chinese man stabbed himself in a Seoul hotel room after leaving a suicide note saying the agency had promised to pay him to fabricate the documents. The officer who hired him also tried to kill himself by inhaling carbon monoxide in his car. (Neither man died.)

The court threw out the espionage charge against Mr. Yu.

While all this was unfolding in 2014, the agency announced the arrest of another suspected spy, also a defector from the North. But the man, Hong Kang-cheol, walked free after a court declared his confessions invalid because he had not been informed of his right to remain silent and consult a lawyer.

Mr. Hong said he had been held in solitary confinement for 84 days and forced to write draft after draft of a confession until a fictional version emerged that satisfied his interrogators.

“I had no freedom to meet visitors, no freedom to move, completely isolated from the outside,” Mr. Hong said after an appeals court upheld his acquittal in February.

Over the years, the intelligence agency has repeatedly pledged not to abuse its power or act as a political tool of presidents. But scandals at the secretive agency have frequently rocked the country, with several of its directors ending up in jail.

The agency declined to comment on “Spy Nation,” which won the top documentary award at the Jeonju International Film Festival in South Korea in May.