In 1982, a British insurance broker named Doug Milne set out in search of new markets. His speciality was kidnapping and ransom insurance, known in the industry as K&R. Milne enrolled in a Spanish-language course in London, and a month later, with rudimentary skills and only one or two solid contacts on the ground, he boarded a flight to Bogotá. On his first day in the Colombian capital, Milne was walking to a meeting with a potential client when, he recalled, “a guy pulled up alongside and this chap who was walking in front of me, his head just exploded”. It was a drive-by assassination.

Milne cancelled the meeting and spent the afternoon in a bar near Bogotá’s entertainment district. “I missed my meeting and I think I left there about 11pm after having drunk a couple of flagons of Tres Esquinas rum,” Milne told me. He was, of course, horrified. But he also realised that he’d come to the right place. While he knew nothing about the victim or the motive, the murder drove home to him the extent to which Colombian society was at the mercy of criminals and guerillas. His clients needed what he had to offer.

Kidnapping and ransom insurance was created in the 1930s, but it wasn’t until the 60s that it began to really catch on, following a spate of kidnappings in Europe by groups such as Eta in Spain, the Red Army Faction in Germany and the Red Brigades in Italy. The appeal was simple: in the event of a kidnapping, the insurance would provide reimbursement for ransom payment.

The business of kidnapping: inside the secret world of hostage negotiation – podcast Read more

There were caveats to prevent fraud and to ensure that the existence of the policy did not actually increase the risk of kidnapping. The first was that the policy had to be kept secret. In fact, it could be voided if its existence became public. The concern was that if the kidnappers knew of the policy, they would demand more money.

The second principle is that the policy will only reimburse the ransom once it is paid. The insurance company never fronts any money. In order to raise the cash, the victim’s family will probably have to liquidate assets – mortgage the house, sell stocks, pool money from other relatives. This process makes the negotiations credible by dragging them out. This is not just about minimising the payout by the insurance company. Quickly making good on a large ransom raises the expectations of future kidnappers. It can make hostage-taking more lucrative and more common.

When K&R insurance first came on the market, the policyholders were left on their own to negotiate with the kidnappers. But in the mid-1970s, an insurance broker named Julian Radcliffe came up with an idea that would revolutionise the industry. Along with a few colleagues, Radcliffe convinced their company to set up a subsidiary focused on hostage response. The subsidiary, which they named Control Risks, would hire security experts – mostly former military and police – to handle negotiations. The cost of hiring the consultant was included in the policy and borne by the insurance company. In 1982, Control Risks became an independent company.

By the early 80s, hostage-taking was on the rise in Latin America, particularly in Colombia. When Milne arrived in Bogotá, he discovered a vast, untapped market. As an insurance broker, he sold a variety of policies offered by different companies available through the Lloyd’s exchange in London. The job of the broker is to serve the client and to advocate for their interests in the event of a claim. The underwriters represent the insurance companies. Specialised, high-risk policies were placed on the Lloyd’s insurance exchange, and Milne would field offers from different underwriters. He would select the policy that best suited his client.

To his South American clients, Milne was as quintessentially British as James Bond. He attended boarding school in Scotland, and dresses in tailored suits with a perfectly positioned pocket square. He enjoys a stiff drink, sometimes two. “When I went to Colombia, everyone wanted to see me,” Milne told me. “I started with a few contacts, but it grew like Topsy. All their friends at the golf club wanted to meet. It suddenly became a viable business.”

Some grumble that in an industry that values discretion, Milne is a bit of self-promoter. But no one denies his success. By the time he wrapped up his stint in Latin America in the 1990s, he had sold hundreds of new policies, recruited a specialised team in London focusing on the Latin America market, and developed a new service to provide risk mitigation – a “preventative training” programme that educated clients on how to reduce the risk of kidnap and how to respond if it does happen. He then convinced insurance companies that they should foot the bill. (After all, both insurance companies and their clients have an interest in reducing the likelihood of a kidnapping.)

Over the past few decades, the K&R insurance business has grown. More than 75% of Fortune 500 companies have K&R insurance policies. Today two insurers – Hiscox in the UK and AIG in the US – dominate the market, and there are also many security firms that specialise in kidnap response. Hostage negotiation has become something of an industry, with conferences, conventions and shared strategies. More than 97% of kidnappings handled by professional negotiators are successfully resolved through the payment of ransom, according to several different security consultants with access to internal industry data. A small percentage of hostages escape, and a very few are rescued through high-risk operations. Less than 1% are killed.

London is the global centre for K&R insurance, but it has not always been a comfortable fit. Different countries take different approaches to dealing with the kidnapping of their nationals – and the UK, along with the US, has long been a leader of the so-called “no concessions” camp, officially refusing to negotiate with terrorists, pay ransom or make concessions.

In April 1986, Jennifer Guinness, the wife of banker and member of the Guinness brewing family John Guinness, was kidnapped by a gang that demanded a ransom of £2m. She was rescued in a police raid only eight days after being abducted. But the fact that a K&R policy had been triggered and Control Risks were brought in to negotiate a possible ransom sparked outrage. “Private security firms such as the ones called in on the Guinness kidnapping are operating at the very frontiers of official tolerance,” a top police official announced.

Facebook Twitter Pinterest Jennifer Guinness with her husband John Guinness at a press conference following her release, April 1986. Photograph: Independent News and Media/Getty Images

The Thatcher government charged that the insurance industry was fuelling a global kidnapping epidemic, facilitating the payment of ransom and undermining the British no-concessions policy. The logic of that policy is that paying ransom puts a target on the back of British citizens, increasing the risk of future kidnappings. It also puts money into the hands of terrorist organisations, which is used to finance their ongoing operations.

In 1986, the issue of K&R insurance was debated in parliament, which passed a motion expressing concern. There was even talk of working through European institutions to impose a ban on K&R insurance throughout the European Union. Recognising that its existence was under threat, the security industry rallied, arguing that since the policies were kept secret, it was clear that people were not being kidnapped because they had insurance. Rather, they were being taken hostage because they had resources – and banning insurance would not change this. Since the policies only provided reimbursement and were always written for amounts less than the net worth of the policyholder, the industry also argued that insurance did not drive up the amount of payment. What’s more, the industry pointed out that the availability of K&R insurance helped international businesses to manage risk, which in turn allowed companies – including British and European companies – to operate in dangerous environments while exercising appropriate “duty of care” toward their employees.

While the British and European debate eventually wound down without new legislation being introduced, individual countries throughout the world continued to wrestle with how best to respond to kidnapping. Italy, for example, passed a law in 1991 that banned the payment of ransom and the sale of K&R insurance. (One consequence was that the families of kidnapped Italian citizens simply stopped reporting the crimes to the authorities.) Meanwhile, Colombia banned ransom payments, then unbanned, then banned them again. Spain operates a very different policy, where its intelligence services are told to bring hostages home at all costs. Because of its willingness to pay, the country has a tremendous record of success.

Through the debates and policy changes, the K&R industry not only survived but thrived. Then came September 11 2001, which changed the terms of the whole discussion. Rather than challenging the K&R industry as a whole, governments increasingly sought to draw a clearer distinction between criminal groups, to whom ransom could legally be paid, and terror groups, to whom it could not. The US and UK governments both maintained lists of Foreign Terrorist Organizations who could not receive ransom payments. In industry parlance, these groups were designated as “proscribed”.

This attempt to draw distinctions between criminal and terrorist organisations raised many tricky questions. It was clear that K&R policies could not reimburse policyholders who paid a ransom to a terrorist group. But could security consultants handle negotiations? Could they help families to raise and assemble the funds? And what about the families themselves? Would they be held legally liable for paying ransom to terrorists? “It’s all a grey area,” Milne acknowledged.

Further complicating the process is the fact that kidnappers often try to hide their identity. Hostage negotiators told me that some terror groups pretend to be criminal organisations so they can collect ransoms. The opposite also occurs. Criminal groups who are ignorant of the legal prohibitions sometimes pretend to be terror organisations in the hopes that the fearsome reputation of these groups will push negotiations along. Under the law, the onus is on the insurance company to demonstrate that kidnappers are “proscribed” in order to invalidate the policy. Negotiators working for the victim’s family would sometimes refrain from asking obvious questions about the group holding the hostage. They simply preferred not to know.

Meanwhile, decisions about which groups were designated as terrorists were often politically determined and sometimes arbitrary. For example, a 2011 case, Masefield AG v Amlin Corporate Member, determined that the payment of ransom to Somali pirates was legal under British law. As a result, Somali pirates were presumed to be criminals rather than terrorists, even when ties to al-Shabaab militants were alleged. Meanwhile, it was illegal to pay ransom to a criminally oriented kidnapping cell in Nigeria if they were seen to have ties to proscribed groups such as Boko Haram.

In effect, the collision of disparate national policies and the insurance market creates complexities that determine who lives and who dies in international kidnapping cases. In 2008, a Canadian journalist, Amanda Lindhout, was kidnapped in Somalia, along with an Australian colleague, Nigel Brennan. As young freelancers, they did not have insurance. Officially, neither Canada nor Australia pay ransom. Driven by desperation, their families found a way forward.

The one factor in their favour was that the group that kidnapped the pair was a criminal and not a terror organisation. Because they were not “terrorists,” the Canadian government entered into negotiations, offering to build a school or provide development aid in exchange for Lindhout’s release. But the kidnappers wanted cash. They tortured Lindhout to put more pressure on her family, which had few resources. Realising that the negotiations were going nowhere, Lindhout’s mother, Lorinda Stewart, decided that the only hope was to pay a ransom. Canadian officials warned Stewart that paying ransom was against the law and that she could be prosecuted for doing so, but she forged ahead.

Once Stewart made the decision, the Royal Canadian Mounted Police, which had been handling the case, withdrew all support. The hostage-negotiation team that had been camped in her living room moved out. Stewart, working with the Brennan family in Australia, eventually raised enough money to hire a security consultant from London-based firm AKE to take over the negotiations. The consultant advised the families that negotiations would take several months and they would have to pay a ransom of around $600,000 each. His prediction was spot on. Lindhout and Brennan were freed in November 2009. Their families ended up in massive debt.

Facebook Twitter Pinterest Amanda Lindhout and Nigel Brennan in Mogadishu, Somalia, after their release, November 2009. Photograph: Farah Abdi Warsameh/AP

In some cases, US and British hostages have also been freed through dramatic rescues. On 25 January 2012, US Navy Seals dropped into Somalia and rescued kidnapped US aid worker Jessica Buchanan and her Danish colleague Poul Thisted, shooting dead nine of their kidnappers. Officials had decided to launch a rescue because Buchanan had developed a kidney infection, and they believed her life was in danger. Negotiations, which were being carried out by a security consultant and monitored by the FBI, were not progressing fast enough. Most importantly, the US had good intelligence on the hostages’ location and ideal weather conditions for a successful rescue.

Despite such successes, rescuing hostages through military force is not a scalable solution to international hostage-taking. Only a handful of countries have the military capacity to pull off such a raid, and they are also extremely risky. According to industry data, either a hostage or a rescuer is killed in half of all rescue operations. One tragic example was the 2009 raid carried out by British special forces in Afghanistan that freed kidnapped New York Times reporter Steve Farrell, but led to the death of a British soldier along with two Afghan civilians. Farrell’s Afghan colleague, journalist Sultan Munadi, was also killed, possibly shot accidentally by British forces. These deaths were all the more tragic because private negotiators who were communicating with the kidnappers already had a deal for both hostages’ release. It was not clear that the British government was ever aware.

In September 2011, a dramatic case tested the resolve of the British authorities’ no-concession policy, when a British couple, Judith (“Jude”) and David Tebbutt, were attacked by Somali pirates while vacationing in Kenya. Their son Ollie, a 25-year-old furniture designer, was at a job site in Glasgow when a colleague came to tell him that the police wanted to see him. “Because my parents were on holiday, I assumed something bad had happened, like maybe a car crash,” Tebbutt recalled.

After a week-long safari, his parents had booked a stay in a secluded resort called the Kiwayu Safari Village on the Kenyan coast. One night, Somali kidnappers raided the property and abducted his mother. His father David was killed trying to resist.

For weeks after being given the news, Ollie was in close contact with the Foreign Office. He was also visited by representatives from SO15, the counterterrorism command of the British Metropolitan police, who were investigating the possible involvement of members of al-Shabaab in the abduction. Eventually, Ollie was able to arrange a proof-of-life telephone call with his mother. The kidnappers wanted a huge sum – around $10m. But Ollie discovered that tucked into a travel insurance policy obtained through his father’s work was a clause that provided kidnapping and ransom insurance. “It was incredibly lucky, really,” Ollie said.

Through the policy, two security consultants from Control Risks were assigned to the case. “That’s when the government said: ‘You have to make a choice – it’s either us or them.’” Ollie recalled. He found the security consultants to be sober professionals. They explained how the negotiations would work, and that the sole focus would on getting his mother back alive. “They were very much like, we do this every day, and this is expected in this part of the world, and this is our pattern for what a Somali kidnap looks like,” he said.

Meanwhile, the government representatives explained that Britain did not pay ransom, and would not countenance any arrangement that put money in the hands of terrorists. While the identity of the kidnappers was murky, the line between al-Shabaab militants, pirates and criminals in Somalia was a fluid one. The best that British officials could offer the family was to essentially walk away – to put what they called “clear water” between the government and any negotiations. Their rather charitable interpretation was that since the kidnappers were demanding money they had to be criminals.

As an only child, Ollie was the family’s point person in the negotiations. He moved into his parents’ home, and over the next six months negotiations were carried out around the kitchen table. They were surprisingly orderly. After each phone call, the kidnappers would arrange a follow-up conversation. Generally, they kept their appointments. A Control Risks consultant would brief him on what to say, and sit by his side. A representative from the Metropolitan police monitored the discussions, but did not participate or interfere.

The Tebbutts were a comfortable middle-class family, but did not have millions. Ollie found the kidnappers had a pretty good sense of the value of their hostage, and over the next few months, under the guidance of the Control Risks negotiator, their demands steadily dropped. They finally agreed to accept a ransom of around £600,000. The only way Ollie was able to come up with that sum was to use the death benefits he received following his father’s murder.

Facebook Twitter Pinterest Judith Tebbutt at Adado airport after she was released in central Somalia, March 2012. Photograph: AFP/Getty Images

In March 2012, Ollie and the negotiator travelled to Nairobi to make the final arrangements. Control Risks contracted a pilot to drop the money, but there were some tense moments when the authorities that controlled the local airstrip demanded a larger cut. Once that was worked out, and the kidnappers indicated they were prepared to release their hostage, things changed. “At that point, the security consultant drove me to a crossroads in Nairobi,” he recalled. “On one side there was a jeep with the Foreign Office guys in it, and I just crossed the road and got in their car. That was the last time I saw anyone from Control Risks.”

From that point on, the British government took over. British officials travelled to the Nairobi airport to collect Jude, and then took her to the British high commission, where she was reunited with her son. Eventually, the full ransom that Ollie had paid was reimbursed under the terms of the K&R policy.

On the one hand, Ollie is grateful that the British government allowed him to pay a ransom despite the fact that his mother’s kidnappers may have been linked to al-Shabaab. (A British researcher told me that he visited the FCO office to discuss the case while Jude was being held, and was told the government was not interested in hearing any information about the terrorist ties of the kidnappers.) On the other hand, his experience caused him to focus on what he sees as the hypocrisy and heartlessness of the government’s position. In order to apply pressure, Jude’s kidnappers were depriving her of food, slowly starving her to death. If negotiations had dragged on for a few more months, “she would have died for sure,” Ollie believes.

Ollie is soft-spoken and understated, but he told me he thinks the British government’s policy is “so crazy”. He continued: “It makes absolutely no sense. I don’t believe for a second that the kidnappers are checking passports or trying to figure out who is from where. They just grab whoever they can. I don’t think the British policy protects people in a way that they claim it does, but they are so entrenched in this idea. The idea that they get to choose who a terrorist group is based on pretty flimsy reasons sometimes. At the same time, governments sell weapons or trade with regimes that are incredibly bad.”

The logic of the no-concessions policy, he believes, is that a certain number of hostages must die in order for the government to show its resolve. If the British government had designated Jude Tebbutt’s kidnappers as terrorists, he says, “my mum would not have come home.”

Security consultants and private negotiators fill a critical role in hostage recovery, and have an undeniable record of success in criminal cases. They can even do some things that governments can’t, such as credibly claim limited resources as a strategy to get the price down. Governments, of course, can’t plead poverty.

Yet the whole system, as imperfect as it already is, breaks down in terrorism-related cases. If the victim is from a “no-concessions” country, security consultants can offer only limited support. If the victim is from a country that negotiates, such as France or Spain, the private security consultant is generally asked to step aside while national intelligence agencies takes over. While the security consultants are pleased to see their clients come home, they are not happy about the massive payouts.

“The market is now too inflated,” one experienced security consultant told me. “Governments have deep pockets and are basically unable to do what a traditional K&R consultant would do, which is to put up resistance, to claim an inability to pay, to bargain, to try and disincentivise the crime.”

Such arguments may seem self-serving, but there is evidence to support them. In one case, a New York Times journalist who was captured in Afghanistan in 2008 tried to argue with his captors, who were demanding $25m and the release of 15 prisoners. He told them they were out of touch. They countered that the French had recently paid $38m for the release of an aid worker, and that an Italian journalist had been ransomed for $15m and the release of several prisoners. Quickly capitulating to high ransom demands – as some European and Asian governments have done – makes kidnapping more attractive and lucrative around the world. While governments might make a distinction between proscribed and criminal groups, kidnappers don’t. And so the markets are inextricably linked.

So what should governments do? If the goal is to bring the hostages home safely while reducing the threat of future kidnapping and minimise the money flowing to terrorist groups, then there are legitimate questions about whether the no-concessions policy is achieving the desired result.

My 977 days held hostage by Somali pirates | Michael Scott Moore Read more

First, a series of studies carried out in recent years provide little evidence that kidnapping victims are targeted according to nationality. Thus refusing to pay ransom does not appear to reduce the incidence of the crime, but does greatly increase the likelihood that the victim will be killed. Perversely, in the current environment, it may actually increase the money flowing to terrorists, because kidnappers can execute their British and American hostages, who because of the no-concessions policy have little value, as means of putting pressure on the European countries that pay multi-million-dollar ransoms. This is why the problem must be defined differently and more narrowly. How can western countries work together not to stop ransom payments to terrorist kidnappers – which is an unattainable goal – but rather to minimise their size? Clearly, the K&R industry has a key role to play.

However, the benefits that security consultants provide are only available if people have access to kidnapping and ransom insurance. Governments should work with the industry to develop innovative ways to extend coverage to vulnerable groups. K&R coverage can be tucked into travel-insurance policies or provided through employers whose staff operate internationally. For high-risk groups, such as freelance journalists and volunteer aid workers in conflict zones, the challenge is more difficult. But governments can work with the industry to develop specialised products, even if these require public subsidies.

Finally, the families of kidnapping victims who lack insurance should be given access to security consultants in extraordinary circumstances, such as those with national security implications. This can be done by creating an industry-wide pro bono standard, whereby security companies increase their volunteer services. It could also be done via some sort of government pool that needy families can access if certain standards are met.

When I met with representatives from the Foreign Office, I found them dug in to their view that the no-concessions policy was simply not up for discussion or debate. This is a shame, because data made available in recent years does not support its central contention, namely that refusing to pay ransom reduces the risk of future crime. Moreover, as Doug Milne notes: “By refusing to allow concessions, you drive the negotiations underground, so the intelligence you might otherwise get following the case disappears. The families feel they could potentially be prosecuted. And the government becomes the enemy.” That’s a bad outcome that does not serve the hostage, or advance Britain’s national security. It’s time to consider a new approach.

Adapted from We Want to Negotiate: The Secret World of Kidnapping, Hostages, and Ransom, published by Columbia Global Reports and available to buy at guardianbookshop.co.uk

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.