Hostage-taking is big business for the pirates and terrorists who increasingly use it as a lucrative source of revenue. But, more surprisingly, business is also booming for risk management experts trying to counter it – and western insurance companies

When James Foley, an experienced war reporter, was kidnapped by four gunmen in north-west Syria on Thanksgiving Day 2012, the chances of getting him out alive were already slim: he was an American and he didn't have insurance. Today, whether a hostage kidnapped by militants overseas is released boils down to nationality and whether or not the hostage works for an employer with "kidnap and ransom", or K&R, insurance.

A 27-year-old German man, who was kidnapped in June 2013 by Isis militants while on a humanitarian aid mission in Syria, was released earlier this summer, German security officials say. To get him out, a "substantial" ransom was paid, according to the German newspaper Die Welt am Sonntag. The German foreign ministry has denied having paid a ransom "in any form".

The ransom demand by Islamic State for Foley and two other American hostages – roughly $132.5m (£80m) and political concessions from the US government – went unanswered; the large size of the ransom demand was likely meant to send a message to the US government and little else, security officials say.

As the risks to westerners working abroad have increased in recent years, K&R insurance has coalesced into a formalised industry with standardised prices, procedures and middlemen. The business has also boomed. Foreign oil, mining and security firms are expanding into new territories and insurance premiums are rising – from $50m (£30m) a decade ago to at least $250m (£150m), according to British security company Control Risks, the largest and best-known crisis response firm. Increased publicity from Somali piracy and Arab spring-related kidnapping cases has accelerated this trend.

Today, at least 75% of Fortune 500 companies hold K&R insurance policies, according to industry estimates obtained from more than a dozen insurers and brokers. Insurers are just now seeing the financial rewards from insuring executives in increasingly risky and remote locales. As one executive at Hiscox put it in a 2009 memo to the company's shareholders: "You should advance to the sound of gunfire." Consultants are hoping to turn what Richard Filon, a British Special Forces officer-turned-negotiator, calls a "black art" into a "more transparent, academic-like business".

In the Sahel region, where the number of foreign oil and mining contractors and aid workers is rising, and where cash-hungry and loosely organised terrorist groups are expanding, the average westerner can be ransomed for $3.75m (£2.25m), according to data provided by Filon's AKE Group, a British security firm that collects kidnap-and-ransom information from around the world. Policies can top £150,000 ($250,000) a year for multinational firms working in dangerous places.

But it's not just that the dollar figures are growing. The increased risk of kidnapping in various hotspots around the world means that more companies want insurance products. "About 70% of them are mining companies going to countries such as Mali or Nigeria, and seeing what's happened in Syria," says Tracie Thompson, a leading kidnap-and-ransom insurance underwriter who recently left her job at AIG to start her own boutique K&R firm in Sydney.

The first kidnap-for-ransom insurance policy, offered by insurance giant Lloyd's of London, dates back to 1932, shortly after the highly publicised kidnapping and murder of US aviator Charles Lindbergh's 20-month-old son by a German ex-convict. But it wasn't until the 1960s and 70s, when a series of Italian bank executives' wives were kidnapped, that K&R insurance evolved from a type of luxury insurance into a full-blown industry.

Fighters of the Ahrar al-Sham brigade, one of the key components of what has evolved into Islamic State. Photograph: Reuters

Still, K&R insurance was a relatively niche market until 2008, when Somali piracy began to attract the attention of the international press. Now, with piracy on the decline due to stepped-up international enforcement, companies are moving into "unknown territories", according to Thompson. "My policies are not off the shelf," she explains. "They are tailored." And, on top of a pricey retainer, they run as high as $1,500 per day, per employee.

But for all the attention paid to K&R in recent years, the industry remains extremely opaque. It is where governments, the world's biggest corporations and nimblest insurance companies interact with known criminals and killers. Governments often refuse to negotiate with armed kidnappers – or if they do, they certainly don't want to broadcast it. And after hostages come home, the involved parties typically sign confidentiality agreements. Details of ransom payments are mostly kept quiet. As a result, little is known about the inner workings of this industry except that it is fuelled by – and partially fuelling – the rise of a new breed of terrorist group that uses kidnapping foreigners as its chief fundraising tool. A New York Times investigation calculated that al-Qaida and its affiliates have earned at least $125m (£75m) in ransom payments in the past five years.

"The nature of the business for some of these companies, whether it be security or oil, lends itself to kidnapping," says Yan Bui, a former kidnap-and-ransom specialist for Clements Worldwide, a Washington-based insurer of American expatriates. "Insurers and crisis responders are just responding to the marketplace."

Kidnap and ransom was once a crime most associated with Latin America, but in just the past five years it has grown into a global business. Africa and Asia-Pacific accounted for more than half of all of the kidnappings recorded worldwide in 2013, according to a report prepared by Control Risks. That's up from just 18% in 2004. Likewise, kidnappings in the Middle East have jumped from 4% of the world total in 2004 to 17% last year. Kidnappings in Latin America have fallen from a 55% share to just 23%.

Even in Africa, where piracy has surged in recent years, the geographical distribution of abductions is beginning to shift. The Horn of Africa, once the global kidnapping capital, has seen a sharp decrease in piracy activities as a result of Nato's counter-piracy taskforce and increased reliance on armed guards by shipping companies. Yet the waters near Ivory Coast and Indonesia are becoming increasingly dangerous, according to a report prepared last year by the piracy security company NYA International. That report shows a dozen ships, many of them fishing vessels and chemical tankers, with hundreds of crewmembers still being held for ransom. It also shows that in 2012, hijacked tankers paid a record-setting $40m (£24m) in ransoms.

James Foley in Aleppo, shortly before he was taken hostage. Photograph: Nicole Tung/freejamesfoley.org

As with most insurance policies, speciality areas such as kidnapping insurance tend to rise and fall along with outbursts of war and conflict. Sales soared in the years following the 9/11 terrorist attacks and the Arab spring revolutions of 2011, according to industry analysts interviewed for this article. But there are no publicly available statistics on the trend.

London-based Control Risks has an exclusive contract with Hiscox and promotional materials provided to clients boast that it has worked on kidnapping cases in more than 120 countries since its founding in 1975. Control Risks noted a 30% increase in kidnapping incidents last year in the Gulf of Guinea, off West Africa.

Fearing heavy losses from an upsurge in hostage cases, insurers have changed their policies since the Arab spring revolutions to require mandatory evacuations from dangerous countries and create so-called "hostage crisis endorsements" in policies, which allow for hostage negotiations to take place when a prisoner exchange or ideological statement has been issued, not just a ransom demand.

"After Libya, insurance companies got absolutely hammered," says Filon, who runs criminal division of AKE. "It may surprise you, but insurance companies didn't fully realise the risks in these places."

AKE, for example, has grown drastically in the past two years and now employs nine full-time "crisis responders", who help clients evacuate their offices and fly them home when security situations worsen. They also offer companies a monthly "kidnap report," which gives a 1-4 score to conflict-riddled countries where contractors and foreign aid groups are working.

Four years ago, executives at Hiscox, the world's biggest kidnapping insurer, set out to change its business model, as the risks to westerners working in the most dangerous of places grew. Somali pirates were pulling in tens of millions of dollars' worth of ransoms and armed gangs had taken hundreds of oil workers hostage across war-torn stretches of Iraq and Afghanistan. In addition to selling kidnapping policies in Europe and Britain, Hiscox opened new offices, took on clients operating in the oil-rich but dangerous Niger delta and worked to write policies for companies working in the long-treacherous Horn of Africa within hours, according to an annual reports prepared for shareholders. Hiscox's headquarters are in Bermuda and the island of Guernsey, primarily for regulatory reasons, and K&R insurance is sold to the customer through intermediaries.

Even as Somali piracy declined last year, Hiscox's K&R business grew as it insured oil pipelines and development projects in dozens of far-flung countries. In late March, Hiscox even created its own in-house counterterrorism team made up of ex-British and French military officers.

As a result, revenue from the company's speciality line, which includes kidnap-and-ransom and terrorism coverage, has generated more than £120m ($200m) in income each year since 2010. Five years ago, a company buying kidnap insurance in Nigeria would need to pay roughly £6,000 ($10,000) a year for £3m ($5m) in coverage, according to Willis Special Contingency Risks, a unit of insurance broker Willis Group Holdings in London. Today, that same coverage goes for about £60,000 ($100,000).

So who is paying all this ransom money? The US and Britain both have strict policies against paying ransoms, though they go mostly unenforced. But many European countries, including Germany, Italy and France, have negotiated with kidnappers to ensure the safe return of their citizens and ransoms have been paid, although these countries deny any direct payments.

• This article was amended on 15 September 2014 to delete quotes attributed to an employee of AKE who disputes that they are an accurate record of the interview or that she was aware the quotes were intended for publication in the Guardian. We apologise for the error. In addition Yan Bui no longer works for Clements Worldwide.