Once upon a time in a land far, far away, there grew a magical fruit. This fruit could be squeezed to produce a very special kind of oil that made cookies more healthy, soap more bubbly and crisps more crispy. The oil could even make lipstick smoother and keep ice-cream from melting. Because of these wondrous qualities, people came from around the world to buy the fruit and its oil.

In the places where the fruit came from, people burned down the forest so they could plant more trees that grew the fruit – making lots of nasty smoke and sending all of the creatures of the forest scurrying away. When the trees were burned, they emitted a gas that heated up the air. Then everybody was upset, because they loved the forest’s creatures and thought the temperature was warm enough already. A few people decided they shouldn’t use the oil any more, but mostly things went on as before, and the forest kept burning.

This is a true story. Except that it is not magic. The fruit of the oil palm tree (Elaeis guineensis), which grows in tropical climates, contains the world’s most versatile vegetable oil. It can handle frying without spoiling, and blends well with other oils. Its combination of different types of fats and its consistency after refining make it a popular ingredient in packaged baked goods. Its low production costs make it cheaper than frying oils such as cottonseed or sunflower. It provides the foaming agent in virtually every shampoo, liquid soap or detergent. Cosmetics manufacturers prefer it to animal tallow for its ease of application and low price. It is increasingly used as a cheap raw material for biofuels, especially in the European Union. It functions as a natural preservative in processed foods, and actually does raise the melting point of ice-cream. Palm oil can be used as an adhesive that binds together the particles in fibreboard. Oil palm trunks and fronds can be made into everything from plywood to the composite body of Malaysia’s national automobile.

Worldwide production of palm oil has been climbing steadily for five decades. Between 1995 and 2015, annual production quadrupled, from 15.2m tonnes to 62.6m tonnes. By 2050, it is expected to quadruple again, reaching 240m tonnes. The footprint of palm oil production is astounding: plantations to produce it account for 10% of permanent global cropland. Today, 3 billion people in 150 countries use products containing palm oil. Globally, we each consume an average of 8kg of palm oil a year.

Of this, 85% comes from Malaysia and Indonesia, where worldwide demand for palm oil has lifted incomes, especially in rural areas – but at the cost of tremendous environmental devastation and often with attendant labour and human rights abuses. Fires set to clear forests and create land for more palm plantations are the top source of greenhouse gas emissions in Indonesia, a country of 261 million people. The financial incentive to produce more palm oil is helping to warm the planet, while destroying the only habitat of Sumatran tigers, Sumatran rhinos and orangutans – driving them towards extinction.

Yet consumers are often unaware they are even using the stuff. Palm Oil Investigations, which dubs itself “the palm oil watchdog”, lists more than 200 common ingredients in food and home and personal care products containing palm oil, only about 10% of which include the telltale word “palm”.

How did palm oil insinuate itself into every corner of our lives? No single innovation caused palm oil consumption to soar. Instead, it was the perfect commodity at the right moment for industry after industry, each of which adopted it to replace ingredients and never turned back. At the same time, producing nations view palm oil as a poverty-reduction scheme, while international finance organisations view it as a growth engine for developing economies. The International Monetary Fund (IMF) has pushed Malaysia and Indonesia to produce more.

As the palm industry expanded, conservationists and environmental organisations such as Greenpeace started to raise the alarm about its devastating effects on carbon emissions and wildlife habitat. (However, it is not impossible to produce palm oil sustainably, and several organisations certify sustainable producers.) In response, a backlash against palm oil has developed: last April, the supermarket Iceland pledged that it would cut palm oil from all its own-brand foods by the end of 2018. In December, Norway banned imports for biofuel production.

But by the time awareness of palm oil’s impact had spread, it was so deeply embedded in the consumer economy that it now may be too late to remove it. (Tellingly, Iceland found it impossible to fulfill its 2018 pledge. Instead, the company ended up removing its branding from foods containing palm oil rather than removing palm oil from all of its branded foods.)

Determining which products contain palm oil, let alone how sustainably it has been sourced, requires an almost supernatural level of consumer consciousness. In any case, greater consumer awareness in the west will not have much impact, given that Europe and the US account for less than 14% of global demand. More than half of global demand comes from Asia.

It was a good 20 years after the first alarms about deforestation in Brazil that consumer action slowed – not stopped – the destruction. With palm oil, “the reality is that the western part of world is [a small share] of palm oil consumption, and the rest of the world doesn’t give a shit”, said Neil Blomquist, managing director of Colorado-based Natural Habitats, which produces palm oil in Ecuador and Sierra Leone to the highest level of sustainability certification. “So there’s not much incentive to change.”

Palm oil’s world domination is the result of five factors: first, it has replaced less healthy fats in foods in the west. Second, producers have pushed to keep its price low. Third, it has replaced more expensive oils in home and personal care products. Fourth, again because it is cheap, it has been widely adopted as cooking oil in Asian countries. Finally, as those Asian countries have grown richer, they have begun to consume more fat, much of it in the form of palm oil.

Widespread adoption of palm oil began with processed foods. In the 1960s, scientists began to warn that butter’s high saturated fat content may increase the risk of heart disease. Food manufacturers, including the British-Dutch conglomerate Unilever, began to replace it with margarine, made with vegetable oils low in saturated fat. By the early 1990s, though, it became clear that the process by which the oils in margarine were made, known as partial hydrogenation, actually created a different kind of fat – trans fat – that was even unhealthier than saturated fat. The Unilever board of directors saw a scientific consensus forming against trans fat and decided to get rid of it. “Unilever was always very conscious of the health interests of consumers of its products,” said James W Kinnear, a Unilever board member at the time.

The switchover happened suddenly. In 1994, a Unilever refineries manager named Gerrit van Duijn received a call from his bosses in Rotterdam. Twenty Unilever plants in 15 countries needed to remove partially hydrogenated oils from 600 fat blends and replace them with trans-fat free components.

A fire at an oil palm plantation in Pekanbaru, Sumatra, due to intensive farming methods and the dry season. Photograph: AFP/Getty

The project, for reasons Van Duijn can’t explain, was called “Paddington”. First, he needed to figure out what could replace trans fat while maintaining its favourable properties, such as remaining solid at room temperature – a necessity for inexpensive butter substitutes as well as manufactured goods such as cookies. In the end, there was only one choice: oil from the oil palm tree – either palm oil (extracted from the fruit) or palm kernel oil (from the seed). No other oil could be refined to the consistency needed for Unilever’s various margarine blends and baked goods without producing trans fat. It was the only alternative to partially hydrogenated oils, Van Duijn told me. Palm oil and palm kernel oil were also lower in saturated fat than butter.

The switch at each plant had to occur simultaneously – the production lines couldn’t handle a mixture of the old oils and the new. “On a certain day, all these tanks had to be emptied of trans-containing components and refilled with trans-free components,” Van Duijn said. “Logistically, that was quite a nightmare.” (Purchasing additional tanks would have been too expensive.)

As Unilever had sometimes used palm oil previously, a supply chain was already up and running. But it took six weeks for the raw material to be shipped from Malaysia to Europe, and Van Duijn had three months to make the switch. He started to buy more and more palm and palm kernel oils, arranging for the shipments to be trucked in to the various plants on schedule. Then one day in 1995, with trucks queued up outside Unilever plants across Europe, it was done.

It was a moment that changed the processed food industry forever. Unilever was the trailblazer; after Van Duijn organised the company’s switch to palm oil, virtually every other food manufacturer followed. In 2001, the American Heart Association issued a statement declaring that “the optimal diet for reducing risk of chronic diseases is one in which saturated fatty acids are reduced and trans fatty acids from manufactured fats are virtually eliminated”. Today, more than two-thirds of palm oil goes into food. Consumption in the EU more than tripled between Project Paddington and 2015. That same year, the US Food and Drug Administration (FDA) gave food manufacturers three years to get rid of all trans fats from every margarine, cookie, cake, pie, popcorn, frozen pizza, doughnut and biscuit sold in the US. Virtually all of it has now been replaced with palm oil.

For all the palm oil that now goes into food in Europe and the US, Asia uses far more: India, China, and Indonesia account for nearly 40% of all palm oil consumed worldwide. Where they once cooked with soya oil, palm oil has replaced it. Growth has been fastest in India, where an accelerating economy has been another factor in palm oil’s newfound popularity.

One of the commonalities of economic development, across the globe and throughout history, is that a population’s consumption of fat grows in lockstep with its income, and the subcontinent has been no exception. Between 1993 and 2013, Indian per capita GDP expanded from $298 to $1,452. Over the same period, fat consumption in rural areas grew by 35% and in urban areas by 25%, and palm oil has been a major ingredient in this escalation. Government-subsidised “fair price shops”, a food distribution network for the poor, started selling imported palm oil in 1978, mainly for cooking; two years later the 290,000 shops were unloading 273,500 tonnes. By 1995, Indian imports of palm oil had climbed to nearly 1m tonnes, reaching more than 9m tonnes by 2015. In those years, the poverty rate fell by half while the population climbed by 36%.

But palm oil is no longer just used for home cooking in India – today it is a big part of the country’s growing junk food industry. India’s fast food market grew 83% just between 2011 and 2016. Between them, Domino’s Pizza, Subway, Pizza Hut, KFC, McDonald’s and Dunkin’ Donuts – all of which use palm oil – now have 2,784 stores in the country, according to reporting by the Nation. Packaged-food sales spiked 138% over roughly the same period; for pennies, you can buy dozens of packaged snacks containing palm oil.

Palm oil’s versatility is not limited to food. Unlike other oils, it can be easily and inexpensively “fractionated” – separated into oils of different consistencies – which disposes it to multiple uses. “It has a tremendous advantage because of its versatility,” said Carl Bek-Nielsen, chief executive director of United Plantations Berhad, a Malaysia-based palm oil producer.

Not long after the processed food business discovered the magical properties of palm oil, industries as diverse as personal care products and transport fuel would also begin using it to replace other oils. But just as trans fats were chosen for perceived benefits, only to turn out to be worse than what they had replaced, palm oil was initially adopted in large part for its perceived environmental friendliness.

As palm oil became more widely used in food the world over, it was also replacing animal products in cleaning products and personal care items such as soap, shampoo, lotion and makeup. Today, 70% of personal care items contain one or more palm oil derivatives.

Historically, soap often came from animal tallow, and shampoo, which originated on the Indian subcontinent, was first made with plant-based surfactants (a substance that acts as a detergent, emulsifier or foaming agent). Later, synthetic ingredients came into favour, with animal tallow joining them in the 20th century. In the 1980s, the personal care industry started to notice a consumer preference for “natural” ingredients, which “a lot of consumers figured was synonymous with plant-based rather than animal-based”, said Chris Sayner, vice-president for corporate sustainability at Croda, a chemical company. Croda’s customers started asking if it could come up with plant-based surfactant formulations without tallow.

Just as Van Duijn had discovered at Unilever, the composition of palm oil and palm kernel oil made them the perfect substitute. Manufacturers looking for alternatives found that palm and palm kernel oils contain the same set of fat types as tallow. No other alternative could provide the same advantages across such a wide range of products. “Alternate sources [to tallow] were looked at,” Sayner recalled. “Palm and palm kernel oil dropped in as the replacement.”

A deforested palm oil concession in Papua, Indonesia.

Photograph: Ulet Ifansasti/Greenpeace

Sayner believes that the BSE outbreak of the early 1990s, when a brain disease among cattle spread to some people who ate beef, triggered a larger shift in consumption habits. “Public opinion, brand equity and marketing all came together to move away from animal-based products in more fashion-oriented industries like personal care.” Companies across Europe and the US that Croda supplied started to make the switch.

The change from animal-based fats to palm oil came with a certain irony. In the past, when tallow was used in products such as soaps, a byproduct of the meat industry – animal fat – was put to good use. Now, in response to consumers’ desire for ingredients perceived as more “natural,” the makers of soaps, detergents and cosmetics have replaced a local waste product with one that must be transported thousands of miles and that causes environmental destruction in the countries where it is produced. (Although, of course, the meat industry comes with its own environmental harm.) “What’s better environmentally than using a byproduct that exists on your doorstep?” Sayner asked.

A similar thing happened with biofuels – the intent to reduce environmental harm had unintended consequences. In 1997, a European commission report called for increasing the percentage of total energy consumption from renewable sources. Three years later it cited the environmental benefits of biofuels for transport, and in 2009 adopted the Renewable Energy Directive (RED), which included a 10% target for the share of transport fuels coming from biofuels by 2020.

Unlike in food and home and personal care products, where palm’s chemical makeup makes it the perfect alternative, when it comes to biofuel, palm, soybean, rapeseed and sunflower oils all perform equally well. But palm has one big advantage over these rival oils: price.

The EU policies “created an unprecedented market for the uptake of palm oil,” said Kalyana Sundram, CEO of the Malaysian Palm Oil Council, a trade group. Legislative attempts in the west to curb the environmental harm of fossil fuels – the US adopted its own biofuel mandate in 2007 – had severe environmental consequences in less developed countries, contributing significantly to global warming.

EU palm oil imports shot up 15% the year after the RED, reaching an all-time high, and 19% the year after that, as biofuel use tripled in the EU between 2011 and 2014; palm oil’s share of biofuel raw material leapt fivefold in that period. Half of the EU’s palm oil now goes to biofuel, double the share prior to the RED. Sustainability criteria were later added – although Oxfam and others have criticised their effectiveness – and earlier this month European commissioners proposed new limits on biofuel crops tied to deforestation. But the damage had already been done.

The oil palm is blessed with many attributes that have helped it on its path to dominance. It is perennial and evergreen, enabling year-round production. It is exceptionally efficient at photosynthesis for a perennial tree, and requires less preparation of the soil than other sources of vegetable oils, reducing costs. It can succeed in soils that can’t sustain other crops. Most importantly, it gives the highest yield per acre of any oilseed crop – almost five times as much oil per acre as rapeseed, almost six times as much as sunflower and more than eight times as much as soybeans. Boycotts of palm oil will only lead to its replacement by other crops needing far more farmland and likely more deforestation.

“The cost of production is far less than any compared [comparable] vegetable or animal fat,” said Sundram, of the Malaysian Palm Oil Council. “Industry is simply palming off the benefits to the consumer.”

For decades, palm’s production advantage went unrealised, until a Scot named Leslie Davidson instigated perhaps the most significant innovation in the industry’s history. Davidson had come to British Malaya in 1951 at the age of 20 to work on a Unilever plantation. Four years later, the company transferred him to Cameroon. The oil palm originated in west Africa, and had been introduced from there to Malaysia in 1875. In Cameroon, Davidson noticed that insects resembling rice weevils surrounded palm fruits. In Malaya, the plantations were employing hundreds of people to hand-pollinate the flowers, yet pollination occurred more efficiently in Cameroon.

When Unilever sent Davidson back to Malaya (now Malaysia) in 1960, he told his bosses he thought the Malaysian industry was going about pollination all wrong, and that insects were the natural pollinators of oil palm. “They told him to mind your own business and don’t get involved,” said Carl Bek-Nielsen, who knew Davidson.

In 1974, Davidson became vice-chair of Unilever International Plantations Group. He recruited three entomologists, led by the Pakistani scientist Rahman Syed, who travelled to Cameroon to investigate. Eventually Syed determined that Davidson’s hunch was correct: a particular species of weevil was pollinating the oil palm trees, and Davidson received permission from the Malaysian government to import some.

On 21 February, 1981, 2,000 Elaeidobius kamerunicus were released at Unilever’s Mamor estate in Johor. Results were seen immediately, with no adverse effects, and the pollinating weevils were distributed across Malaysia. The following year, the country saw an increase in yield of 400,000 tonnes of palm oil and 300,000 tonnes of palm kernels.

The new pollination technique was a key factor in palm oil’s growth. As yields climbed, and the cost of labour to manually pollinate the trees was more efficiently deployed for picking the fruit, there was an explosion in the volume of land devoted to oil-palm plantations. Davidson had radically changed the future of Malaysia and Indonesia.

A worker with palm oil fruits at a plantation in Mamuju, Indonesia. Photograph: Antara Foto Agency/Reuters

But the changes wouldn’t have occurred without pushes from policymakers in both countries. “We’ve seen a lot of effort from both governments into supporting the sector because it’s an easy way to develop the economy,” said Raquel Moreno-Peñaranda, research fellow at United Nations University’s Institute of Advanced Studies in Tokyo, who studies agricultural systems and advises governments. Malaysia’s minister of primary industries, Teresa Kok, told the European Palm Oil Conference in Madrid in October: “Palm oil is synonymous with poverty eradication.” Malaysia began its programme to boost palm exports as a means of poverty reduction in 1961, four years after independence from Britain. Rubber had been a key crop, but with prices falling, the government initiated a programme to replace rubber plantations with oil palm. In 1968, Malaysia provided palm oil producers with a series of tax breaks. Industry subsequently invested heavily in milling technology to extract the oil from the fruit. In the early 1970s, fractionation was developed, expanding the applications of palm oil for both food and other uses.

More recently, plantation owners have found profitable uses for waste such as empty fruit bunches, palm fronds, palm fruit peels, and palm kernel shells. Mill effluent that was once dumped into nearby streams now produces electricity. These new revenue streams reduce planters’ risk by providing income even when palm oil prices are down (such as right now), and have helped them face headwinds such as the increasing costs of labour and fertiliser.

But the push for increased palm oil production has not only come from inside Malaysia and Indonesia. World Bank policies in the 1970s encouraged the Indonesian government to expand palm among small farmers. The 1998 economic crisis in Asia shattered exports of manufactured goods from the region, but commodity exports, which were sold in dollars, “came in like a life vest in rough choppy seas,” Bek-Nielsen recalled. The IMF’s bailout package for Indonesia required that it generate revenue by cultivating natural resources and erase export taxes the government had imposed to keep prices low at home. The measures further incentivised expansion of palm plantations. Alongside the IMF, private finance has helped boost production: Dutch banks alone provided more than $12bn in loans to Indonesian palm producers in the years 1995-99.

The short-term benefits to plantation owners and labourers, producer-nations’ governments and financiers have come with enormous long-term costs to the global climate. Forests destroyed for oil palm plantations are among the most carbon-rich in the world. When they are burned, that carbon is released.

Palm oil now accounts for 13.7% of Malaysia’s gross national income and is Indonesia’s top export. In October, at the European Palm Oil Association meeting in Madrid, government officials from the two countries trumpeted the successes in poverty reduction they had achieved thanks to palm oil (though growers in Indonesia, at least, have disputed these claims, calling on government and industry to do more for farmers independent of the big plantations). Officials further insisted that deforestation was being halted and sustainability achieved, even as another speaker told the attendees deforestation had actually increased in some areas over the previous decade. (In September, Indonesia’s president signed a three-year moratorium on new palm-plantation development.)

Commodity-producing countries need only answer to their buyers, though, while those buyers must respond to consumers. In 2004, the environmental NGO Friends of the Earth UK released a report detailing deforestation rates from palm-oil production. As the outcry spread, and concern rose among producers that the continued deforestation would become a risk to their reputations, the World Wildlife Federation that year convinced a small number of palm growers, manufacturers and retailers to establish the Roundtable on Sustainable Palm Oil. A decade later, most of the major users of palm oil had committed to production the RSPO deemed “sustainable” and 19% of global product was certified as such by the organisation. But the Environmental Investigation Agency, an offshoot of Greenpeace, three years ago found RSPO to be “woefully substandard” and “in some cases … colluding … to disguise violations”. (RSPO responded in a statement that it “takes very seriously the claims contained in the EIA report, and welcomes it as an opportunity for intensifying this dialogue, and further [improving] its certification system.”)

It is extraordinarily difficult to make sure that palm oil is being sustainably produced. A single palm oil mill – there are hundreds in Malaysia alone – can buy fruit from a multitude of suppliers, and with all its formulations and derivatives, palm oil has one of the most complicated supply chains of any ingredient. Even when the sustainability certification system is working as it is supposed to, environmentalists have criticised such programmes. For instance, a product can earn a “certified sustainable” label even if 99% of the palm oil it includes came from freshly deforested land. The RSPO says having less strict certification criteria encourages participation, the hope being that manufacturers of retail products will ramp up to higher levels once they see they can sell certified palm oil for a higher price.

Before the European Palm Oil Association meeting, RSPO’s head of European operations, Inke van der Sluijs, admitted that “very few companies do [the highest level of sustainability certification] because of the complexity and length of the supply chain”. RSPO is widely viewed by environmentalists as the most robust of several certification systems and encourages manufacturers to use RSPO-certified oil. Nevertheless, half of certified-sustainable palm oil isn’t sold as sustainable: until a sufficient number of consumers are willing to pay the higher price for certified palm oil, little will change.

Moreover, the vast majority of palm oil is traced only as far as the mill where it’s processed, not to the field where it’s produced. Eyes on the Forest, a coalition of Indonesian NGOs that includes the WWF – the same organisation that spurred palm oil certification – said in a 2016 report that “mill traceability [on its own] wastes time and money without offering a solution to the issues of illegal product entering the supply chains”. There is now a growing effort to deploy technology to trace each bunch of fruit to a field and farmer, which would finally ensure new deforestation isn’t occurring to produce palm oil.

The other hope for halting deforestation for palm is increasing yields, the idea being that if more oil can come from existing plantations it will obviate the need to expand the planting area into biodiverse forest. Rajinder Singh, leader of the genomics group at the Malaysian Palm Oil Board, a government agency, has been identifying genetic signatures associated with certain traits so high-yielding palms can be selected and land isn’t wasted on trees that don’t produce much. The best plantations currently yield around six or seven tonnes of oil per hectare, but Singh said, “we’ve seen individual palms that can give almost double” the amount of oil compared to common strains. As trees reach the end of their productive life of 25-30 years, they could be replaced by more prolific strains.

But even doubling yields won’t meet the near-quadrupling of demand expected by 2050. There is no easy solution. Replacing palm with other oils will only accelerate deforestation, since none of its competitors boast anywhere near its yield per unit of land: palm accounts for 6.6% of cultivated land for oils and fats, while delivering 38.7% of the output, according to the European Palm Oil Alliance, an industry group. Colombia is aggressively pursuing palm oil development in areas formerly devoted to illegal crops such as coca, but it has a lot of catching up to match Asia’s output.

Palm oil has become ubiquitous because it is the perfect ingredient for a number of growing industries, the perfect export for developing economies, and the perfect commodity for the globalised economy that links them. Wealthy consumers are capitalising on the cheap labour and valuable rainforest that developing nations have in abundance and are willing to part with at a discount to accelerate their economic growth.

But that model isn’t sustainable. If things continue, the forests and their creatures will be gone, and the cost of labour will increase as some workers move up the economic ladder and realise there are better things they could be doing than picking fruit. Palm oil producers and consumers will be left with nothing.

Products that are sustainable are those produced and consumed locally; when buyers are able to witness the production process, they will demand that it occur in line with their values. When it’s out of sight, it’s difficult to get enough of them to care. Changing that may require more than a little magic.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

• This piece has been edited to correct the attribution of a report, to “Eyes on the Forest, a coalition of Indonesian NGOs that includes the WWF”, rather than to the WWF alone.