Nearly three decades ago, when I was an overweight teenager, I sometimes ate six pieces of sliced white toast in a row, each one slathered in butter or jam. I remember the spongy texture of the bread as I took it from its plastic bag. No matter how much of this supermarket toast I ate, I hardly felt sated. It was like eating without really eating. Other days, I would buy a box of Crunchy Nut Cornflakes or a tube of Pringles: sour cream and onion flavour stackable snack chips, which were an exciting novelty at the time, having only arrived in the UK in 1991. Although the carton was big enough to feed a crowd, I could demolish most of it by myself in a sitting. Each chip, with its salty and powdery sour cream coating, sent me back for another one. I loved the way the chips – curved like roof tiles – would dissolve slightly on my tongue.

fter one of these binges – because that is what they were – I would speak to myself with self-loathing. “What is wrong with you?” I would say to the tear-stained face in the mirror. I blamed myself for my lack of self-control. But now, all these years later, having mostly lost my taste for sliced bread, sugary cereals and snack chips, I feel I was asking myself the wrong question. It shouldn’t have been “What is wrong with you?” but “What is wrong with this food?”

Back in the 90s, there was no word to cover all the items I used to binge on. Some of the things I over-ate – crisps or chocolate or fast-food burgers – could be classified as junk food, but others, such as bread and cereal, were more like household staples. These various foods seemed to have nothing in common except for the fact that I found them very easy to eat a lot of, especially when sad. As I ate my Pringles and my white bread, I felt like a failure for not being able to stop. I had no idea that there would one day be a technical explanation for why I found them so hard to resist. The word is “ultra-processed” and it refers to foods that tend to be low in essential nutrients, high in sugar, oil and salt and liable to be overconsumed.

Which foods qualify as ultra-processed? It’s almost easier to say which are not. I got a cup of coffee the other day at a train station cafe and the only snacks for sale that were not ultra-processed were a banana and a packet of nuts. The other options were: a panini made from ultra-processed bread, flavoured crisps, chocolate bars, long-life muffins and sweet wafer biscuits – all ultra-processed.

What characterises ultra-processed foods is that they are so altered that it can be hard to recognise the underlying ingredients. These are concoctions of concoctions, engineered from ingredients that are already highly refined, such as cheap vegetable oils, flours, whey proteins and sugars, which are then whipped up into something more appetising with the help of industrial additives such as emulsifiers.

Ultra-processed foods (or UPF) now account for more than half of all the calories eaten in the UK and US, and other countries are fast catching up. UPFs are now simply part of the flavour of modern life. These foods are convenient, affordable, highly profitable, strongly flavoured, aggressively marketed – and on sale in supermarkets everywhere. The foods themselves may be familiar, yet the term “ultra-processed” is less so. None of the friends I spoke with while writing this piece could recall ever having heard it in daily conversation. But everyone had a pretty good hunch what it meant. One recognised the concept as described by the US food writer Michael Pollan – “edible foodlike substances”.

Some UPFs, such as sliced bread or mass-produced cakes, have been around for many decades, but the percentage of UPFs in the average person’s diet has never been anything like as high as it is today. It would be unusual for most of us to get through the day without consuming at least a few ultra-processed items.

You might say that ultra-processed is just a pompous way to describe many of your normal, everyday pleasures. It could be your morning bowl of Cheerios or your evening pot of flavoured yoghurt. It’s savoury snacks and sweet baked goods. It’s chicken nuggets or vegan hotdogs, as the case may be. It’s the doughnut you buy when you are being indulgent, and the premium protein bar you eat at the gym for a quick energy boost. It’s the long-life almond milk in your coffee and the Diet Coke you drink in the afternoon. Consumed in isolation and moderation, each of these products may be perfectly wholesome. With their long shelf life, ultra-processed foods are designed to be microbiologically safe. The question is what happens to our bodies when UPFs become as prevalent as they are at the moment.

Evidence now suggests that diets heavy in UPFs can cause overeating and obesity. Consumers may blame themselves for overindulging in these foods, but what if it is in the nature of these products to be overeaten?

In 2014, the Brazilian government took the radical step of advising its citizens to avoid UPFs outright. The country was acting out of a sense of urgency, because the number of young Brazilian adults with obesity had risen so far and so fast, more than doubling between 2002 and 2013 (from 7.5% of the population to 17.5%). These radical new guidelines urged Brazilians to avoid snacking, and to make time for wholesome food in their lives, to eat regular meals in company when possible, to learn how to cook and to teach children to be “wary of all forms of food advertising”.

The biggest departure in the Brazilian guidelines was to treat food processing as the single most important issue in public health. This new set of rules framed unhealthy food less in terms of the nutrients it contains (fats, carbohydrates etc) and more by the degree to which it is processed (preserved, emulsified, sweetened etc). No government diet guidelines had ever categorised foods this way before. One of the first rules in the Brazilian guidelines was to “avoid consumption of ultra-processed products”. They condemned at a stroke not just fast foods or sugary snacks, but also many foods which have been reformulated to seem health-giving, from “lite” margarines to vitamin-fortified breakfast cereals.

From a British perspective – where the official NHS Eatwell guide still classifies low-fat margarines and packaged cereals as “healthier” options – it looks extreme to warn consumers off all ultra-processed foods (what, even Heinz tomato soup?). But there is evidence to back up the Brazilian position. Over the past decade, large-scale studies from France, Brazil, the US and Spain have suggested that high consumption of UPFs is associated with higher rates of obesity. When eaten in large amounts (and it’s hard to eat them any other way) they have also been linked to a whole host of conditions, from depression to asthma to heart disease to gastrointestinal disorders. In 2018, a study from France – following more than 100,000 adults – found that a 10% increase in the proportion of UPFs in someone’s diet led to a higher overall cancer risk. “Ultra-processed” has emerged as the most persuasive new metric for measuring what has gone wrong with modern food.

Why should food processing matter for our health? “Processed food” is a blurry term and for years, the food industry has exploited these blurred lines as a way to defend its additive-laden products. Unless you grow, forage or catch all your own food, almost everything you consume has been processed to some extent. A pint of milk is pasteurised, a pea may be frozen. Cooking is a process. Fermentation is a process. Artisanal, organic kimchi is a processed food, and so is the finest French goat’s cheese. No big deal.

But UPFs are different. They are processed in ways that go far beyond cooking or fermentation, and they may also come plastered with health claims. Even a sugary multi-coloured breakfast cereal may state that it is “a good source of fibre” and “made with whole grains”. Bettina Elias Siegel, the author of Kid Food: The Challenge of Feeding Children in a Highly Processed World, says that in the US, people tend to categorise food in a binary way. There is “junk food” and then there is everything else. For Siegel, “ultra-processed” is a helpful tool for showing new parents that “there’s a huge difference between a cooked carrot and a bag of industrially produced, carrot-flavoured veggie puffs” aimed at toddlers, even if those veggie puffs are cynically marketed as “natural”.

The concept of UPFs was born in the early years of this millennium when a Brazilian scientist called Carlos Monteiro noticed a paradox. People appeared to be buying less sugar, yet obesity and type 2 diabetes were going up. A team of Brazilian nutrition researchers led by Monteiro, based at the university of Sao Paulo, had been tracking the nation’s diet since the 80s, asking households to record the foods they bought. One of the biggest trends to jump out of the data was that, while the amount of sugar and oil people were buying was going down, their sugar consumption was vastly increasing, because of all of the ready-to-eat sugary products that were now available, from packaged cakes to chocolate breakfast cereal, that were easy to eat in large quantities without thinking about it.

Photograph: Katrina Wittkamp/Getty

To Monteiro, the bag of sugar on the kitchen counter is a healthy sign, not because sugar itself has any goodness in it, but because it belongs to a person who cooks. Monteiro’s data suggested to him that the households who were still buying sugar were also the ones who were still making the old Brazilian dishes such as rice and beans.

Monteiro is a doctor by training, and when you talk to him, he still has the idealistic zeal of someone who wants to prevent human suffering. He had started off in the 70s treating poor people in rural villages, and was startled to see how quickly the problems of under-nutrition were replaced by those of tooth decay and obesity, particularly among children. When Monteiro looked at the foods that had increased the most in the Brazilian diet – from cookies and sodas to crackers and savoury snacks – what they had in common was that they were all highly processed. Yet he noticed that many of these commonly eaten foods did not even feature in the standard food pyramids of US nutrition guidelines, which show rows of different whole foods according to how much people consume, with rice and wheat at the bottom, then fruits and vegetables, then fish and dairy and so on. These pyramids are based on the assumption that people are still cooking from scratch, as they did in the 50s. “It is time to demolish the pyramid”, wrote Monteiro in 2011.

Once something has been classified, it can be studied. In the 10 years since Monteiro first announced the concept, numerous peer-reviewed studies on UPFs have been published confirming the links he suspected between these foods and higher rates of disease. By giving a collective name to ultra-processed foods for the first time, Monteiro has gone some way to transforming the entire field of public health nutrition.

As he sees it, there are four basic kinds of food, graded by the degree to which they are processed. Taken together, these four groups form what Monteiro calls the Nova system (meaning a new star). The first category – group 1 – are the least processed, and includes anything from a bunch of parsley to a carrot, from a steak to a raisin. A pedant will point out that none of these things are strictly unprocessed by the time they are sold: the carrot is washed, the steak is refrigerated, the raisin is dried. To answer these objections, Monteiro renamed this group “unprocessed and minimally processed foods”.

The second group is called “processed culinary ingredients”. These include butter and salt, sugar and lard, oil and flour – all used in small quantities with group 1 foods to make them more delicious: a pat of butter melting on broccoli, a sprinkling of salt on a piece of fish, a spoonful of sugar in a bowl of strawberries.

Next in the Nova system comes group 3, or “processed foods”. This category consists of foods that have been preserved, pickled, fermented or salted. Examples would be canned tomatoes and pulses, pickles, traditionally made bread (such as sourdough), smoked fish and cured meats. Monteiro notes that when used sparingly, these processed foods can result in “delicious dishes” and nutritionally balanced meals.

The final category, group 4, is unlike any of the others. Group 4 foods tend to consist largely of the sugars, oils and starches from group 2, but instead of being used sparingly to make fresh food more delicious, these ingredients are now transformed through colours, emulsifiers, flavourings and other additives to become more palatable. They contain ingredients unfamiliar to domestic kitchens such as soy protein isolate (in cereal bars or shakes with added protein) and “mechanically separated meat” (turkey hotdogs, sausage rolls).

Group 4 foods differ from other foods not just in substance, but in use. Because they are aggressively promoted and ready-to-eat, these highly profitable items have vast market advantages over the minimally processed foods in group 1. Monteiro and his colleagues have observed from evidence around the world that these group 4 items are liable to “replace freshly made regular meals and dishes, with snacking any time, anywhere”. For Monteiro, there is no doubt that these ultra-processed foods are implicated in obesity as well as a range of non-communicable diseases such as heart disease and type 2 diabetes.

Not everyone in the world of nutrition is convinced by the Nova system of food classification. Some critics of Monteiro have complained that ultra-processed is just another way to describe foods that are sugary or fatty or salty or low in fibre, or all of these at once. If you look at the UPFs that are consumed in the largest quantities, the majority of them take the form of sweet treats or sugary drinks. The question is whether these foods would still be harmful if the levels of sugar and oil could be reduced.

The first time the nutrition researcher Kevin Hall heard anyone talk about ultra-processed food, he thought it was “a nonsense definition”. It was 2016 and Hall – who studies how people put on weight at the National Institute of Diabetes and Digestive and Kidney Diseases at Bethesda, Maryland – was at a conference chatting with a representative from PepsiCo who scornfully mentioned the new Brazilian set of food guidelines and specifically the directive to avoid ultra-processed foods. Hall agreed that this was a silly rule because, as far as he was concerned, obesity had nothing to do with food processing.

Anyone can see that some foods are processed to a higher degree than others – an Oreo is not the same as an orange – but Hall knew of no scientific proof that said the degree of processed food in a person’s diet could cause them to gain weight. Hall is a physicist by training and he is a self-confessed “reductionist”. He likes to take things apart and see how they work. He is therefore attracted to the idea that food is nothing more than the sum of its nutrient parts: fats plus carbs plus protein and fibre, and so on. The whole notion of ultra-processed foods annoyed him because it seemed too fuzzy.

When Hall started to read through the scientific literature on ultra-processed foods, he noticed that all of the damning evidence against them took the form of correlation rather than absolute proof. Like most studies on the harmful effects of particular foods, these studies fell under the umbrella of epidemiology: the study of patterns of health across populations. Hall – and he is not alone here – finds such studies less than convincing. Correlation is not causation, as the saying goes.

Just because people who eat a lot of UPFs are more likely to be obese or suffer from cancer does not mean that obesity and cancer are caused by UPFs, per se. “Typically, it’s people in lower economic brackets who eat a lot of these foods,” Hall said. He thought UPFs were being wrongly blamed for the poor health outcomes of living in poverty.

At the end of 2018, Hall and his colleagues became the first scientists to test – in randomised controlled conditions – whether diets high in ultra-processed foods could actually cause overeating and weight gain.

For four weeks, 10 men and 10 women agreed to be confined to a clinic under Hall’s care and agreed to eat only what they were given, wearing loose clothes so that they would not notice so much if their weight changed. This might sound like a small study, but carefully controlled trials like this are considered the gold standard for science, and are especially rare in the field of nutrition because of the difficulty and expense of persuading humans to live and eat in laboratory conditions. Barry Popkin, a professor of nutrition at the University of North Carolina, has praised Hall’s study – published in Cell Metabolism – for being “as good a clinical trial as you can get”.

For two weeks, Hall’s participants ate mostly ultra-processed meals such as turkey sandwiches with crisps, and for another two weeks they ate mostly unprocessed food such as spinach omelette with sweet potato hash. The researchers worked hard to design both sets of meals to be tasty and familiar to all participants. Day one on the ultra-processed diet included a breakfast of Cheerios with whole milk and a blueberry muffin, a lunch of canned beef ravioli followed by cookies and a pre-cooked TV dinner of steak and mashed potatoes with canned corn and low-fat chocolate milk. Day one on the unprocessed diet started with a breakfast of Greek yoghurt with walnuts, strawberries and bananas, a lunch of spinach, chicken and bulgur salad with grapes to follow, and dinner of roast beef, rice pilaf and vegetables, with peeled oranges to finish. The subjects were told to eat as much or as little as they liked.

Hall set up the study to match the two diets as closely as possible for calories, sugar, protein, fibre and fat. This wasn’t easy, because most ultra-processed foods are low in fibre and protein and higher in sugar. To compensate for the lack of fibre, the participants were given diet lemonade laced with soluble fibre to go with their meals during the two weeks on the ultra-processed diet.

It turned out that, during the weeks of the ultra-processed diet, the volunteers ate an extra 500 calories a day, equivalent to a whole quarter pounder with cheese. Blood tests showed that the hormones in the body responsible for hunger remained elevated on the ultra-processed diet compared to the unprocessed diet, which confirms the feeling I used to have that however much I ate, these foods didn’t sate my hunger.

Photograph: Jochen Tack/Alamy

Hall’s study provided evidence that an ultra-processed diet – with its soft textures and strong flavours – really does cause over-eating and weight gain, regardless of the sugar content. Over just two weeks, the subjects gained an average of 1kg. This is a far more dramatic result than you would expect to see over such a short space of time (especially since the volunteers rated both types of food as equally pleasant).

After Hall’s study was published in July 2019, it was impossible to dismiss Monteiro’s proposition that the rise of UPFs increases the risk of obesity. Monteiro told me that as a result of Hall’s study, he and his colleagues in Brazil found they were suddenly being taken seriously.

Now that we have evidence of a link between diets high in UPFs and obesity, it seems clear that a healthy diet should be based on fresh, home-cooked food. To help champion home cooking among Brazilians, Monteiro recruited the cookery writer Rita Lobo, whose website Panelinha (“network”) is the most popular food site in Brazil, with 3m hits a month. Lobo said that when she tells people about UPFs, the first reaction is panic and anger. “They say: ‘Oh my God! I’m not going to be able to eat my yoghurt or my cereal bar! What am I going to eat?’” After a while, however, she says that the concept of ultra-processed foods is “almost a relief” to people, because it liberates them from the polarities and restriction created by fad diets or “clean eating’”. People are thrilled, Lobo says, when they realise they can have desserts again, as long as they are freshly made.

But modern patterns of work do not make it easy to find the time to cook every day. For households who have learned to rely on ultra-processed convenience foods, returning to home cooking can seem daunting – and expensive. Hall’s researchers in Maryland spent 40% more money purchasing the food for the unprocessed diet. (However, I noticed that the menu included large prime cuts of meat or fish every day; it would be interesting to see how the cost would have compared with a larger number of vegetarian meals or cheaper cuts of meat.)

In Brazil, cooking from scratch still tends to be cheaper than eating ultra-processed food, Lobo says. UPFs are a relative novelty in Brazil and memories of a firm tradition of home cooking have not died yet here. “In Brazil, it doesn’t matter if you are rich or poor, you grew up eating rice and beans. The problem for you [in the UK],” Lobo remarks, “is that you don’t know what your ‘rice and beans’ is.”

In Britain and the US, our relationship with ultra-processed food is so extensive and goes back so many decades that these products have become our soul food, a beloved repertoire of dishes. It’s what our mothers fed us. If you want to bond with someone who was a child in 1970s Britain, mention that you have childhood memories of being given Findus Crispy Pancakes and spaghetti hoops followed by Angel Delight for tea. I have noticed that Australian friends have similar conversations about the childhood joys of Tim Tams chocolate biscuits. In the curious coding of the British class system, a taste for industrial branded foods is a way to reassure others that you are OK. What kind of snob would disparage a Creme Egg or fail to recognise the joy of licking cheesy Wotsit dust from your fingers?

I am as much of a sucker for this branded food nostalgia as anyone. There is a part of my brain – the part that is still an eight-year-old at a birthday party – that will always feel that Iced Gems (ultra-processed cookies topped with ultra-processed frosting) are pure magic. But I’ve started to feel a creeping unease that our ardent affection for these foods has been mostly manufactured by the food corporations who profit from selling them. For the thousands of people trapped in binge-eating disorder – as I once was – UPFs are false friends.

The multinational food industry has a vested interest in rubbishing Monteiro’s ideas about how UPFs are detrimental to our health. And much of the most vociferous criticism of his Nova system has come from sources close to the industry. A 2018 paper co-authored by Melissa Mialon, a French food engineer and public health researcher, identified 32 materials online criticising Nova, most of which were not peer-reviewed. The paper showed that, out of 38 writers critical of Nova, 33 had links to the ultra-processed food industry.

For many in the developing world, the prevalence of ultra-processed foods is making it hard for those on a limited budget to feed their children a wholesome diet. Victor Aguayo, chief of nutrition at Unicef, tells me over the phone that, as ultra-processed foods become cheaper and other foods, such as vegetables and fish, become more expensive, the UPFs are taking up a bigger volume of children’s diets. What’s more, the pleasurable textures and aggressive marketing of these foods makes them “appealing and aspirational” both to children and parents, says Aguayo.

Soon after the arrival in Nepal of brightly coloured packages that, as Aguayo describes them, “look like food for children: the cookies, the savoury snacks, the cereals”, aid workers started to see an epidemic of “both overweight and micronutrient deficiency” including anaemia among Nepalese children under the age of five.

Aguayo says there is an urgent need to change the food environment to make the healthy options the easy, affordable and available ones. Ecuador, Uruguay and Peru have followed Brazil’s example in urging their citizens to steer clear of ultra-processed foods. Uruguay’s dietary guidelines – issued in 2016 – tells Uruguayans to “base your diet on natural foods, and avoid the regular consumption of ultra-processed products”. How easy this will be to do is another matter.

In Australia, Canada or the UK, to be told to avoid ultra-processed food – as the Brazilian guidelines do – would mean rejecting half or more of what is for sale as food, including many basic staples that people depend on, such as bread. The vast majority of supermarket loaves count as ultra-processed, regardless of how much they boast of being multiseed, malted or glowing with ancient grains.

Earlier this year, Monteiro and his colleagues published a paper titled “Ultra-processed foods: what they are and how to identify them”, offering some rules of thumb. The paper explains that “the practical way to identify if a product is ultra-processed is to check to see if its list of ingredients contains at least one food substance never or rarely used in kitchens, or classes of additives whose function is to make the final product palatable or more appealing (‘cosmetic additives’)”. Tell-tale ingredients include “invert sugar, maltodextrin, dextrose, lactose, soluble or insoluble fibre, hydrogenated or interesterified oil”. Or it may contain additives such as “flavour enhancers, colours, emulsifiers, emulsifying salts, sweeteners, thickeners and anti-foaming, bulking, carbonating, foaming, gelling and glazing agents”.

But not everyone has time to search every label for the presence of glazing agents. A website called Open Food Facts, run by mostly French volunteers, has started the herculean labour of creating an open database of packaged foods around the world and listing where they fit into on the Nova system. Froot Loops: Nova 4. Unsalted butter: Nova 2. Sardines in olive oil: Nova 3. Vanilla Alpro yoghurt: Nova 4. Stéphane Gigandet, who runs the site, says that he started analysing food by Nova a year ago and “it is not an easy task”.

For most modern eaters, avoiding all ultra-processed foods is unsettling and unrealistic, particularly if you are on a low income or vegan or frail or disabled, or someone who really loves the occasional cheese-and-ham toastie made from sliced white bread. In his early papers, Monteiro wrote of reducing ultra-processed items as a proportion of the total diet rather than cutting them out altogether. Likewise, the French Ministry of Health has announced that it wants to reduce consumption of Nova 4 products by 20% over the next three years.

We still don’t really know what it is about ultra-processed food that generates weight gain. The rate of chewing may be a factor. In Hall’s study, during the weeks on the ultra-processed diet people ate their meals faster, maybe because the foods tended to be softer and easier to chew. On the unprocessed diet, a hormone called PYY, which reduces appetite, was elevated, suggesting that homemade food keeps us fuller for longer. The effect of additives such as artificial sweeteners on the gut microbiome is another theory. Later this year, new research from physicist Albert-László Barabási will reveal more about the way that ultra-processing actually alters food at a molecular level.

In a two-part blog on ultra-processed foods in 2018 (Rise of the Ultra Foods) Anthony Warner, a former food industry development chef who tweets and campaigns as Angry Chef, argued that Nova was stoking fear and guilt about food and “adding to the stress of already difficult lives” by making people feel judged for their food choices. But having read Kevin Hall’s study, he wrote an article in May 2019 admitting: “I was wrong about ultra-processed food – it really is making you fat.” Warner said the study convinced him that “eating rate, texture and palatability” of UPFs lead to overeating, and ended with a call for more research.

Hall tells me that he is in the process of constructing another study on ultra-processed food and obesity. This time, the people on the ultra-processed diet would also be eating larger amounts of unprocessed foods, such as crunchy vegetables with low energy density, while still getting more than 80% of their calories from ultra-processed food – equivalent to adding a side salad or a portion of broccoli to your dinner of frozen pizza. This is much closer to how most families actually eat.

Even if scientists do succeed in pinning down the mechanism or mechanisms by which ultra-processed foods make us gain weight, it’s not clear what policy-makers should do about UPFs, except for giving people the support and resources they need to cook more fresh meals at home. To follow the Brazilian advice entails a total rethink of the food system.

For as long as we believed that single nutrients were the main cause of poor diets, industrial foods could be endlessly tweaked to fit with the theory of the day. When fat was seen as the devil, the food industry gave us a panoply of low-fat products. The result of the sugar taxes around the world has been a raft of new artificially sweetened drinks. But if you accept the argument that processing is itself part of the problem, all of this tweaking and reformulation becomes so much meaningless window-dressing.

An ultra-processed food can be reformulated in countless ways, but the one thing it can’t be transformed into is an unprocessed food. Hall remains hopeful that there may turn out to be some way to adjust the manufacture of ultra-processed foods to make them less harmful to health. A huge number of people on low incomes, he notes, are relying on these “relatively inexpensive tasty things” for daily sustenance. But he is keenly aware that the problems of nutrition cannot be cured by ever more sophisticated processing. “How do you take an Oreo and make it non-ultra-processed?” he asks. “You can’t!”

• This article was amended on 13 February 2020. An earlier version referred to American friends reminiscing about Tim Tams; it should have said Australian. It also described Melissa Mialon as a Brazilian nutritionist; she is a French food engineer.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.