‘Virtually zero.’ That’s a reasonable estimate of the probability that public health authorities in the foreseeable future will successfully curb the worldwide epidemics of obesity and diabetes, at least according to Margaret Chan, the director general of the World Health Organization (WHO) – a person who should know. Virtually zero is the likelihood, Chan said at the National Academy of Medicine’s annual meeting in October, that she and her many colleagues worldwide will successfully prevent ‘a bad situation’ from ‘getting much worse’. That Chan also described these epidemics as a ‘slow-motion disaster’ suggests the critical nature of the problem: ‘population-wide’ explosions in the prevalence of obesity along with increases in the occurrence of diabetes that frankly strain the imagination: a disease that leads to blindness, kidney failure, amputation, heart disease and premature death, and that was virtually non-existent in hospital inpatient records from the mid-19th century, now afflicts one in 11 Americans; in some populations, as many as one in two adults are diabetic.

In the midst of such a public health crisis, the obvious question to ask is why. Many reasons can be imagined for any public health failure, but we have no precedents for a failure of this magnitude. As such, the simplest explanation is that we’re not targeting the right agent of disease; that our understanding of the aetiology of both obesity and diabetes is somehow flawed, perhaps tragically so.

Researchers in harder sciences have a name for such situations: ‘pathological science’, defined by the Nobel Laureate chemist Irving Langmuir in 1953 as ‘the science of things that aren’t so’. Where experimental investigation is prohibitively expensive or impossible to do, mistaken assumptions, misconceived paradigms and pathological science can survive indefinitely. Whether this is the case with the current epidemics is an all-too-regrettable possibility: perhaps we’ve simply misconceived the reality of the link between diet, lifestyle and the related disorders of obesity and diabetes? As the Oxford scholar Robert Burton suggested in The Anatomy of Melancholy (1621), in cases in which the cures are ‘imperfect, lame, and to no purpose’ it’s quite possible that the causes are misunderstood.

The history of obesity and nutrition research suggests that this is indeed what has happened. In the decades leading up to the Second World War, German and Austrian clinical investigators had concluded that common obesity was clearly caused by a hormonal disturbance; starting in the 1960s, other research would link that disturbance to the sugar in our diets. But the German and Austrian thinking evaporated with the war, and the possibility that sugar was to blame never took hold, dismissed by a nutrition community who, by the 1970s, became fixated on dietary fat as the trigger of our chronic diseases. Now, with an explosion of the epidemic and compelling new research, it’s time to reconsider both our causal thinking on obesity and diabetes, and the possibility that sugar is playing the critical role.

When researchers and public health authorities today discuss their failure to curb the rising tide of obesity and diabetes, they offer the explanation that these disorders are ‘multifactorial and complex’, implying that failure is somehow understandable. But this obscures the reality that prescriptions to prevent and treat the two depend almost entirely on two simple causal concepts, neither one of which is necessarily correct.

The first assumption equates obesity and Type 2 diabetes (the common form of the disease, formerly known as ‘adult-onset’ until it began appearing in children as well). Because obesity and Type 2 diabetes are so closely associated in both individuals and populations, the assumption is that it’s the obesity – or at least the accumulation of excess fat – that causes the diabetes. By this logic, whatever causes obesity is ultimately the cause of the diabetes as well.

The second assumption then strives to explain ‘the fundamental cause’ of the obesity itself: an energy imbalance between calories consumed on one hand, and calories expended on the other hand.

This thinking, espoused by the WHO and virtually every other medical authority, is a paradigm in the true Kuhnian sense of the word. Researchers and public health authorities describe obesity as a disorder of ‘energy balance’. This conception underlies virtually all aspects of obesity research from prevention through treatment, and, by association, diabetes. As such, it has also shaped how we think about the role of what is now, finally, considered a prime suspect – refined or ‘added’ sugars, and specifically, sucrose (table sugar) and high-fructose corn syrup.

The WHO and other health organisations have recently taken to arguing that sugar and particularly sugary beverages should be taxed heavily or regulated. But they do so not because they say sugar causes disease – using the same definition of causality that we use when we say cigarettes cause lung cancer – but, rather, because, from their perspective, sugar represents ‘empty calories’ that we eat in excess. By this thinking, we still get fatter because we eat too much or exercise too little. The solution is to eat in moderation, and consume sugar in moderation or balance it with more physical activity.

The energy balance paradigm implies that the only way in which foods influence our body fat is through their energy content, or calories – that is, through the energy that we absorb without excreting, and so make available to be oxidised or stored. This is the only variable that matters. It’s the implication of the phrase ‘a calorie is a calorie’, which, by the 1960s, had become a mantra of nutrition and obesity researchers, evoked invariably to support the dogma that only calories count when it comes to understanding and treating human obesity.

This logic has been the lifeblood of the sugar industry. If sugar was uniquely toxic, in that it possessed some special property that made us respond to it by accumulating fat or becoming diabetic, then government health agencies would have to regulate it. If all sugar does is add calories to the diet, just as any other food does, then it is, in effect, benign. When the sugar industry embarked in 1956 on a nationwide advertising offensive to knock down reports that sugar is ‘fattening’, it did so on the seemingly sound scientific basis that ‘[s]ugar is neither a “reducing food” nor a “fattening food”’, as the industry advertisements explained. ‘There are no such things. All foods supply calories and there is no difference between the calories that come from sugar or steak or grapefruit or ice cream.’

Thinking of obesity as an energy-balance disorder is as meaningless as calling poverty a money-balance problem

Even 60 years later, in 2015, when The New York Times reported that academic researchers were doing the bidding of Coca-Cola by taking its money to fund a Global Energy Balance Network and ‘shift blame for obesity away from bad diets’, this was still the logic invoked in sugar’s defence: if you believe that obesity is caused by a mere caloric surplus, then the solution to the epidemic is not necessarily to avoid Coca-Cola, but to either consume it (and everything else) in moderation or to burn off the excess calories with physical activity. For the sugar industry and the purveyors, such as Coca-Cola, of sugar-rich foods and beverages, this remarkably resilient, century-old conception of why some of us get fat (or are born fat) and others don’t (or aren’t) has been the gift that keeps on giving.

So here’s another way to frame what is now the imperative question: is the energy-balance hypothesis of obesity correct? Is it the right paradigm to understand the disorder? The competing hypothesis has existed for over a century: in this paradigm, obesity is not an energy-balance disorder but a disorder of excess fat accumulation and so, clearly, a hormonal and metabolic disorder – the result of an ‘endocrine disturbance’, as it was phrased in the 1930s by Eugene Du Bois, then the leading American authority on metabolism. By this logic, the foods we eat influence fat accumulation not because of their caloric content but because of their macronutrient content, the proteins, fats and carbohydrates they contain. This paradigm attends to how organisms (humans, of course, in particular) orchestrate the careful ‘partitioning’ of the macronutrient fuels they consume, determining whether they will be burned for energy or stored or used to rebuild tissues and organs. It proposes that dysregulation of this exquisitely-evolved, finely-tuned homeostatic system (a system that is biologically balanced) is the necessary component to explain both the excessive storage of calories of fat – obesity – and the diabetes that accompanies it.

This alternate hypothesis implies that sugar has unique effects in the human body leading directly to both diabetes and obesity, independent of the calories consumed. By this way of thinking, refined sugars are indeed toxic, albeit over the course of years or decades. We get fat and diabetic not because we eat too much of them – although that is implied tautologically merely by the terms ‘overconsumption’ and ‘overeating’ – but because they have unique physiological, metabolic and hormonal effects that directly trigger these disorders. If all this is right, then thinking of obesity as an energy-balance disorder is as meaningless as calling poverty a money-balance problem (caused, of course, by earning too little or spending too much, or both). By conceiving of obesity as a problem caused by the behaviours of excessive consumption and physical inactivity, researchers not only took a physiological defect – the excess accumulation of fat, often to a massive extent – and turned it into a behavioural problem. But they made a critical error, one that has grown over the course of decades into an idea that seems too big to fail.

Understanding how this happened requires we attend to history. The modern era of nutrition science dates to the late 1860s, when German researchers pioneered the use of room-sized devices called calorimeters. These allowed them to measure the energy expended by human or animal subjects under different conditions of diet and activity. For the next half a century, effectively all nutrition research was directed toward studying energy balance (the energy content of foods and the energy expended or excreted by those who ate it) and the protein, vitamins, minerals and fibre necessary for health and wellbeing. This was a function of the research tools available at the time, and it has remained the foundation of nutrition wisdom ever since.

Today, when nutritionists say that sugar consists of ‘empty calories’, they’re defining it in the terms of this century-old research and the tools available to researchers of that era. When obesity researchers blame obesity on the imbalance of energy consumed to expended, they’re doing the same. Both are assuming that the science that came after, including the emergence of entire disciplines of medicine, is irrelevant.

The idea of obesity as an energy-balance disorder emerged directly from what was considered one of the great triumphs in nutrition in the late 19th-century: the confirmation that the laws of thermodynamics – conservation of energy, specifically – applied not just to inanimate matter but to living organisms and humans. In line with this research, nutritionists embraced calories and energy as the currency of their discipline, and physicians, speculating as to the cause of obesity, naturally did the same. By the early 1900s, the German diabetes specialist Carl von Noorden was proposing that ‘the ingestion of a quantity of food greater than that required by the body, leads to an accumulation of fat, and to obesity, should the disproportion be continued over a considerable period’.

In the 1920s, Von Noorden’s ideas were taken up in the United States by Louis Newburgh, a physician at the University of Michigan, who espoused what he considered an indisputable truth: ‘All obese persons are alike in one fundamental respect – they literally overeat.’ By assuming that this overeating must be the cause of obesity, Newburgh proceeded to blame the disorder on some combination of a ‘perverted appetite’ (excessive energy consumption) and a ‘lessened outflow of energy’ (insufficient expenditure). To explain why obese individuals failed to respond to this imbalance by either eating less or exercising more – both, after all, should be under conscious control – Newburgh also suggested that the overeating and/or under-expending were often compounded by ‘various human weaknesses such as overindulgence and ignorance’, thus blaming the victim and beginning the process that would turn obesity research in the 1960s into a subdiscipline of psychology and behavioural science.

This logic is still with us today. By 1939, Newburgh’s biography at the University of Michigan was already crediting him with the discovery that ‘the whole problem of weight lies in regulation of the inflow and outflow of calories’ and for having ‘undermined conclusively the generally held theory that obesity is the result of some fundamental fault’.

The existence of a fundamental fault, however, could not be dismissed so lightly, as German and Austrian investigators were still arguing at the time. They had concluded that obesity could be explained only by such a fault, a hormonal or regulatory defect. Worth noting is that the German and Austrian research communities had pioneered all the fields of science relevant to understanding obesity – including nutrition, metabolism, endocrinology and genetics. They dominated medical science, just as they did physics and chemistry through the Second World War. This was an era during which the lingua franca of science – medical or otherwise – was German, and when individuals serious about pursuing science travelled to Germany and Austria to learn from, if not mentor with, these authorities.

Coincident with von Noorden’s suggestion that obesity was an energy-balance disorder, his contemporary, Gustav von Bergmann, who would become the leading German authority on internal medicine, argued that it was clearly not. Von Bergmann pointed out that the overconsumption of energy that von Noorden was blaming as the cause of obesity – more energy in than out – was merely a description of what happened when the mass of any system increased, not an explanation at all.

The purpose of a hypothesis in science, quite simply, is to offer an explanation for what we observe, either in nature or the laboratory. How many of these observations can the hypothesis explain or predict in a simple and straightforward way? Yet the energy-balance conception fails to explain anything: it cannot explain why calories of fat are trapped in fat tissue rather than oxidised for fuel, nor such simple observations as the genetic basis of obesity (identical twins, after all, are identical not just in their facial features, height and colouring, but in body type too) or why fat accumulates differently in men and women.

With obesity, ‘a sort of anarchy exists, the adipose tissue does not fit into the precisely regulated management of the whole organism’

By von Bergmann’s logic, obesity was clearly not a problem of energy balance, but of fat trapping (just as global warming is not an energy-balance problem, but an energy-trapping one). The question that had to be answered is why this trapping occurs. Any viable hypothesis of obesity had to explain why the fat tissue of the obese is so avid in hoarding calories as fat, rather than allowing that fat to be metabolised and provide energy for the body.

By 1930, Julius Bauer of the University of Vienna – the ‘noted Vienna authority on internal diseases’, as The New York Times called him – had taken up von Bergmann’s ideas, arguing that obesity had to result from a dysregulation of the biological factors that normally work to keep fat accumulation under check. Bauer argued that fat cells are clearly being driven by these factors to hoard excessive calories as fat, and this in turn would deprive the rest of the body of the energy it needed to thrive. In this hormonal/regulatory conception, excessive fat-accumulation causes hunger and physical inactivity, not the other way around.

Bauer likened the fat tissue of an obese person to that of ‘a malignant tumour or … the foetus, the uterus or the breasts of a pregnant woman’, all with independent agendas, causing them to take up calories of fuel from the circulation and hoard them or put them to localised use, regardless of how much the person might be eating or exercising. With obesity, wrote Bauer, ‘a sort of anarchy exists, the adipose tissue lives for itself and does not fit into the precisely regulated management of the whole organism’.

By 1938, Russell Wilder, head of the department of medicine at the Mayo Clinic, was writing that this German/Austrian hypothesis ‘deserves attentive consideration’, and that ‘the effect after meals of withdrawing from the circulation even a little more fat than usual might well account both for the delayed sense of satiety and for the frequently abnormal taste for carbohydrate encountered in obese persons … A slight tendency in this direction would have a profound effect in the course of time.’

In 1940, when Hugo Rony, an endocrinologist at Northwestern University in Chicago, published the first academic treatise written on obesity in the US, he asserted that the hormonal/regulatory hypothesis was ‘more or less fully accepted’ by the European authorities.

And then it vanished. The German and Austrian medical-research community evaporated with the rise of Hitler, and the nexus of medical science shifted from Germany and Austria to the US, a nation not devastated by the war; the lingua franca of medical science shifted as well from German to English. With those shifts, arguably the best thinking of the era in medical science would no longer be read, nor would it be referenced. The conception of obesity as a hormonal regulatory disorder faded out of fashion.

In the post-war era of nutrition and obesity research, Newburgh’s energy balance conception was fixed as the obesity paradigm, not because it answered any meaningful questions about obesity and how, why and when we accumulate excess fat, but because it was a US conception at a time when young American physicians, many with little scientific training, came to dominate the field.

Embrace of the energy-balance paradigm and, with it, the death of the hormonal/regulatory hypothesis, can be seen clearly in the citation records. In 1941, Bauer published what would be his second and last English-language article on obesity: a 27-page review in Archives of Internal Medicine entitled ‘Obesity: Its Pathogenesis, Etiology and Treatment’. (By then, he had fled to the US and was living unaffiliated in Los Angeles). He spent the first third of the article critiquing, point by point, Newburgh’s ‘energy theory of obesity’, and the remainder discussing the ‘biologic theory’, and the evidence for why obesity had to be a hormonal/regulatory disorder. In 1942, Newburgh countered with a 64-page review in the same journal, refuting the biologic hypothesis and insisting that obesity is ‘invariably the result of a disproportion between the inflow and the outflow of energy’. In 1944, Newburgh published a second review, this one in Physiological Reviews, again insisting that von Bergmann and Bauer’s ideas had been refuted.

By 1959, Bauer’s article had been referenced only 10 times and would not be cited again in the indexed medical literature for another half a century. Meanwhile, Newburgh’s two articles on obesity as an energy-balance disorder would continue to be cited through to the end of the 1970s – accumulating 69 and 64 citations by that time respectively, enormous numbers for that era.

Despite its almost-universal acceptance, the energy theory remained at loggerheads with much of the science. For instance, animal models of obesity – the first of which was discussed in the literature in the late 1930s – consistently refuted Newburgh’s arguments and supported Bauer’s. Obese animals would frequently manifest what Newburgh might have described as a perverted appetite (technically, hyperphagia): as they grew fatter they would be exceedingly hungry and consume great amounts of food. But they would invariably get obese, or at least significantly fatter, even when they didn’t eat any more, or weren’t allowed to eat any more than control animals, often littermates, that remained lean. Some of these animals would remain excessively fat even as they were being starved.

Insulin partitions how we use the fuels we consume: it directs fat cells to store fat

Whatever the defect or fundamental fault that caused these animals to accumulate excessive fat, a perverted appetite (ie, overeating) could be ruled out. The defect had to be working either to cause the fat cells to hoard calories as fat, or to suppress the animals’ ability to burn fatty acids for fuel. Or both.

Not until the 1960s, though, would researchers elucidate the basic mechanisms of fat accumulation. To do so required invention of a technology that allowed researchers to accurately measure the level of hormones circulating in the bloodstream. This was the work of Rosalyn Yalow, a medical physicist, and Solomon Berson, a physician. When Yalow was awarded the Nobel Prize for the work in 1977 (by then, Berson was not alive to share it), the Nobel Foundation described it aptly as bringing about ‘a revolution in biological and medical research’. Those interested in obesity could now finally answer the questions on which the pre-war European clinicians could only speculate: what hormones regulate the storage of fat in fat cells and its use for fuel by the rest of the body?

Answers began coming with the very first publications out of Yalow and Berson’s laboratory and were swiftly confirmed. As it turns out, virtually all hormones work to mobilise fatty acids from fat cells so that they can then be used for fuel. The one dominant exception to this fuel-mobilisation signalling is insulin, which partitions how we use the fuels we consume: in particular, it directs fat cells to store fat, while facilitating the uptake and oxidation of glucose (blood sugar) by muscle and organ cells. In other words, when insulin is secreted – primarily in response to the carbohydrates in our diet – it directs our cells to burn carbohydrate as fuel and store fat. And so, the one biological factor necessary to mobilise fat from storage and have it used for fuel, as Yalow and Berson suggested in 1965, is ‘the negative stimulus of insulin deficiency’. Put simply, when insulin levels in circulation are elevated, we store fat and use glucose for fuel; as insulin levels drop, fat is mobilized and we burn it instead.

Yalow and Berson themselves described insulin as a ‘lipogenic’, or fat-forming hormone. This lipogenic signal must be turned off, or at least muted significantly, for the fat cells to release their stored fat and for the body to metabolise it for energy. While obesity researchers like to say that the sine qua non of a weight reduction diet is calorie-restriction, this alternative, biologically-based hypothesis would say that the sine qua non is lowering insulin. The more we consume carbohydrates, though, and particularly sugar, the higher i our insulin levels will be.

The potential role of insulin in obesity was illuminated further by a second revelation from Yalow and Berson’s early research: both Type 2 diabetics and the obese tend to have elevated levels of blood sugar and abnormally high levels of circulating insulin. This implies that the cells of their muscles and organs are resistant to the insulin circulating in their blood, an observation that was also quickly and widely confirmed. By the mid-1960s, both physicians and researchers were realising that Type 2 diabetes was not a disease of insulin deficiency – as Type 1 diabetes is – at least not at first, but one of insulin resistance. But if insulin is a fat-forming hormone and Type 2 diabetes is a disorder of insulin resistance, it then follows that high circulating levels of insulin in the blood, rather than insulin deficiency, could be the cause of the disease and obesity as well.

Perhaps the obese get that way not because they eat too much or exercise too little, but because they have elevated levels of insulin or their fat tissue is excessively sensitive to the insulin they secrete. Perhaps the relationship between obesity and Type 2 diabetes is not one of cause and effect, as doctors have said for years.

Berson and Yalow saw it another way: ‘We generally accept that obesity predisposes to diabetes; but does not mild diabetes predispose to obesity?’ the team wrote in 1965. ‘Since insulin is a most potent lipogenic agent, chronic [elevated insulin] would favour the accumulation of body fat.’

If Yalow and Berson’s speculation were to be true, and it certainly made sense from a biological perspective, then obesity could clearly be a hormonal/regulatory defect and Bauer and von Bergmann would have been right. Embracing this conclusion, though, depended on explaining why we become insulin-resistant. By rejecting a hormonal hypothesis of obesity two decades earlier, obesity researchers had predetermined how they would answer the question: by assuming that insulin resistance was caused by obesity, and insisting that obesity itself was caused merely by taking in more calories than expended. And that’s what they did.

The problem, as ever, appears to have been cognitive dissonance: Yalow and Berson’s revelations led both directly and indirectly to the notion that diets restricted in carbohydrates – and restricted in sugar most of all – would be uniquely effective in slimming the obese. By the mid-1960s, these carbohydrate-restricted diets, typically high in fat, were becoming fashionable, promoted by working physicians often in the form of hugely successful diet books.

Academic nutritionists led by Fred Stare and Jean Mayer of Harvard denounced these diets as dangerous fads because of their high fat content and perhaps, in Stare’s case, because of funding from the sugar and grain industries. They suggested that the physician-authors were trying to con the obese with the fraudulent argument that they could become lean without doing the hard work of curbing their perverted appetites.

This battle played out through the mid-1970s, with the academic nutritionists and obesity researchers on one side, and the physicians-turned-diet-book-authors on the other. The obesity researchers began the 1960s believing that obesity was indeed an eating disorder – Newburgh’s perverted appetite. The ongoing revolution in endocrinology, spurred by Yalow and Berson’s groundbreaking invention, failed to convince them otherwise.

The cognitive dissonance created by the biological revelations of the role of insulin in fat storage can still be seen in textbooks today. These books – for instance, the Lehninger Principles of Biochemistry, now in its sixth edition – have discussions of the regulation of fat accumulation in fat cells, in which the process is said to be driven by ‘high blood glucose elicit[ing] the release of insulin’, which favours fat storage ‘while inhibiting fatty acid mobilisation in adipose tissue’. And yet they also have sections on human obesity that state dogmatically that it is ‘the result of taking in more calories in the diet than are expended by the body’s energy-consuming activities’. Both exist side by side in the same books. Both cannot be true. The unjustifiable implication is that the mechanism determining whether or not our fat cells accumulate excessive fat are somehow different from those determining whether we become fat ourselves, despite our excess fat accumulation being merely a summation of all the excess fat stored in those cells.

Focusing on the problems of eating too much and exercising too little, health authorities have failed to target the correct causes

A far more parsimonious hypothesis is that the same thing that makes our fat cells fat makes us fat: ‘high blood glucose’ and concomitant elevated levels of insulin and the insulin resistance itself, both caused by the carbohydrate content of our diets. Insulin is secreted in response to rising blood sugar, and rising blood sugar is a response to a carbohydrate-rich meal. Sugar is implicated, in particular, because its chemical structure includes a large proportion of the carbohydrate fructose, and fructose is preferentially metabolised in the liver. As such, it is a prime suspect for the fat accumulation in liver cells that is hypothesised to be the trigger of insulin resistance itself.

If we accept von Bergmann and Bauer’s thinking that obesity is a hormonal/regulatory disorder and combine it with the revelations of the 1960s about the hormonal regulation of fat accumulation and the insulin resistance that is associated with obesity and diabetes, then the result is a very simple hypothesis that explains not just obesity but also the current epidemics and our failures to curb them. The sugars and refined grains that make up such a high proportion of the foods we consume in modern Westernised diets trigger the dysregulation of a homeostatic system that has evolved to depend on insulin to regulate both fat accumulation and blood sugar. Hence, the same dietary factors – sugars and refined grains – trigger both obesity and diabetes. By focusing on the problems of eating too much and exercising too little, public health authorities have simply failed to target the correct causes.

Scientific understanding is always driven by the tools available to do the research. These tools dictate the questions that can be asked, and the answers that can be obtained – and that, in turn, tends to shape causal hypotheses and paradigms. Ideally, when new technology comes along and new questions can be asked, then new answers can be obtained, and paradigms can shift. But this requires that the research community be open to new evidence and new ways of thinking. In nutrition and obesity research, particularly at critical times in the evolution of the science, this was simply not the case. With the epidemics of obesity and diabetes having long ago passed into crisis level, isn’t it time we finally considered seriously the possibility that our prescriptions and approaches to prevention and treatment of these diseases are simply wrong, based on incorrect paradigms and a century of misguided science?

Adapted from ‘The Case Against Sugar’ by Gary Taubes. Copyright © 2016 by Penguin Random House. Adapted by permission of Alfred A Knopf, a division of Penguin Random House LLC. All rights reserved. No part of this piece may be reproduced or reprinted without permission in writing from the publisher.