I decided to take a try at the great problem of our time: how to lose weight without any effort. So I did an experiment on myself. I was ripe for it, if truth be told. Here I am eight months later and 50 pounds lighter, so something must have worked. My approach to the problem was different from the usual perspective. I’m a psychologist, not a doctor. From the start I suspected that weight regulation was a matter of psychology, not physiology.

If weight were a matter of calories in and calories out, we’d all be the weight we choose. Everyone’s gotten the memo. We all know the ‘eat less’ principle. Losing weight should be as easy as choosing a shirt colour. And yet, somehow it isn’t, and the United States grows heavier. It’s time to consider the problem through an alternative lens.

Whatever else it is, hunger is a motivated state of mind. Psychologists have been studying such states for at least a century. We all feel hungry before dinner and full after a banquet, but those moments are the tip of the iceberg. Hunger is a process that’s always present, always running in the background, only occasionally rising into consciousness. It’s more like a mood. When it slowly rises or eases back down, even when it’s beneath consciousness, it alters our decisions. It warps our priorities and our emotional investment in long-term goals. It even changes our sensory perceptions – often quite profoundly.

You sit down to dinner and say: ‘That tiny, little hamburger? Why do they have to make them so small? I’ll have to eat three just to break even.’ That’s the hunger mood making food look smaller. If you’re full, the exact same hamburger looks enormous. It isn’t just the food itself. Your own body image is warped. When the hunger mood rises, you feel a little thinner, the diet feels like it’s working and you can afford a self-indulgence. When satiety kicks in, you feel like a whale.

Even memory can be warped. Suppose you keep a log of everything you eat. Is that log trustworthy? Not only have you drastically misjudged the size of your meals, but you’ve almost certainly forgotten items. Depending on your hunger state, you might snarf up three pieces of bread and after the meal sincerely remember only one. One recent study found that most of the calories people eat come through snacks between meals. But when you ask people, they deny it. They’re surprised to find out just how much they snack.

The hunger mood is hard to control, precisely because it operates outside of consciousness. This might be why obesity is such an intractable problem.

The hunger mood is controlled by the brain stem. The part most responsible for regulating hunger and other basic motivated states is called the hypothalamus, and it sits at the bottom of your brain. It has sensors that literally taste the blood. They detect levels of fat, protein and glucose, as well as blood pressure and temperature. The hypothalamus gathers this data and combines it with sensory signals that percolate in through other systems in the brain – fullness in the gut, the feel and taste and smell of food, the sight of food, even the time of day and other surrounding circumstances.

Given all this data, the neural circuits train up on our dietary habits. That’s why we get hungry at certain times of the day – not because of an empty stomach, but because of a sophisticated neural processor that anticipates the need for more nutrition. If you skip a meal, at first you feel acutely hungry, but then you actually begin to feel less hungry again as that accustomed mealtime passes by. That’s also why we get full at the end of a meal. Again, not because of a full stomach. If that’s your only signal, then you’re drastically overeating. As counterintuitive as it might sound, there’s normally a healthy gap between feeling full and having your stomach actually full. Psychological fullness is a feeling of sufficiency that comes from a much more complex computation. The hypothalamus in effect says: ‘You’ve just eaten a burger. I know from past experience with burgers that in about two hours the protein and fat in your blood will rise. Therefore, in anticipation, I’ll turn off your hunger now.’ The system learns, anticipates, and regulates. It operates in the background. We can consciously interfere with it, but not usually to good effect.

Take in fewer calories and you’ll lose weight. But explicitly try to reduce calories, and you’ll do the exact opposite

Here’s what happens when you interfere with your hypothalamus – when medical advice collides with psychology. Let’s say you decide to cut back on calories. You eat less for a day. The result? It’s like picking up a stick and poking a tiger. Your hunger mood rises and for the next five days you’re eating bigger meals and more snacks, perhaps only vaguely realising it. People tend to judge how much they’ve eaten partly by how full they feel afterward. But since that feeling of fullness is partly psychological, if your hunger mood is up, you might eat more than usual, feel less full than usual, and so mistakenly think that you’ve cut back. You might feel like you’re making progress. After all, you’re constantly vigilant. Sure, now and then you slip up, but you get yourself right back on track again. You feel good about yourself until you get on a scale and notice that your weight isn’t responding. It might go down one day and then blip up the next two days. Dancing under the surface of consciousness, your hunger mood is warping your perceptions and choices.

I’m not denying the physics here. If you take in fewer calories, you’ll lose weight. But if you explicitly try to reduce calories, you’re likely to do the exact opposite. Almost everyone who tries to diet goes through that battle of the bulge. Diets cause the psychological struggle that causes weight gain.

Let’s say you try another standard piece of advice: exercise. If you burn calories at the gym you’ll definitely lose weight, right? Isn’t that just physics? Except that, after you work out, for the rest of the day you’re so spent that you might actually burn fewer calories on a gym day than on a regular one. Not only that, but after a workout you’ve assuaged your guilt. Your emotional investment in the cause relaxes. You treat yourself to a chocolate chip muffin. You might try to be good and decline the muffin, but the exercise revs up that subtle hunger mood lurking under the surface and then you don’t even know any more how much you’re overeating. Meals grow bigger while seeming to grow smaller. Extra snacks sneak in.

Let’s say you’ve tried all the standard advice – every diet out there. Some of them might even work for a short time, until you fall off the wagon and gain back even more than before. After a while you start to doubt your willpower. If the prevailing medical theory is correct, if weight is a matter of calorie control, then your problem is a weak character. It’s your own fault. That’s the message beamed across our culture from all directions.

But the concept of willpower is anathema in psychology. Cognitive control is much more subtle, complex and limited in its ability than the lay notion of willpower. That notion is false and harmful to mental health. What is willpower anyway? It’s pitting long-term rewards against short-term rewards, and you’re going to fall off that wagon sooner or later. Every time you fall off, you do more damage than you can undo by climbing back on again. And even when you think you’re firmly on the wagon, most of the psychological complexity runs under the surface of consciousness and therefore you can’t possibly realise how much you’re sabotaging your own efforts.

Where does that leave you? At the end of that seemingly inevitable progression, you’re demoralised and depressed. You can do anything else you put your mind to but somehow you can’t manage the weight loss. And so you enter a disastrous spiral. If you’re going to be miserable anyway, you might as well indulge yourself. The food at least mitigates the misery. You slip into comfort eating, self-medication and addiction, and lose all motivation. You fall into the deepest part of the psychological quagmire and your chance of recovery is small. A recent study showed that if you’re obese, your chance of getting back into the normal range is less than one in 100.

Most doctors, trainers, and healthcare professionals think about weight from the perspective of chemistry. It’s calories in versus calories out. Eat less, exercise more. Different schools of thought posit that all calories are equivalent, or that fat calories are especially bad, or that carbohydrate calories are particularly to be avoided. All these approaches focus on the way that calories are digested and deployed in the body. They ignore psychology. Most studies treat the psychology of hunger as an inconvenience. A ‘properly’ controlled study forces participants to eat a set amount of calories, thus screening out the annoying influence of autonomous human behaviour. And still, for all that has been learned from this mainstream medical approach, the advice is failing us. More than two-thirds of the US population is overweight. More than one-third is obese.

As I reviewed the discouraging rise of obesity in the US and all over the world, and the discouraging shrinking space between my own belly and my desktop, it seemed to me that the mainstream focus is almost entirely wrong. The obesity epidemic is not an issue of calories or willpower. I began to suspect that our problem with obesity is a problem of poisoning the normal regulatory system. We possess a system that’s intricate and beautifully calibrated. It evolved over millions of years to be good at its job. It should work in the background without any conscious effort, but for more than two-thirds of us it doesn’t. What are we doing to ourselves to screw up the hunger and satiety system?

For about a year, I experimented on myself. I used what’s called an event-related design, which involved some arduous sacrifice (or at least some boredom). Simply put, I ate the same damn thing every day to establish a consistent baseline. I measured weight, waistline, and kept notes on everything I could think of. Then I changed one thing in one meal and monitored its tiny, perturbing effect over the next several days. When the measurements went back to baseline, I’d try a new perturbation. Each tweak by itself gave a small signal, but after a while I could average across many events and watch the pattern emerge. Of course I had no illusions of discovering anything new. This wasn’t formal science. It had a sample size of one. The point was to find out which of all the conflicting advice flying back and forth out there resonated with my own personal data. What should I believe?

As usual, the most instructive part of the experiment turned out to be an incidental observation. Never mind whether some foods grew or shrank my poundage. I noticed instead that some acts grew or shrank my level of hunger. I knew when my hunger mood was up, even if I didn’t consciously feel hungry, because somehow I’d end up at the lunch deli early. And after I finished eating, it didn’t seem like I’d had as much food as usual. Maybe they’d slipped me a smaller sandwich?

When my hunger mood was down, the roster of priorities would shift and I’d get caught up in my work. Somehow lunch would get delayed by an hour. My moment-by-moment decision-making was warped. Each time it happened it seemed as though there was some other reason for it, but I couldn’t ignore the pattern accumulating in my notes.

Three bad habits appeared to consistently boost my hunger. I call them the super-high death-carb diet, the low-fat craze, and the calorie-counting trap.

The super-high death-carb diet has become normal US fare. We get up in the morning and eat a croissant, or pancakes with syrup, or a muffin. Or cereal and milk. The cereal is all carbs. Then comes lunch. Suppose I’m unhealthy and eat a fast-food, McDonald’s lunch. We think of it as greasy food, but beyond the grease the burger has a bun and the ketchup is sugar paste. The fries are all carbs. The large soda is sugar water. The grease is only a tiny part of the meal. Maybe you feel morally superior and prefer a ‘healthy’ lunch, a deli sandwich that’s mainly French bread. And chips. And a Snapple. All carbs.

The afternoon snack is some sugary beverage at Starbucks and a cookie. Or a power bar, which is a candy bar with spin. If you’re good, maybe a banana, which is as high carb as you get in the fruit world. Dinner? Piled with potatoes, pasta, rice, bread. We think we’re healthy eating sushi but it’s mostly rice. Maybe you go for a nice healthy soup. It’s thickened with flour and has noodles and potatoes. And every meal comes with soda, or juice, or ice tea, or some other sweetened drink. Then dessert. Then a snack before bed. It’s all carbs. You can’t walk through a supermarket without being assaulted by carbs on all sides. Some people talk about complex carbs versus refined sugar. They have a point, but take out the refined sugar and it’s still a staggering amount of carbohydrates. The super-high death-carb diet has warped our sense of normal.

The low-carb people might be right for the wrong reasons. Starting with Robert Atkins, the American cardiologist who first popularised the diet, an entire physiological theory has sprung up. In that theory, if you cut out enough carb, your body switches from using glucose to using ketones as the main energy-transporting molecule in your blood. By using ketones, the body begins to draw on its fat reserves. Moreover, by reducing blood sugar, you reduce insulin, the main hormone that promotes the deposition of fat in the body. Less carbs, less fat. The theory sounds good and might have some validity, but its impact on obesity remains controversial. One recent paper seems to smack it down entirely.

The study monitored two groups of people. For six days, one group ate low-carb, the other low-fat. Both were strictly forced to eat the same number of calories. The result? The low-carb group did not lose more weight. Actually, the low-fat group did. The low-carb people might have reduced their insulin, but the theory didn’t really translate into magical weight loss. Given all that contradiction, what can we say about the low-carb approach?

Skip breakfast, cut calories at lunch, eat a small dinner, be constantly mindful of the calorie count, and you poke the hunger tiger

The theory and the experiments might be right as far as they go, but they miss the most important point. They emphasise how calories are deployed in the body instead of emphasising the motivated state of hunger. It would be encouraging to see more studies on how different diets affect hunger regulation. It is now well-established that a high-carbohydrate diet increases your hunger. A low-carb diet removes that stimulant. Taking all this together, the evidence suggests that a low-carb diet doesn’t make you lose weight because of its effect on your energy utilisation. It makes you lose weight because you eat less. Or (perhaps more accurately), the ridiculous, super-high death-carb diet stokes up the hunger mechanism and your eating goes out of control.

Because that hunger state runs mostly beneath consciousness, it’s easy to misattribute the result. But in the end, if you follow the death-carb diet to its conclusion, you can’t help noticing the effect on your appetite. Extremely obese people reach a point where they’re always hungry, never full. They can eat six dinners’ worth until their stomachs feel stretched and terrible, about to split in the middle, but the brain isn’t satisfied.

The low-fat craze works the same way. I grew up in the era when public service commercials on TV warned us about the dangers of fat. Poor data and a rush to conclusions might have led the medical community to that recommendation. Don’t eat butter. Don’t eat eggs. Don’t drink whole milk. Take the skin off chicken. Eat low-fat yogurt (which is still chock-full of sugar). Dietary fat might have its medical downside; I don’t think the data are perfectly clear yet. But cutting out the fat has led to a disaster. As numerous studies have now established, fat reduces hunger. Take it away and the hunger mood soars. It’s not a simple relationship, and the effect is gradual. Remember, your hypothalamus takes in complex data and learns associations over time. Give it a few months of training with a diet that’s stripped of fat, and it will ratchet up your sense of hunger.

But the most insidious attack on the hunger mechanism might be the chronic diet. The calorie-counting trap. The more you try to micromanage your automatic hunger control mechanism, the more you mess with its dynamics. Skip breakfast, cut calories at lunch, eat a small dinner, be constantly mindful of the calorie count, and you poke the hunger tiger. All you do is put yourself in the vicious cycle of trying to exert willpower and failing. That’s when you enter the downward spiral.

All three of these effects – high carb, low fat, and calorie counting – are increasingly evident in the scientific literature on diet, and also showed up in my self-observations. Amazingly, even a small tweak to one meal on one day had a noticeable effect on my hunger mood.

At the end of all my self-observations and meditations, the time had come to put the theory to a test. I tried a simple formula. First, moderately low-carb. The Atkins and Paleo diet purists would scoff. I reduced my carbohydrate intake by about 90 per cent and in doing so came nowhere near a low-carb diet. I wanted to avoid the super-high death-carb diet that most of us eat most of the time. Second, a little higher fat. I know some people swear by high fat and snack on entire sticks of butter. I don’t know what the research is on that kind of thing, but all I wanted was to avoid the extremity of a diet stripped of fat. Third, I could eat as much as I like at each meal. That last proposition was the hardest. When you want to lose weight, it’s hard to wrap your mind around the concept of eating more. I simply had to trust a bizarre psychological twist: if I try to eat less, I’ll end up eating more.

I could give a list of foods – salmon, peanut butter, pork chops, apples, tomatoes, chicken with the skin, tofu, eggs, and on and on – but really the concept is more revealing than the details. The diet had nothing to do with standard health advice. It had nothing to do with how those particular foods chemically affect my body. I wasn’t thinking of my arteries or my liver or my insulin. The approach was designed to speak to my unconscious hunger control mechanism, to encourage it to eat less. And it worked at a slow drip of about two pounds a week, trailing off finally to a much more comfortable weight. Twenty years of accumulation, 50 extra pounds (I cringe to admit it) went away in a few months.

There is no effort in an all-I-want diet of moderately fat comfort food. I simply sat back and watched my brainstem do its thing

The beauty of the method was that it required no effort. By effort, I mean that dubious concept of willpower. Pitting long-term goals against short-term rewards. When the hunger mood rises, the personal struggle is heartbreaking. I know all about that struggle and the weird thing is, the struggle is alluring. It might be dreadful, and it might be counterproductive, but it makes you feel like you’re doing something. Our society is impressed by hard work. Think of those people exercising maniacally on that TV show The Biggest Loser. We expect progress to be punishing, and we admire the people who push themselves to super-human limits. Another psychological trap, I guess. None of that self-flagellation turned out to be necessary. I had to reconcile myself to what felt like a lazy method. There is really no effort in an all-I-want diet full of moderately fat comfort food. I simply sat back and watched my brainstem do its thing.

I don’t think I’m alone in this experience. Others have tried a similar diet, though perhaps for other reasons. Advocating for one particular weight-loss diet isn’t my point. My message is this: your weight is in large measure about your psychology. It’s about the hunger mood. Obesity is a crippling social problem, but to our detriment the research has almost uniformly ignored this aspect of the situation. Consider this to be a call to science to focus a great deal more on the psychology of the hunger mood.

In some ways, the hunger system is like the breathing system. The brain has an unconscious mechanism that regulates breathing. Suppose that system got shut down so that it was up to you to consciously control your own breath, adjusting its rate and depth depending on factors such as blood oxygen, carbon dioxide level, physical exertion, and so on. What would happen? You’d die in about 10 minutes. You’d lose track of the necessities. The intellectual, conscious mind is not really good at these matters of regulating the internal environment. It’s better to leave the job as much as possible to the dedicated systems that evolved to do it. What you can do with your conscious mind is to set the general parameters. Put yourself in a place where your automatic systems can operate correctly. Don’t put a plastic bag over your head. Likewise, don’t eat the super-high death-carb, low-fat diet. Don’t micromanage your brainstem by counting every calorie. You might be surprised at how well your health self-regulates.