This time last year, Google searches for the keto diet outranked those for paleo for the first time. Twelve months on, keto recipe books are flying high in the publishing charts, there are emerging permutations (vegan-keto, the vegetable-heavy ketotarian) and a new industry of high-fat snacks aimed at fans is booming.

A ketogenic diet (or keto) is predominantly made up of high-fat foods – including butter, oils, meat, fish, eggs and cheese – and very low-carb vegetables such as cauliflower and leafy greens. This forces the body to burn fat for fuel, rather than glucose. In order to maintain that state, known as ketosis, followers typically restrict their carbohydrate intake to less than 50g a day. And, as protein can also be broken down into glucose, that has to be restricted, too.

By anyone’s standards, this is a tough regime to stick to – adherents even have to limit their fruit intake. Jan Vyjidak, 38, a management consultant who has contributed research into energy metabolism, including carbohydrate restriction, to Public Health Collaboration, a charity that promotes healthy lifestyles, has been on the keto diet for nearly a decade. He came to it, he says, to manage his health and improve his athletic performance. “It transformed my life,” he says. “It wiped out my constant hunger, and I was able to stop all medication for my asthma and psoriasis within six months.”

Being low in fibre, high in saturated fat and red meat, and extremely restrictive, the keto diet does not really align with nutritional guidelines issued by government experts. The diet’s critics say it is, at best, a passing trend and, at worst, a socially acceptable form of disordered eating. Its followers, however, swear by it for weight loss and health, sometimes with a fanatical passion.

Claims about the keto diet have been blown way out of proportion, driven by anecdote, hyperbole and its celebrity following (among them, Halle Berry and Vanessa Hudgens). As a registered nutritionist, I am interested in the evidence. Is it possible that such a restrictive diet, so high in fat, could be healthy or sustainable?

What we know today as keto was developed as a clinical tool more than 100 years ago. In 1911, doctors noticed that children with epilepsy stopped having seizures after two days of absolute fasting, the point when their bodies would have been forced into ketosis. Scientists later noted that ketosis could be achieved through a low-carb, high-fat (LCHF) diet without the risk of infection and mortality rates associated with absolute fasting. Today, children resistant to anti-epilepsy drugs still follow ketogenic diets to prevent seizures, under the management of specialist dietitians.

The finding has led researchers to explore other clinical applications of the keto diet. There is emerging evidence indicating that it could be used as a treatment for type 2 diabetes, but with only one major study done so far, it is too early to draw conclusions about the long-term risks or side-effects. The same can be said about the use of keto diets for cancer recovery. “Dietary interventions for medical conditions are never risk-free, and there is never one size fits all,” says Helen West, a registered dietitian and co-founder of the Rooted Project, an organisation dedicated to making evidence-based nutritional advice more accessible. “Our job as clinicians is to explain the risks and the benefits of a particular dietary intervention, based on the scientific literature and our experience, to help people make an informed choice.”

The fear, of course, is that many people don’t seek out information and support from regulated healthcare professionals before making major changes to their diet. Most newcomers are drawn to keto for its potential weight loss benefits, and, while it remains a topic of debate among nutritional scientists, its proponents typically gloss over the unknowns.

Keto advocates claim it gives them an edge for athletic performance by turning them into fat-burning machines. The theory put forward is that, in ketosis, you adapt to use stored fat for energy more efficiently – but this is not borne out by research.

“If you eat more fat, you’ll use more fat for energy,” says Alan Flanagan, a nutritional scientist. Published research to date has not shown any meaningful benefit to following a keto diet for athletic performance, he says; in fact, it may impede the ability to exercise at higher intensity.

Keto is not a free pass to go hard on the butter and bacon (too much of which may increase the risk of colorectal cancer, according to the World Health Organization). Some studies of LCHF diets have shown improvements in blood lipid profiles, which measure the levels of cholesterol and some fats, but that is typically an outcome of weight loss. Recent research found that, after three weeks of following a LCHF diet, young and healthy adult participants recorded a 44% increase, compared with controls, in low-density lipoprotein cholesterol – the “bad” kind, ultimately increasing the risk of cardiovascular disease.

That is not even accounting for the fact that following a keto diet is hard. You have to be disciplined, vigilant and hyperfocused on every food choice – you can’t casually knock back an apple because that may push you out of ketosis. This is not simply a matter of “willpower”; it is also a question of means, energy, resources and time. It prompts the question: why is this incredibly challenging, rigid, expensive diet so popular?

One answer is the “modern caveman” narrative, as strong among keto advocates as it was with its paleo predecessor. This speaks to the compelling, but misplaced, notion that illness is simply a result of modern food processing practices and lifestyles, and can be reversed by returning to a simpler time.

But a diet is a diet is a diet, and they all work the same way – by reducing the net amount of energy consumed. Keto diets may have a slight advantage, as the high fat intake is hypothesised to have satiating properties, helping you stay fuller for longer. But in reality, we have little evidence to show that keto is more effective in the long run than any other diet – or, indeed, that any diet succeeds in keeping off weight.

The real driving force behind keto’s popularity is our myopic focus on weight as the sole determinant of health, keeping us on the dieting merry-go-round as those diets become more extreme with each rotation. Yet dieting has been shown to increase body shame, anxiety, depression and disordered eating patterns, particularly binge eating and bulimia. It is also ineffective, being one of the strongest predictors for future weight gain.

The premium placed on weight means we often fail to consider improvements that can be made to health independent of it, such as physical fitness. “Health-promoting behaviours can all have a beneficial effect, even if switching your lifestyle to incorporate them doesn’t result in weight loss,” says West. “Sadly, the focus on weight loss as the only valuable goal … can lead to people pursuing weight loss at any cost, and feeling that healthy behaviours are only worthwhile if they result in an aesthetic change.”

One possible solution to diet dogma is an approach called intuitive eating: learning to eat mainly in response to physiological hunger and satiety cues. Although a relatively new area of investigation, there are promising findings: a 2017 paper found an increased awareness of internal cues to eat had “the potential to address problematic eating behaviours and the challenges many face with controlling their food intake”.

Intuitive eaters seem to have less anxiety and fewer internalised rules about food. Learning to be guided by your body also allows for flexibility and long-term sustainability, without the high likelihood of “failure” imposed by extreme external rules. For some, intuitive eating may be one answer to the question of how to break a nonstop cycle of dieting. Improving fitness is another. But the answer to long-term, sustainable good health is probably not wrapping bacon round everything.

Just Eat It: How Intuitive Eating Can Help You Get Your Shit Together Around Food by Laura Thomas (Bluebird) is out on 10 January