I am on a small ward at St Thomas’ hospital in London. There are six beds with a view of the Thames, which is lit by November sunshine. I’m in good health, but scanning my timetable – I’m here taking part in a two-week scientific study – is making me feel anaemic. “8.35am: 1st blood draw. 10.05am: 2nd blood draw. 10.20am: 3rd blood draw. 10.50am: 4th blood draw…” on it goes. There will be 10 blood draws in total today, each filling three vials, to be tested for levels of glucose, fat, insulin and other clues as to how efficiently my body metabolises different foods. But these are not just any foods; they are lab-baked muffins with precisely varying amounts of fat, protein, carbohydrate and fibre.

The aim of this study – Predict, a collaboration between King’s College London and Harvard and Stanford medical schools in the US – is to measure thousands of people’s responses to different foods and discover why, when it comes to health, different diets suit different people. The hope is that when enough data has been gathered researchers (using AI), and eventually an app called Zoe, will be able to predict individual healthy and unhealthy food choices for each of us.

Personalised nutrition is a public-health holy grail. Geneticists have striven for decades to develop diets based on genotype as a sophisticated form of preventive medicine. The idea is that eating what is optimal for your unique physiology could shield you from the particular diseases to which you may be susceptible, from certain cancers to diabetes. Such is the complexity of genetics, however, that our understanding is still too fragmented. Not to mention disproving any one-size-fits-all diet advice.

Tim Spector, a geneticist at King’s College London, who has in recent years become a leading researcher into the role of gut microbes in health, came up with Predict to help achieve the dream of healthcare becoming preventive, rather than, as it is now, focused on treating disease. “It’s a real paradigm shift of how people are going to give advice in the future,” he says. “We were stuck in this time warp of: ‘Fasting blood test for cholesterol, step on the scales and everyone’s the same.’ It’s clearly much more complicated than that. That’s why diets, if it’s a low-fat diet, a low-carb diet or calorie counting, don’t suit everybody. Despite all the advice, we’re all getting fatter.”

The researchers need my blood on tap all day, so I have a cannula inserted into a vein in my left arm. Next, I get a blood-glucose monitor fitted to the back of my right arm. This great gadget, called the FreeStyle Libre, is now widely used by people with type 1 diabetes. A discreet plastic disc about the size of a £2 coin is stuck on to my skin. Only when I remove it after two weeks do I see that a thin metal probe, a few millimetres long, has been in my arm the entire time. The data stored on the disc is read by a handheld device that shows what my blood sugar has been up to.

By late morning I have a splitting headache and feel tired and confused. I tell a nurse, fearing that I’ve started running out of blood. She merrily scans my arm to reveal that I’m hypoglycaemic, my glucose level having shot up after breakfast and then straight down into the red zone.

The breakfast muffins my fellow participants and I were given were an off-putting shade of indigo, unwholesomely oily and had to be washed down within 10 minutes with yucky-flavoured milk. I now know that they contained twice as much fat as my regular muesli with yoghurt and berries, three times the sugar and precisely no fibre. The lunch muffins are mercifully beige and more fibrous. Back home over the next 11 days, I’ll be eating many more muffins with different nutritional configurations unknown to me. My task will be to attempt to live normally, while eating muffins or other set meals (which include one breakfast of an unbearably large glass of water with glucose dissolved in it) at set times. I’m allowed to eat my own dinners, though, and my own food for the last two days.

I have to log all food and drink (even water) in the app, complete with photographs and the weights of individual ingredients. The Predict app tries to make things easy with a friendly dietician only an instant message away, and it prompts if I’ve messed up (cooking from scratch becomes quite the kerfuffle). The blood-letting continues at home. For the first four mornings, at three precise times, I must prick a finger and squeeze out five blobs of blood on to cards, to be refrigerated in foil pouches until ready to be collectively posted back to the lab.

Speaking with Spector seven months later, before the release of his first results, it seems to have been worth the effort. “The variability between people is even greater than I thought,” he says. “The range of responses to the same foods is huge. To my mind that knocks on the head any idea of an average response or average nutrition. That’s the really big message there.” Analysis of the Predict data has found that the macronutrient breakdown – such as you find on food labels – can account for only 40% of how an individual might respond to meals.

As a geneticist, Spector started the research registry Twins UK at Kings College in 1992, and has amassed data on 13,000 twins. Almost 250 pairs of identical twins have taken part in Predict and, surprisingly, there were big differences in their responses to foods, too. “Clearly our response to food isn’t like many other things like height or weight or anything like that,” says Spector.

This means that genetics isn’t the dominant factor in shaping our food responses, which is good news. We’re stuck with our genes, whereas, says Spector, “if you want to modify your responses by changing your behaviour, you can do a lot about it”. Improving how well you can process and clear blood sugar, for instance, is possible by “selecting foods that produce a smaller peak, altering meal times to suit your circadian rhythm, improving your diversity of gut microbes and exercising within the same time to reduce the peaks”. Other likely mechanisms under investigation include sleeping better and going to bed earlier.

Vital to Predict’s success will be finding a convenient way to determine which foods help or harm which people. The Zoe app will need biological clues – or biomarkers – in order to make predictions, ideally obtained from a quick blood test or stool sample. This is why the researchers are measuring hundreds of metabolites in participants’ blood after eating – such as ketones – as well as our gut microbe diversity.

We all have up to 2kg of bacteria, yeasts and other microbes living in our guts and emerging science is finding that our gut microbiomes could be even more revealing about the workings of our bodies than our genes. Humans share more than 99% of our genes, but only 25% of our microbes. Even the study’s identical twins share only 37% of their gut microbes. “The gut microbiome is so related to health,” says Spector. “It’s related to your weight, your propensity to diabetes, it controls how you metabolise fat …” There is evidence it can also govern gene expression, mood and appetite, make vitamins, and may well be the ultimate biomarker for our food responses. Like genetics, the microbiome is so complex that progress in unravelling its mechanics is slow; Spector suspects that simpler biomarkers found in the blood may become apparent first. Time, and the data, will tell.

Despite the full-on nature of taking part in the study, Spector says 98% of participants completed it. “Normally, about 20% of people drop out,” he says. This retention rate is largely, I suspect, due to self-interest. Taking part in Predict offers a sneak preview of the personalised nutrition of the future, and a chance to learn about your metabolic quirks.

Spector goes through some of my results with me, pointing out, for instance, that if I switch from avocado to peanut butter, considerably more glucose lingers in my bloodstream afterwards – even though the peanut butter meal is slightly lower in fat and calories. He says that another regular quick lunch – a low-calorie, bacon and lentil soup – “for you, wasn’t particularly healthy” – as it leaves me with unexpectedly high blood glucose. Spector, who ditched his go-to tuna sandwich after taking part in the study, suggests it may be worth keeping an eye on how lentils affect me in future. On the plus side, the data reveals that my usual breakfast seems to be fine.

Next we look at my overall metabolic profile, which scores how well I respond to (or process) fats and carbohydrates compared with all the participants (who are all healthy and disease free). “Your glucose response is pretty much average for your age. It’s not super good, but it’s definitely not bad. And your fat response is slightly better than the average for your age and sex,” he says.

I am chuffed to discover that my gut microbiome richness is relatively high. “You got a score of 70 which is really good,” says Spector. “Virtually nobody got over 80. I’m assuming you have a good, diverse plant-based diet?” Indeed I have been geekily making an effort to feed my microbes in recent years, even fermenting my own sauerkraut, so this is encouraging news. More positive data comes from my Dexa Scan result, which shows that I have low visceral fat around the liver and intestines.

For now, we just have a snapshot, he says, “but as we get more data from other people our models will improve, and in six months or a year we want to be able to give you some form of an app that will give you a prediction of what response you’re going to have to certain foods”. This will then evolve into Zoe – the great hope for the personalised-health revolution. “But we’re not there yet,” says Spector. “This is part of the journey.”

• Join the study at joinzoe.com