While a huge number of us have been persuaded by advertisers that we need our daily 'nutraceutical' dose, the benefits they claim have yet to be verified by the European Food Standards Agency

The following correction was was printed in the Guardian's Corrections and clarifications column, Tuesday 28 July 2009

In the article below we reported that the European Food Safety Authority had yet to verify the claims made by most of those marketing "nutraceutical" foods, including those said to promote digestion. A subheading mistakenly referred to that body as the European Food Standards Agency.

I recently came across an advert for a probiotic product. "Improves weight gain by 10%," it said, "… proven in more than 40 scientifically designed and controlled trials." A probiotics manufacturer boasting about weight gain? Surely some mistake? But the advert was insistent: "Maintaining the integrity of the gastro-intestinal flora effects increased weight gain," it went on.

I had strayed on to a farmers' website. This is how probiotics are marketed for pigs.

Teenagers taking their biology GCSE this summer have had to parrot a different characterisation of probiotics, as I discovered when helping one with revision. As part of their syllabus they learn that probiotics "strengthen the immune system", a phrase disconcertingly close to the pitch used by manufacturers to sell probiotics such as Yakult and Actimel to ordinary consumers. Speedier fattening, needless to say, is not a promise for which many of us are prepared to pay a premium.

It is not clear who coined the word probiotics – which simply comes from the Greek "for life". Nor is it clear how the word spread into UK school science teaching. In fact, probiotics – the name given to specialised bacteria added to foods that are said to improve digestion or general health – seemed to emerge out of nowhere in the 1990s. The live cultures or bacteria in plain live yoghurt have been said for generations to boost beneficial gut microflora, particularly when you have taken antibiotics. But plain old live yoghurt is cheap; probiotics like these are highly packaged and highly processed with the high margin for growth manufacturers need.

They are part of a whole category of heavily marketed new foods, variously called nutraceuticals or functional foods, making claims to promote our health. Human beings have managed without them for millennia, but in just 10 years an extraordinary number of us have been persuaded by the food industry that we need them for the sake of our health.

Nearly 60% of UK households now regularly buy probiotic drinks. The market is worth £164m a year in this country alone. How and why that happened is a fascinating commentary of the nature of advanced capitalism and its genius for making consumers want whatever it has to sell. The food market in affluent countries is saturated. Growth cannot come just from making us eat more, since there is a limit to our physical needs. But tap into our deep-seated emotional needs and, as political commentator Neal Lawson points out in his new book All Consuming, there is no limit to what we can be persuaded to buy.

Yakult, a yoghurt drink made by the Japanese company of the same name, was the pioneer. It burst on to the European market in the 1990s as a fermented milk drink with an added strain of healthy bacterium, Lactobacillus casei Shirota. It was launched in the UK in 1996 in a heavily sweetened drink in what look like little toy milk bottles. The utilitarian design and miniaturisation of its packaging managed to give it both an aura of healthy, almost medicinal, purpose and to make it as attractive to children as doll's house furniture. Sales took off, helped considerably by a marketing campaign worth £40m in the UK alone. Danone was quick to follow with Actimel, also packaged in dinky bottles; it now has 64% of the UK market, outselling Yakult. Danone subsequently formed a strategic alliance with Yakult and owns 20% of its shares.

So how did these drinks gain such a hold? Thanks to a detailed submission about Actimel by Danone's advertising agency to the industry's advertising effectiveness awards in 2006, I was able to gain an inside view of how the marketers managed to persuade us to buy their probiotics on a mass scale.

In 1999, that submission explains, Danone had set its sights on the UK market for yoghurt and "Actimel was chosen as lead foot soldier" with the aim of getting us "drinking Actimel every day". Persuading us that we needed to have a daily dose of this sort of premium-priced functional food was not plain sailing. Actimel spent millions on a TV advertising campaign in Britain in 2001, but by 2002 it was clear the British public was not entirely convinced by Actimel. "Forty per cent of Actimel's advertising audience [held] a highly sceptical view as to its benefit …" the submission explained.

Undeterred, the company and its ad agency set out to find a solution. Consumer research showed that if they could claim in TV advertising that Actimel's Lactobacillus casei Imunitass worked to "balance the body's intestinal flora thereby strengthening the body's natural defences" they could get people trying it. Unfortunately the ad industry's self-regulating body wasn't impressed. "The Broadcast Advertising Clearance Centre did not accept Danone's scientific evidence supporting its natural defences claim," the submission admits. It goes on: "So the question became how could we build penetration, and justify a relative price premium, among sceptics without telling them what the product does or illustrating the biological function?"

It was back to the drawing board, or to a "strategic hothouse" in summer 2002, the submission continues. They were determined to find a "strong emotive brand benefit" to get the mass market buying it as often as possible.

Yet more research showed how they could lead consumers down a certain thought process to get them to buy. They worked out that "immune system" was the catch-all phrase to sum up the body's natural defences to disease. So the logic was as follows: Actimel contains friendly live bacteria called L.c. Imunitass (note the cod Latin) therefore L.c. Imunitass is involved with the immune system; a strong immune system means feeling healthy. The feeling healthy "hot button", as the submission puts it, was pressed when represented as freedom from fatigue and resistance to catching coughs and colds … in short, feeling good.

Over the next two to three years Danone spent nearly £22m advertising Actimel in the UK. The ads used subliminal clues such as bad weather and people with coughs and colds to infer its "strengthen natural defences" and "helps your immune system" messages and showed supercharged women running around after drinking Actimel for two weeks to imply that it gave you energy. Sales rose 426%.

According to the ad agency, the profit margins on this low-cost high-price little yoghurt drink are about 40%. It reckoned Danone would have made over £36m profit as a result, getting back £1.67 for every £1 it spent on advertising.

At this time, while we were being first persuaded to buy into this new category of probiotics, there was no independent assessment by our regulators of whether probiotic products do what they claim. What independent work there was had not been encouraging. The respected Drug and Therapeutics Bulletin said when it reviewed the scientific literature in 2004 and 2005 that the evidence as to whether probiotics actually work was patchy in relation to the gut and unreliable in terms of improving general wellbeing or helping with allergies. The evidence was strongest for helping those with diarrhoea caused by antibiotics or with flare-ups of inflammatory bowel disease. People suffering from these problems might find them useful, but what the food industry needs is for as many of us as possible to believe we need to eat them as often as possible, to take them as though they were a daily dose of medicine.

I asked Danone and Yakult back in 2006 how they could justify their claims. Yakult, which markets its products as "self-defence for your gut … where the majority of your immune system is located", said it had its own independent research to support them and the fact that its bacteria survived digestion. It pointed out that Yakult is categorised as "a food for specified health use" by the Ministry of Health in Japan. "To achieve this status foods must be rigorously and independently examined to verify that consumption of the product can have a beneficial effect," it said.

Danone said it too had done extensive research on the value of its probiotics and that various further studies were underway to show that its bacteria survived. "Many beneficial effects for probiotics in general have either been demonstrated or investigated by the scientific community as a whole," it said.

It added that its probiotics were developed after more than 10 years of clinical research, it had conducted over 25 scientific studies on its bacteria, showing that Actimel had a measurable beneficial effect for healthy people when taken on a daily basis and was doing substantial further research.

In 2007 I asked the Food Standards Agency what advice it would give to consumers who wanted to know whether it was worth paying extra for probiotic products, but was told that it did not issue advice regarding the consumption of legally marketed foods products such as these. It had conducted laboratory tests on probiotics and found in simulated digestion that overall the addition of these bacteria did not affect the number of total bacteria in the gut and the supposedly healthy Lactobacillus bacteria – the good ones you are paying all that money to get into your digestive system – remained "subdominant" in laboratory conditions in a 10-day experimental period. In other words, the composition of the gut flora was not significantly changed.

But, it said, these tests were to check whether probiotics were safe rather than whether they worked, since other foods eaten at the same time and what they were digested with would make a big difference.

Until 2007 there had been pretty much a free-for-all over functional foods' marketing slogans in Europe. But that year new EU regulations were finally agreed, largely as a result of pressure from consumer groups who demanded that food companies should be prevented from using health claims to sell products unless they have proved them first and unless the products are healthy enough overall to merit carrying a health claim. Under the new rules, member states were asked to submit health claims from their manufacturers with dossiers of scientific evidence to be independently assessed by European Food Safety Authority (Efsa).

The UK Food Standards Agency collected over 4,000 claims from industry, weeded out half of them, and submitted about 2,000 for the new approvals process. Across Europe some 44,000 claims were submitted. Efsa scientists, stunned by the volume of claims, managed to whittle these down to 4,000 for detailed assessment.

This week, it emerged that Efsa has now rejected some of the claims made for high-profile functional food products by mainstream manufacturers. In fact, the authority's scientists have assessed about 70 health claims so far and have rejected 66 – the overwhelming majority.

They ruled, for example, that there was insufficient scientific evidence to support the health claim made by Unilever that its Lipton Tea helps increase alertness, or by Ocean Spray that its cranberry juice helps protect against urinary tract infections.

So what about probiotics? To date, Efsa has published five opinions on probiotics' claims, although the claims of the leading brands have yet to be assessed. Efsa scientists "avoid using the term probiotics", a spokeswoman told me, but confirmed that all five opinions on probiotics published so far have been negative.

The leading manufacturers of probiotics, Danone and Yakult, both told me this week that they are confident that the scientific evidence for their claims is robust and that they will be approved by Efsa. But the fact is, nearly a decade after they achieved mass consumption, we are still waiting to hear whether Efsa's scientists think they work or not.

Following the stream of negative opinions from Efsa for other health claims, the food industry as a whole has been quietly getting more and more concerned. Its trade groups have called for urgent meetings with Efsa about how the process is working, and it has accused the authority in the trade media of using "gold standards" to assess the health-supporting dossiers that are more appropriate to new pharmaceuticals. It has also complained of lack of clarity in technical aspects of the process.

Danone withdrew its applications for approval of the health claims it makes for Actimel (helps strengthen the body's natural defences) and Activia (improves digestive comfort; improves slow transit) in April. It says, however, that it will resubmit them when technical confusions have been sorted out and that it is absolutely confident that they are well supported by science and will win Efsa approval.

"The three health claim applications are related to Danone's fresh dairy products Activia and Actimel, that are among the probiotic products with the most extensive scientific documentation. Their efficacy has been shown in many different studies and the results have been published in highly reputed international scientific journals," Danone told us in a statement. "We welcome the new process and remain committed to working with Efsa."

The company declined to give a date for its resubmission or to comment on the fact that its US arm, Dannon, earlier this year reached an out of court settlement in one of several class actions brought against it over its probiotic health claims. Dannon vigorously denied the accusation originally made in the action that it was misleading consumers with a multimillion-dollar advertising campaign that claimed its probiotics had been clinically proven to help regulate the digestive system and strengthen the body's defences when consumed daily. The plaintiffs claimed compensation for the 30% premium over other dairy products paid by consumers and a corrective advertising campaign.

The new EU regulations will not be fully implemented until 2020. Whether Efsa in the end rules that the leading manufacturers' claims about probiotics are indeed supported by the scientific evidence they submit or not, even Danone has conceded to me in the past that "results reported by individual Actimel users can be as individual as their gut flora profile and overall diet and other lifestyle factors".

In other words, there is no silver bullet food ingredient that will make us feel healthy and well. But perhaps even the fattening pigs could have told us that.

Felicity Lawrence's Eat Your Heart Out: Why the Food Business is Bad for the Planet and Your Health, is published by Penguin