As the pace of medical innovation slows to a crawl, how do drug companies stay in profit? By 'discovering' new illnesses to fit existing products. But, says Ben Goldacre, for most problems the cure will never be found in a pill

When you've been working with bullshit for as long as I have, you start to spot recurring themes: quacks and the pharmaceutical industry use the exact same tricks to sell their pills, everybody loves a "science bit" - even if it's wrong - and when people introduce pseudoscience into any explanation, it's usually because there's something else they're trying desperately not to talk about. But my favourite is this: alternative therapists, the media, and the drug industry all conspire to sell us reductionist, bio-medical explanations for problems that might more sensibly and constructively be thought of as social, political, or personal. And this medicalisation of everyday life isn't done to us; in fact, we eat it up.

In 2007 the British Medical Journal published a large, well-conducted, randomised controlled trial, performed at lots of different locations, run by publicly funded scientists, that delivered a strikingly positive result: it showed that one treatment could significantly improve children's antisocial behaviour. The treatment was entirely safe, and the study was even accompanied by a very compelling cost-effectiveness analysis.

Did this story get reported as front-page news in the Daily Mail, natural home of miracle cures (and sinister hidden scares)? Was it followed up on the health pages, with an accompanying photo feature, describing one child's miraculous recovery, and an interview with an attractive happy mother with whom we could all identify?

No. This story was unanimously ignored by the entire British news media, despite their preoccupation with antisocial behaviour, school performance and miracle cures, for one very simple reason: the research was not about a pill. It was about a cheap, practical parenting programme.

Meanwhile, for over five years now, newspapers and television stations have tried to persuade us, with "science", that fish-oil pills have been proven to improve children's school performance, IQ, behaviour, attention, and more. As I have documented with almost farcical repetitiveness in this paper, these

so-called "fish-oil trials" were so badly designed that they amounted to little more than a sham. In the case of the biggest, "the Durham trial", the county council has refused even to release the results, which I have every reason to believe were unflattering.

I'm not desperately interested in whether fish-oil capsules improve children's IQ, and I say this for a number of reasons. Firstly, I'm not a consumer journalist, or a lifestyle guru, and I am not in the business of handing out "readers' health advice". Also, if you think about it rationally, any beneficial effects of fish oil on school performance will probably not be all that dramatic. We do not have an epidemic of thick vegetarians, and humans have shown themselves to be as versatile as their diets are diverse, from Alaska to the Sinai desert.

But I wouldn't start with molecules, or pills, as a solution to these kinds of problems. The capsules Durham are promoting cost 80p per child per day, while it spends only 65p per child per day on school meals, so you might start there. Or you might restrict junk-food advertising to children, as the government has recently done. You might look at education and awareness about food and diet, as Jamie Oliver recently did very well, without recourse to dodgy pseudoscience or miracle pills.

But you might also step away from obsessing over food just for once and look at parenting skills, teacher recruitment and retention, orsocial exclusion, or classroom size, or social inequality and the widening income gap. Or parenting programmes, as we said right at the beginning. In fact, Durham's GCSE results, where the "trial" was performed, improved far more in the year before the fish-oil pills were introduced, after a huge input of extra funding and, more importantly, extra effort from local teachers and the community. But the media don't report stories like that: because "pill solves complex social problem", even if it's not true, is a much better angle.

The fish-oil story is by no means unique: repeatedly, in a bid to sell pills, people sell a wider explanatory framework, and as George Orwell first noted, the true genius in advertising is to sell you the solution and the problem. Pharmaceutical companies with serotonergic antidepressant drugs to sell have worked hard, in their direct-to-consumer advertisements and their lobbying, to push the "serotonin hypothesis" for depression - the idea that low serotonin causes low mood - even though the scientific evidence for this theory is growing thinner every year. Meanwhile the nutrition supplements industry promotes dietary deficiencies as their treatable cause for low mood.

But this fish-oil story is also a classic example of a phenomenon more widely described as "medicalisation", the expansion of the biomedical remit into domains where it may not be helpful or necessary. In the past, commentators have portrayed this as something that doctors inflict on a passive and unsuspecting world, an expansion of the medical empire; in reality, it seems that these reductionist biomedical stories can appeal to us all, because complex problems often have depressingly complex causes, and the solutions can be taxing and unsatisfactory.

In its most aggressive form, this process has been characterised as "disease-mongering". It can be seen throughout the world of quack cures - and being alive to it is like having the scales removed from your eyes - but to understand its relevance to the pharmaceutical industry, we need a quick primer in medical history.

Before 1935 doctors were basically useless. We had insulin, morphine for pain relief - a drug with superficial charm, at least - and we could do operations fairly cleanly, although with huge doses of anaesthetics, because we hadn't yet sorted out well-targeted muscle-relaxant drugs. Then suddenly, between the 1930s and the 1970s, science poured out an almost constant stream of miracle cures.

Everything we associate with modern medicine happened in that time: antibiotics, dialysis, transplants, intensive-care units, heart surgery, every drug you've ever heard of, and more. For people who were ill, the difference was spectacular. If you got TB in the 1920s you died, pale and emaciated, in the style of a romantic poet. If you got TB in the 1970s, then in all likelihood you would live to a ripe old age. You might have to take rifampicin and isoniazid for months on end - they're not nice drugs, and the side-effects make your eyeballs and urine turn pink - but if all goes well you will live to see inventions unimaginable in your childhood.

Times have changed. The pharmaceutical industry is in trouble: the golden age of medicine has creaked to a halt, the low-hanging fruit of medical research has all been harvested, and the industry is rapidly running out of new drugs. Fifty "novel molecular entities" a year were registered in the 1990s, but now it's down to 20, and many of those are just copies of other companies' products, changed only enough to justify a new patent. So the story of "disease mongering" goes like this: because they cannot find new treatments for the diseases we already have, the pill companies have instead had to invent new diseases for the treatments they already have.

Recent favourites include social anxiety disorder (a new use for SSRI antidepressant drugs), female sexual dysfunction (a new use for Viagra in women), the widening diagnostic boundaries of "restless leg syndrome", and of course "night eating syndrome" (another attempt to sell SSRI medication, bordering on self-parody) to name just a few: all problems, in a very real sense, but perhaps not necessarily the stuff of pills, and perhaps not all best viewed in reductionist biomedical terms. In fact, you might consider that reframing intelligence, loss of libido, shyness and tiredness as medical pill problems is a crass, exploitative, and frankly disempowering act.

Selling us crude biomedical mechanisms may well enhance the placebo benefits from pills, but these stories are also seductive precisely because of what they edit out. In the media coverage around the rebranding of Viagra as a treatment for women in the early noughties, and the invention of female sexual dysfunction, for example, it wasn't just the tablets that were being sold: it was the explanation.

Glossy magazines told stories about couples with relationship problems who went to their GP, and the GP didn't understand their problem (the first paragraph of any medical story in the media). Then they went to the specialist, and he didn't help either. But then they went to a private clinic. Here they did blood tests - hormone profiles, esoteric imaging studies of clitoral bloodflow - and then they understood. The solution was in a pill, but that was only half the story, and the diagnosis was almost more important: she had a mechanical problem Rarely was there a mention of any other factors, that she was feeling tired from overwork, that he was exhausted from being a new father, or finding it hard to come to terms with the fact that his wife was now the milky mother of his children, and no longer the nubile sex vixen he first snogged on the floor of the student union building to the sound of Don't You Want Me? by the Human League in 1983.

This is because we don't want to talk about these issues, any more than we want to talk about social inequality, the disintegration of local communities, the breakdown of the family, the impact of employment uncertainty, changing expectations and notions of personhood, or any of the other complex, difficult factors that play into the apparent rise of antisocial behaviour in schools.

This wishful deafness to the clamour of reality reaches its purest form in our newfound obsession with food, as if it was the most important lifestyle risk factor for ill health, as if every technical detail should be devoured and acted on, for the promise of eternal zest. From the Daily Mail's ongoing project to divide all the inanimate objects in the world into ones that either cause or cure cancer, to daytime television's obsession with the healing power of this week's magic berry, there is no end to this material. Should you believe it? No. I have demonstrated time and again how these claims are flawed in their own specific cases. But something more interesting is being ignored in the background.

The World Health Organisation's Commission on the Social Determinants of Health reported this week, and it contained some chilling figures. Life expectancy in the poorest area of Glasgow - Calton - is 28 years less than in Lenzie, a middle-class area just eight miles away. That is a lot less life, and it isn't just because the people in Lenzie are careful to eat goji berries for extra antioxidants, and a handful of brazil nuts every day, thus ensuring they're not deficient in selenium, as per nutritionists' advice.

People die at different rates because of a complex nexus of interlocking social and political issues including work life, employment status, social stability, family support, housing, smoking, drugs, and possibly diet, although the evidence on that, frankly, is pretty thin, and you certainly wouldn't start there.

But we do, because it's such a delicious fantasy, because it's commodifiable and pushed by expert PR agencies, and in some respects this is one of the most destructive features of the whole nutritionist project, graphically exemplified by figures such as Dr Gillian McKeith PhD. Food has become a distraction from the real causes of ill health, and also, in some respects, a manifesto of rightwing individualism. You are what you eat, and people die young because they deserve it. You hear it from people as they walk past the local council estate and point at a mother feeding her child crisps: "Well, when you look at what they feed them," they say, "it's got to be diet, hasn't it?" They choose death, through ignorance and laziness, but you choose life, fresh fish, olive oil, and that's why you're healthy. You're going to see 80. You deserve it. Not like them.

Genuine public-health interventions to address the social and lifestyle causes of disease are far less lucrative, and far less of a spectacle, than anything a lifestyle magazine editor or television commissioner would dare to touch. What prime-time TV show looks at food deserts created by giant supermarket chains, the very companies with which the stellar media nutritionists so often have their lucrative commercial contracts? What glossy magazine focuses on how social inequality drives health inequality? Where's the human interest in prohibiting the promotion of bad foods, facilitating access to healthier foods by means of taxation, or maintaining a clear labelling system?

There is no glamour in "enabling environments" that naturally promote exercise, or urban planning measures that prioritise cyclists, pedestrians and public transport over the car. There are no votes, it seems, in reducing the ever-increasing inequality between senior executive and shop-floor pay. When do you ever hear about elegant ideas like "walking school buses"? Somewhere near you, a softly spoken public-health official has probably tried to interest your local paper in a story on them; presumably the latest urgent food-fad news left no space.

We love this stuff. It isn't done to us, we invite it, and we buy it, because we want to live in a simple universe of rules with justice, easy answers and predictable consequences. We want pills to solve complex social problems like school performance. We want berries to stop us from dying and to delineate the difference between us and the lumpen peasants around us. We want nice simple stories that make sense of the world.nd if you make us think about anything else more complicated, we will open our mouths, let out a bubble or two, and float off - bored and entirely unphased - to huddle at the other end of our shiny little fishbowl eating goji berries.

· This is an edited extract from Bad Science by Ben Goldacre, published by Fourth Estate today at £12.99. To order a copy for £10.99 with free p&p, call 0870 836 0875 or visit guardianbookshop.co.uk

· Ben Goldacre discusses his new book, homeopathy degrees and science in themedia in our Science Weekly podcast theguardian.com/audio