We are, on average, 3st heavier than we were in the 60s. And not because we're eating more or exercising less – we just unwittingly became sugar addicts

Up a rickety staircase at the Newarke Houses Museum in Leicester, England hangs a portrait of Britain's first obese man, painted in 1806. Daniel Lambert weighed 53st (335kg) and was considered a medical oddity. Too heavy to work, Lambert came up with an ingenious idea: he would charge people a shilling to see him. Lambert made a fortune, and his portrait shows him at the end of his life: affluent and respected – a celebrated son of Leicester.

Two hundred years on, I'm in a bariatric ambulance (an alternative term for obese, favoured by the medical world because it's less shaming to patients) investigating why the UK is in the midst of an obesity crisis. The crew pick up a dozen Daniel Lamberts every week. Fifty-three stone is nothing special, it's at the lower end of the weight spectrum, with only the 80st patients worthy of mention when a shift finishes. The specially designed ambulance carries an array of bariatric gizmos including a "spatula" to help with people who have fallen out of bed or, on a recent occasion, an obese man jammed between the two walls in his hallway. As well as the ambulance, there's a convoy of support vehicles including a winch to lift patients onto a reinforced stretcher. In extreme cases, the cost of removing a patient to hospital can be up to £100,000, as seen in the recent case of 63st teenager Georgia Davis.

But these people are not where the heartland of the obesity crisis lies. On average, in the UK, we are all – every man, woman and child – three stone heavier than we were in the mid-60s. We haven't noticed it happening, but this glacial shift has been mapped by bigger car seats, swimming cubicles, XL trousers dropped to L (L dropped to M). An elasticated nation with an ever-expanding sense of normality.

Why are we so fat? We have not become greedier as a race. We are not, contrary to popular wisdom, less active – a 12-year study, which began in 2000 at Plymouth hospital, measured children's physical activity and found it the same as 50 years ago. But something has changed: and that something is very simple. It's the food we eat. More specifically, the sheer amount of sugar in that food, sugar we're often unaware of.

The story begins in 1971. Richard Nixon was facing re-election. The Vietnam war was threatening his popularity at home, but just as big an issue with voters was the soaring cost of food. If Nixon was to survive, he needed food prices to go down, and that required getting a very powerful lobby on board – the farmers. Nixon appointed Earl Butz, an academic from the farming heartland of Indiana, to broker a compromise. Butz, an agriculture expert, had a radical plan that would transform the food we eat, and in doing so, the shape of the human race.

Butz pushed farmers into a new, industrial scale of production, and into farming one crop in particular: corn. US cattle were fattened by the immense increases in corn production. Burgers became bigger. Fries, fried in corn oil, became fattier. Corn became the engine for the massive surge in the quantities of cheaper food being supplied to American supermarkets: everything from cereals, to biscuits and flour found new uses for corn. As a result of Butz's free-market reforms, American farmers, almost overnight, went from parochial small-holders to multimillionaire businessmen with a global market. One Indiana farmer believes that America could have won the cold war by simply starving the Russians of corn. But instead they chose to make money.

By the mid-70s, there was a surplus of corn. Butz flew to Japan to look into a scientific innovation that would change everything: the mass development of high fructose corn syrup (HFCS), or glucose-fructose syrup as it's often referred to in the UK, a highly sweet, gloppy syrup, produced from surplus corn, that was also incredibly cheap. HFCS had been discovered in the 50s, but it was only in the 70s that a process had been found to harness it for mass production. HFCS was soon pumped into every conceivable food: pizzas, coleslaw, meat. It provided that "just baked" sheen on bread and cakes, made everything sweeter, and extended shelf life from days to years. A silent revolution of the amount of sugar that was going into our bodies was taking place. In Britain, the food on our plates became pure science – each processed milligram tweaked and sweetened for maximum palatability. And the general public were clueless that these changes were taking place.

There was one product in particular that it had a dramatic effect on – soft drinks. Hank Cardello, the former head of marketing at Coca-Cola, tells me that in 1984, Coke in the US swapped from sugar to HFCS (In the UK, it continued to use sugar). As a market leader, Coke's decision sent a message of endorsement to the rest of the industry, which quickly followed suit. There was "no downside" to HFCS, Cardello says. It was two-thirds the price of sugar, and even the risk of messing with the taste was a risk worth taking when you looked at the margin, especially as there were no apparent health risks. At that time, "obesity wasn't even on the radar" says Cardello.

But another health issue was on the radar: heart disease, and in the mid-70s, a fierce debate was raging behind the closed doors of academia over what was causing it. An American nutritionist called Ancel Keys blamed fat, while a British researcher at the University of London Professor John Yudkin, blamed sugar. But Yudkin's work was rubbished by what many believe, including Professor Robert Lustig, one of the world's leading endocrinologists, was a concerted campaign to discredit Yudkin. Much of the criticism came from fellow academics, whose research was aligning far more closely with the direction the food industry was intending to take. Yudkin's colleague at the time, Dr Richard Bruckdorfer at UCL says: "There was a huge lobby from [the food] industry, particularly from the sugar industry, and Yudkin complained bitterly that they were subverting some of his ideas." Yudkin was, Lustig says simply, "thrown under the bus", because there was a huge financial gain to be made by fingering fat, not sugar, as the culprit of heart disease.

The food industry had its eyes on the creation of a new genre of food, something they knew the public would embrace with huge enthusiasm, believing it to be better for their health – "low fat". It promised an immense business opportunity forged from the potential disaster of heart disease. But, says Lustig, there was a problem. "When you take the fat out of a recipe, food tastes like cardboard, and you need to replace it with something – that something being sugar."

Overnight, new products arrived on the shelves that seemed too good to be true. Low-fat yoghurts, spreads, even desserts and biscuits. All with the fat taken out, and replaced with sugar. Britain was one of the most enthusiastic adopters of what food writer Gary Taubes, author of Why We Get Fat, calls "the low-fat dogma", with sales rocketing.

By the mid-80s, health experts such as Professor Philip James, a world-renowned British scientist who was one of the first to identify obesity as an issue, were noticing that people were getting fatter and no one could explain why. The food industry was keen to point out that individuals must be responsible for their own calorie consumption, but even those who exercised and ate low-fat products were gaining weight. In 1966 the proportion of people with a BMI of over 30 (classified as obese) was just 1.2% for men and 1.8% for women. By 1989 the figures had risen to 10.6% for men and 14.0% for women. And no one was joining the dots between HFCS and fat.

Moreover, there was something else going on. The more sugar we ate, the more we wanted, and the hungrier we became. At New York University, Professor Anthony Sclafani, a nutritionist studying appetite and weight gain, noticed something strange about his lab rats. When they ate rat food, they put on weight normally. But when they ate processed food from a supermarket, they ballooned in a matter of days. Their appetite for sugary foods was insatiable: they just carried on eating.

According to Professor Jean-Marc Schwarz of San Francisco hospital, who is currently studying the precise way in which the major organs of the body metabolise sugar, this momentum creates "a tsunami" of sugar. The effect this has on different organs in the body is only now being understood by scientists. Around the liver, it coalesces as fat, leading to diseases such as type-2 diabetes. Other studies have found that sugar may even coat semen and result in obese men becoming less fertile. One researcher told me that, ultimately, perhaps nothing needs to be done about obesity, as obese people will wipe themselves out.

The organ of most interest, however, is the gut. According to Schwarz and Sclafani, the gut is a highly complex nervous system. It is the body's "second brain", and this second brain becomes conditioned to wanting more sugar, sending messages back to the brain that are impossible to fight.

The Sugar Association is keen to point out that sugar intake alone "is not linked to any lifestyle disease". But evidence to the contrary appears to be emerging. In February, Lustig, Laura Schmidt and Claire Brindis of the University of California wrote an opinion article for the journal Nature citing the growing body of scientific evidence showing that fructose can trigger processes that lead to liver toxicity and a host of other chronic diseases, and in March, the New York Times reported a study that had been published in the journal Circulation, which found that men who drank sweetened beverages most often were 20% more likely to have had a heart attack than those who drank the least. David Kessler, the former head of the US government's most powerful food agency, the FDA, and the person responsible for introducing warnings on cigarette packets in the early 90s, believes that sugar, through its metabolisation by the gut and hence the brain, is extremely addictive, just like cigarettes or alcohol. He believes that sugar is hedonic – eating it is "highly pleasurable. It gives you this momentary bliss. When you're eating food that is highly hedonic, it sort of takes over your brain."

In London, Dr Tony Goldstone is mapping out the specific parts of the brain that are stimulated by this process. According to Goldstone, one of the by-products of obesity is that a hormone called leptin ceases to work properly. Normally, leptin is produced by the body to tell you that you are full. However, in obese people, it becomes severely depleted, and it is thought that a high intake of sugar is a key reason. When the leptin doesn't work, your body simply doesn't realise you should stop eating.

Leptin raises a big question: did the food industry knowingly create foods that were addictive, that would make you feel as though you were never satisfied and always wanted more? Kessler is cautious in his response: "Did they understand the neuroscience? No. But they learned experientially what worked." This is highly controversial. If it could be proved that at that some point the food industry became aware of the long-term, detrimental effects their products were having on the public, and continued to develop and sell them, the scandal would rival that of what happened to the tobacco industry.

The food industry's defence has always been that the science doesn't prove its culpability. Susan Neely, president of the American Beverage Association, a lobby group for the soft-drinks industry, says: "there's a lot of work to try to establish causality, and I don't know that I've seen any study that does that." But it looks as though things might be changing. According to Professor Kelly Brownell at Yale University, one of the world's foremost experts on obesity and its causes, the science will soon be irrefutable and we may then be just a few years away from the first successful lawsuit.

The relationship between the food industry and the scientists conducting research into obesity is also complicated by the issue of funding. There is not a great deal of money set aside for this work and so the food industry has become a vital source of income. But this means that the very same science going into combating obesity could also be used to hone the products that are making us obese. Many of the scientists I spoke to are wary about going on the record because they fear their funding will be taken away if they speak out.

The relationship between government and the food industry is also far from straightforward. Health secretary Andrew Lansley worked, until 2009, as a non-executive director of Profero, a marketing agency whose clients have included Pizza Hut, Mars and PepsiCo. In opposition, Lansley asked public health expert Professor Simon Capewell to contribute to future policy on obesity. Capewell was amazed at the degree to which the food industry was also being consulted: the equivalent, he says, "of putting Dracula in charge of the blood bank". Lansley has made no secret of his work for Profero, and denies a conflict of interest, saying that he did not work directly with the company's clients. And the government argues, not unreasonably, that it's essential to have the industry on board to get anything done. But the relationships are not always kept at arms length. Professor James was part of a WHO committee to recommend global limits on sugar in 1990. As the report was being drafted, something extraordinary happened: the US secretary of state for health Tommy Thompson flew to Geneva to lobby on behalf of the sugar industry. "Those recommendations were never made," says James.

In New York, Mayor Bloomberg is currently planning to reduce soft drink super-sizing while last week, a former executive at Coca-Cola Todd Putman spoke publicly about the need for soft drink companies to move their focus to "healthy products". But it's not going to be easy to bring about change. A previous attempt to bring in a soda tax was stopped by intense lobbying on Capitol Hill. The soft-drinks industry paid for a new ward at Philadelphia Children's Hospital, and the tax went away. It was a children's obesity ward.

Why has Kessler, when he has had such success with his warnings on cigarette packets, not done the same thing for processed foods high in sugar? Because, he tells me, when the warnings came in on cigarettes, the game was already up in the west for the tobacco industry. Their new markets were the far east, India and China. It was no concession at all. The food industry is a different matter. For one thing, the food lobby is more powerful than the tobacco lobby. The industry is tied into a complex matrix of other interests: drugs, chemicals, even dieting products. The panoply of satellite industries that make money from obesity means the food industry's relationship to obesity is an incredibly complex one.

Anne Milton, the minister for public health, tells me that legislation against the food industry isn't being ruled out, because of the escalating costs to the NHS. Previous governments have always taken the route of partnership. Why? Because the food industry provides hundreds of thousands of jobs and billions in revenue. It is immensely powerful, and any politician who takes it on does so at their peril. "Let's get one thing straight," Milton tells me, however. "I am not scared of the food industry."

And I believe her, because now, there is something far bigger to be frightened of. Eventually, the point will be reached when the cost to the NHS of obesity, which is now £5bn a year, outweighs the revenue from the UK snacks and confectionery market, which is currently approximately £8bn a year. Then the solution to obesity will become very simple.

• The Men Who Made Us Fat, 9pm, Thursday, BBC2.