It was on a Friday afternoon in May 2009 when Andrew Lansley's Public Health Commission met, as usual, in the newly restored 1930s splendour of Unilever House on Victoria Embankment in London. It was gathering for its final plenary session, having been tasked by Lansley, now health secretary but then in opposition, to come up with new policies for the Conservatives to tackle the big public health crises of obesity, diet-related disease, and alcohol abuse.

Obesity has trebled in the last 20 years, diet-related disease is estimated to cost the health service £6bn a year and rising, alcohol misuse £2.7bn a year, and the effects of lack of exercise a further £1.8bn a year. Nearly a quarter of all adults and one in seven of children in England are now obese. Alcohol deaths have doubled in the last 15 years, reflecting a doubling of alcohol consumption per capita over the last 40 years.

The commission's job was to assess Lansley's idea that a deal between business and government should form the basis of his health strategy after the election. It was about to produce its report: We're All in This Together, Improving the Long Term Health of the Nation.

In the chair of the commission, by invitation of Lansley, was Dave Lewis, UK and Ireland chairman of Unilever, one of the largest processors of industrial fats in the world.

With him were Lucy Neville-Rolfe, corporate affairs director of Tesco, the supermarket that has been a leading opponent of the traffic light food labelling scheme favoured by the Food Standards Agency, and Lady Buscombe, Conservative peer and former head of the Advertising Association, where she established herself as a formidable political champion of the ad industry's right to operate free of restrictions.

Asda's corporate affairs director, Paul Kelly, formerly PR head of Compass, the school meals company of turkey twizzler fame, had to send his apologies. Mark Leverton, policy director of Diageo, manufacturer of leading vodka, whisky and beer brands, joined them by phone.

Diageo, in fact, had closer links with the Lib Dems than the Conservatives – its corporate relations director, Ian Wright, was one of three people who paid donations directly into Nick Clegg's personal bank account to fund a researcher – but that would come in useful later once the election results were known. Bolstering the alcohol industry's presence in person was Jeremy Beadles, chief executive of its lobby group, the Wine and Spirit Trade Association.

The area of increasing physical activity to fight obesity was covered by Fred Turok, chairman of the Fitness Industry Association, the lobby group for private gyms and professional personal trainers. Public interest groups were represented at the meeting by a handful of health and consumer charities and two leading liver and alcohol specialists.

The secretariat for the Public Health Commission that day was, as usual, provided by Unilever and its marketing team. They were led by Unilever's public affairs director George Gordon, and joined by Martin le Jeune, director of the corporate PR agency Open Road (its clients include Unilever, Sky, and the alcohol industry's Portman Group). Le Jeune is former public affairs director of Sky, a former director of Fishburn Hedges PR agency (clients Diageo, Nestlé), and is a member of a group calling itself the "progressive conservatives", who are dedicated to "progress achieved by maximising liberty in both economic and social fields".

The commission's fifth meeting in Unilever House had "set the scope for progress" on the contentious issues for the industry of food and alcohol labelling, and on portion sizes.

It must have felt like a new dawn for the food and drinks industries. After more than four years of determined and co-ordinated lobbying, they were about to achieve the corporate PR agency dream: being invited to write the policy themselves. And, if the Conservatives won the election, in Lansley they would have a health secretary who understood them.

He not only subscribed to the libertarian view that public health should be more a matter of personal responsibility than government action; he bought in to the whole pro-business PR view of the world. (At that time, Lansley was a paid director of the marketing agency Profero, whose clients have included Pepsi, Mars, Pizza Hut and Diageo's Guinness. He gave up the directorship at the end of 2009.)

Partnership for change

Lansley attaches huge importance to public health, believing that too much emphasis has been put on treating illness in the NHS rather than preventing it in the first place. He talks about the ageing population and rising costs adding to the economic imperative to rebalance prevention and cure – but says change can only come through a partnership between individuals, business, charities and local and national government, and by understanding behavioural science.

By the time he outlined his vision for public health as a responsibility deal between business and government in 2008, Lansley had already adopted several of the industry's favoured approaches to the food, drink and health crises, promising that "government and FSA promotion of traffic light labelling will stop"; that there would be no mandatory extension of advertising restrictions; and that alcohol strategy would focus on the responsible drinking messages and improved labelling the industry preferred to regulation.

Lansley also committed to avoiding a narrow focus on "fear of junk foods" that might demonise individual manufacturers' products, and to talking instead in terms of diets as a whole, of the balance of energy in and energy out, and of portion size. He had said the government and the Food Standards Agency (FSA) would "highlight the continuing contribution made by business to improving diet by reformulating its products".

And so the FSA, the regulator that had caused food and drink manufacturers and retailers so much irritation, was about to have its comeuppance. Government policy on public health and nutrition would soon be taken away from it and placed back in political hands.

The work of the Public Health Commission was not only setting out a strategy for a new Conservative government, it had created a working group of industry partners who could be drafted in effortlessly to the new administration's public health structure once Lansley had become secretary of state for health.

Now, a few months in to the new coalition government and with Lansley due to publish his white paper on public health in the next few weeks, the same cast of characters has been invited in to Whitehall. The Guardian has found that they make up the bulk of new "responsibility deal" networks the health secretary has set up to be at the heart of his public health policy. They are drawing up deals between the Department of Health and business, having been asked to volunteer measures to tackle obesity, diet-related disease, alcohol abuse and lack of exercise.

In a reversal of normal government process, recommendations are not being prepared by civil servants before the meetings, but by the working groups themselves and then sent to civil servants and the wider group for comment, according to sources close to the deals. A senior corporate source welcomed this, saying it was a recognition by the new government that a lot of the best expertise lies with industry and voluntary sector groups.

In the chair of the alcohol deal with Lib Dem health minister Paul Burstow is the Wine and Spirit Trade Association head, Jeremy Beadles. The Fitness Industry Association's Fred Turok is in the chair of the physical activity deal with Conservative minister Simon Burns.

The food deal is chaired by Lansley himself and Dr Susan Jebb, the leading Medical Research Council obesity academic known for her pragmatic approach to working with industry and for endorsing the methods of Weight Watchers.

A responsibility deal network on behavioural change is co-chaired by the National Heart Forum and public health minister Anne Milton. A further deal, health at work, is being chaired by Dame Carol Black with minister Lord Howe.

The overarching board, set up and chaired by Lansley to oversee all five of these business responsibility deal networks, also includes many of the contributors to the health commission at Unilever House. As well as local government representatives, health charities and a regional health director, there are from the industry lobbies Unilever, Tesco, and Asda – plus other leading retailers, Diageo, the Wine and Spirit Trade Association, the Advertising Association, the Fitness Industry Association, Compass and Mars.

Lansley's special adviser on policy development at the Department of Health, meanwhile, is Bill Morgan, who used to work at Mandate Communications, a corporate PR agency whose clients have included health organisations and Kraft/Cadbury, Dominos Pizza, and drug companies.

Stuck on red

Looking back, senior FSA sources identify those Unilever House meetings as the "beginning of the FSA abolition movement". They could see the writing on the wall for their traffic light labelling scheme, despite the research that showed it was the most helpful one for consumers.

Nevertheless, they were startled when newspapers reported in July that Lansley was planning to abolish the FSA altogether. They were startled not least because he had no powers to do so. Labour had set up the agency in 2000 in the wake of the BSE and E coli food crises by an act of parliament as a non-ministerial government department. It could only be abolished by parliament.

The FSA's chair, former Labour minister Lord Rooker, had been told by Lansley just before the election that the agency would lose its public health and nutrition role, which he himself would take back, and that it would go to the Department for Environment, Food and Rural Affairs (Defra) if the Conservatives won.

In fact, in the early chaotic days that tend to accompany transitions to new administrations, it became clear that the Defra ministers didn't want the FSA anyway. The news of the agency's exaggerated death, which had come from Tory sources, was interpreted as a deliberate attempt to destabilise the FSA and soften the public response when it was formally announced that it would instead have its powers much reduced.

Relations between the agency as regulator and the food industry it regulated had been severely strained for some time. The FSA's successful move, with broadcast regulator Ofcom, to introduce restrictions on TV advertising of foods high in salt, fat or sugar to children had already hit food companies hard. The breakfast cereal industry was particularly affected, with the vast majority of its marketing effort falling foul of the new rules.

The FSA's proposal to introduce traffic light food labelling on processed foods that would put a red light on foods high in salt, sugar or fat was where the companies drew their line in the sand. They thought a red light would be understood by consumers as "stop", and were not prepared to negotiate on a device that would damage their sales.

Although some retailers such as Sainsbury's, Asda and Waitrose accepted the FSA's scientific research supporting the value of traffic lights and introduced a version of them, Tesco and Morrisons were adamantly opposed.

The response from manufacturers to all these public health measures was a lobbying effort of unprecedented intensity, both in the UK and in Europe, according to senior sources from the regulator and Whitehall. Tesco, Kellogg's, Unilever and Kraft led a campaign to derail the FSA's labelling scheme by launching their own rival system based on guideline daily amounts (GDAs), which FSA research showed consumers found GDAs harder to understand.

The industry also spent hundreds of millions of euros lobbying against traffic lights at European level.

"It was appalling the way manufacturers and Tesco conspired to defeat traffic light labelling despite the willingness of other retailers to give it a shot," said Richard Ayre, a member of the FSA board for seven years, adding that tensions between the FSA and the Department of Health had existed from the beginning.

By the end of their term, Labour ministers had become impatient with the FSA's independence. The Department of Health wanted to reassert political control. It decided not to confirm a second term for the FSA's chair, Deirdre Hutton, making clear to recruiters that ministers wanted someone with "Westminster experience" instead.

Rooker was appointed to take over, a move health experts believe made the agency's position more exposed when the coalition came in. By October, around 70 nutrition experts from the FSA had moved to the Department of Health, along with their files.

Public interest health experts are still trying to absorb the scale of Lansley's pro-business shakeup. They are concerned that when it comes to nutrition, the food sector is now unregulated. They are cooperating with the deals only because there appears to be no other mechanism to tackle public health problems on the table.

Professor Tim Lang, expert adviser on the government's obesity committee, explains: "What's clearly happening is that the government has dealt with some sore points for industry. It's already tamed the National Institute for Clinical Excellence, responding to the drugs industry not wanting curbs on its powers. The diet and health responsibility deals fit with the rest of the approach.

"The strong message is 'work with business'. But the idea that we can solve these huge systemic problems with slight small changes makes me very nervous. It completely misunderstands how obesity reflects a whole drift of economy and culture in the last 40-50 years."

Professor Sir Ian Gilmore, leading liver specialist, who is on the alcohol responsibility deal, shares the reservations. "I am very supportive of the secretary of state taking a position of strength on public health. But I am very concerned with the emphasis on voluntary partnerships with industry. We have to understand that their agenda is very different."