Gone With the Smoke is already no more than a distant whiff of bubblegum-flavoured vapour. The vape shop and lounge, one of many in San Francisco, has been forced to close. So have Vapor Den (“eclectic lounge & hipster go-to”) and Happy Vape. From late January, it became illegal to sell e-cigarettes and e-liquids in San Francisco. Even online sales to addresses within the city limits have been stopped. Stores outside the city that dispatch e-cigarettes to an SF postcode will face prosecution.

While vaping is banned, sales of legal marijuana and tobacco will continue as usual. San Francisco has often been considered more progressive than the rest of the US in its approach to drugs and unorthodox lifestyles: marijuana was legalised in California for medical use in 1996, after a campaign by Aids activists from the city, and for recreational use in 2016. Vaping, on the other hand, has crossed a line.

Behind the outright ban on sales of e-cigarettes in San Francisco is a panic about teenagers vaping. More than one in four American teens have tried vaping, according to the US Centers for Disease Control and Prevention. A study published in the New England Journal of Medicine reported that 12% of 16- and 17-year-olds were addicted to nicotine, and raised the alarm about the effect of nicotine on the adolescent brain. The Food and Drug Administration (FDA) recently described the use of e-cigarettes as a “crisis among America’s youth”.

“San Francisco has never been afraid to lead. That will always be the case when the health of our children is on the line,” announced the San Francisco city attorney Dennis Herrera in a statement after the ban was passed by the the city legislature last June. He was scathing about the FDA’s failure to control e-cigarette sales. “Now, youth vaping is an epidemic. If the federal government is not going to act to protect our kids, San Francisco will,” he said.

In the UK, meanwhile, the medical establishment is endorsing vaping as an aid to giving up smoking. My local vape shop in London is colourful, thriving, offers a panoply of flavours and displays a banner the length of its storefront proclaiming: “Vaping is 97% safer than smoking, according to NHS and Cancer Research UK.” (The correct figure is 95%, according to a report in August 2015 by Public Health England, PHE, the government executive agency and watchdog that offers guidelines on health protection issues.)

A transatlantic schism has opened up over vaping and health. In the US, the war on vaping is being pursued by activists, politicians and scientists who believe that tobacco companies are cynically promoting e-cigarettes as a means to get people addicted to nicotine, which will – sooner or later – lead them to cigarettes. In the UK, anti-smoking campaigners and health experts counter that for many adult smokers, vaping offers the best hope of avoiding a premature death.

The two sides periodically break into open hostilities. The claim by PHE that vaping is 95% safer than smoking tobacco, frequently quoted by e-cigarette manufacturers and sellers, has been criticised as misleading by anti-smoking campaigners in the US. Matt Myers, who heads the Campaign for Tobacco-Free Kids in Washington DC, the biggest anti-smoking organisation in the world, has called the 95%-safer figure “mere fiction”.

Prof Ann McNeill of King’s College London, a tobacco and addiction expert who advises PHE, defends its position. “We are battling against misinformation on a massive scale,” she says. McNeill acknowledges there has been a rise in vaping among kids in the US and Canada, but does not see it as a reason for panic. “I don’t think it merits discussion of an ‘epidemic’. That word is overblown,” she said.

The soaring popularity of vaping among the young in the US is largely down to Juul – a tiny black or chrome device that looks like a USB stick and fits into the palm of the hand. In 2004, two design graduate students came up with the idea for an electronic alternative to smoking. They launched Juul in 2015, which quickly defined the market; by July 2019, Juul accounted for 75% of US e-cigarette sales. Since then, its fortunes have taken a dive. The company is accused, in dozens of lawsuits from San Diego to New York City, of targeting young users via social media campaigns featuring youthful models. Opponents claim Juul pods are easier for novice vapers to inhale, since they contain nicotine salts instead of straight nicotine, further softened with teen-friendly flavourings such as mango, cool cucumber and creme brulee. Juul has repeatedly denied it has marketed to teens.

There was fresh alarm in the US last year when 2,500 cases of lung disease and 55 deaths were associated with vaping. E-cigarettes work by heating liquid containing nicotine to produce vapour, which is then inhaled. There is no smoke or tar involved, but there are small amounts of chemical flavourings, including diacetyl, which has been linked to lung disease, and propylene glycol or vegetable glycerin. If the e-liquid overheats, formaldehyde can be formed. In fact, none of these chemicals were to blame in these cases – it turned out that the people who fell ill were using bootleg devices containing cannabis – but the reputational damage was done.

In the wake of this alarming spate of hospitalisations and deaths, the New York state governor, Andrew Cuomo, took emergency action in September, attempting to ban flavoured e-cigarette products. President Trump entered the fray, telling the FDA to act and imposing a temporary ban on any flavours that might appeal to young people. In December, New York City mayor, Bill de Blasio, signed a law banning flavours. “Manufacturers of fruit and candy-flavoured e-cigarettes are intentionally and recklessly targeting young people,” said Cuomo, citing flavours including bubblegum, cotton candy and Captain Crunch. The state ban was overturned in January 2020 by Justice Catherine Cholakis, who said it was an overreach.

In the UK, the NHS continues to edge as close as it can to approving the use of e-cigarettes to quit smoking. NHS hospitals in the West Midlands have sanctioned vape stores on the premises, while PHE launched, via YouTube, a video showing two white-coated experts with bell jars demonstrating how e-cigarettes are free of all the disgusting and damaging tar in a conventional cigarette.

The science is furiously disputed. Academics on both sides are accused of cherry-picking data to suit their own prejudices. PHE is a global authority on health issues such as vaccination and obesity, but on vaping, it is looking increasingly isolated. Deborah Arnott, head of Action on Smoking and Health (Ash) in the UK, says that Britain is losing ground in the e-cigarette debate because of the virulent campaign in the US against vaping. “The noise is causing problems in how we are perceived,” Arnott said. “We’re being written off.”

What should be settled by science has become a clash of faiths. In the US, the debate is dominated by those who believe people should “just say no” to drugs. In the UK, there is more support for the idea of “harm reduction”, in which addicts take controlled amounts of their drug, be it alcohol, heroin or in this case, nicotine, to keep them stable.

PHE’s support for e-cigarettes as a tool to help people quit smoking is shared by other respected health bodies, such as the Royal College of Physicians and Cancer Research UK. They point out that the UK regulates smoking and vaping far more rigorously than the US. The UK has rules on age, and health warnings, and caps on the nicotine content. Marketing to young people is forbidden – e-cigarettes cannot be advertised on TV. There is less nicotine in Juul pods or e-cigarette cartridges sold in the UK: Juul in the US contains up to 59mg per ml, while nicotine levels in e-cigarettes across Europe are capped at 20mg per ml by an EU directive enshrined in British law. In the US, at this point, there is no middle ground between unrestricted sales and an outright ban.

Facebook Twitter Pinterest A still from PHE’s video demonstrating how vapes are free from the tar and damaging substances found in cigarettes. Photograph: PHE

All sides agree that vaping nicotine is safer than getting it from cigarettes. Nicotine by itself is “relatively harmless”, according to the NHS, while the harm from cigarettes is in the smoke produced by burning tobacco and the residue of tar it leaves, which damages the airways, causing lung disease and cancer.

“People smoke for nicotine but they die from the tar,” wrote Michael Russell, a professor in addiction at the Institute of Psychiatry in London and a pioneer of harm reduction, in 1976. His work laid the foundations for the introduction of nicotine replacement therapy – the nicotine patches and gum the NHS hands out today.

Russell, who died in 2009, wanted to develop a low-tar cigarette that would be high in nicotine, to give smokers the hit they wanted without inhaling more deeply. His research on the low tar product was funded by the tobacco company RJ Reynolds, now owned by British American Tobacco (BAT). This relationship with a tobacco company, which Russell later claimed was normal practice at the time, is now seen as fatally compromising. It has since been used to undermine his research and attack his reputation.

Ann McNeill, who worked with Russell as a young researcher, believes he was ahead of his time. “His pioneering research improved the quality of life of smokers and saved the lives of many more,” she said in a paper celebrating Russell, co-written with Debbie Robson of the UK Centre for Tobacco & Alcohol Studies. “It is a sad indictment of our community that his work is still not recognised adequately by all those working in tobacco control, some of whom still fail to recognise the centrality of nicotine in tobacco use and the implications of this.”

Anti-tobacco activists – whether they are for or against e-cigarettes – believe they are defending the gains they have made in the battle against smoking. Vaping’s defenders say the tobacco firms are diversifying into a product that won’t kill their customer base. Hardline opponents are convinced e-cigarette sales will keep the hated tobacco companies in business.

The major tobacco companies – Philip Morris, Imperial, British American Tobacco and Japan Tobacco – did not take long to realise the potential of e-cigarettes, and all are now players in the vaping business. Blu, launched in the US in 2009 by an Australian entrepreneur, was bought by Lorillard Tobacco, and later acquired by the British company Imperial. In 2013, BAT launched Vype. In 2015, RJ Reynolds, makers of Camel and Lucky Strike, produced Vuse, which was the most popular brand in the US before Juul came along. Altria, the parent company of Philip Morris USA, acquired a 35% stake in Juul.

Most scientists and health campaigners in the US will have no dealings with the tobacco industry, because of its history of devious marketing practices and underhand tactics. Under the terms of a World Health Organization (WHO) treaty in the early 00s, governments agree not to have any discussions with tobacco industry representatives (over trade terms, for example, taxation, regulation or investment). But anti-smoking campaigners now fear that, by promoting the benefit of their e-cigarettes as an alternative to smoking, tobacco companies are acquiring respectability. WHO shares that anxiety and has advised measures to control e-cigarettes. Many countries, including Brazil, Thailand, Singapore, the Seychelles and Uruguay, have banned e-cigarettes as a result, while others have imposed regulations limiting their use.

A number of health professionals and academics have dedicated their careers to exposing lies about the safety of tobacco products and stopping the promotion of cigarettes around the world. Recent hard-won victories include smoking bans in public spaces and plain packaging with severe health warnings. Despite their efforts, there are still over 1 billion smokers in the world. The global cigarette market was worth $888bn (£682bn) in 2018 and forecast to rise to $1,124bn by 2024.

Stanton Glantz, professor of medicine at the Center for Tobacco Control, Research and Education at the University of California San Francisco, is the loudest of the anti-tobacco lobbyists – in his choice of Hawaiian shirts as well as his pronouncements. Glantz claims he was agnostic when e-cigarettes first appeared. He isn’t now. In December, he tweeted: “Using e-cigs increases exposure to toxic chemicals for most users; they would be better off just smoking.”

This was a new extreme, even for Glantz. Alex Berezow, vice-president of scientific affairs at the American Council on Science and Health, described the tweet as “mind-boggling”. “Unfortunately, Dr Glantz has become something of an ideologue. His (justifiable) animosity toward the tobacco industry has been turned (unjustifiably) to other industries, such as vaping,” he wrote on his blog. The research paper that had prompted Glantz’s tweet, Berezow pointed out, actually shows that e-cigarette users get less exposure to toxic chemicals than tobacco smokers – not more.

Glantz, who you have to interrupt if you want to ask a question, told me that in the tweet, he was talking about dual-users – people who are both smoking and vaping. “Maybe it was worded inarticulately,” he conceded. But he won’t back down. He claims the evidence suggests that most people are dual users (in the UK, about a third of vapers are still smoking as well, according to a survey by YouGov).

A giant in the anti-tobacco lobby, Glantz does not understand how researchers he respects can support vaping. Glantz claims that confidence in e-cigarettes, at PHE and among the UK scientists who condone it, is starting to crack. He is convinced the “95% safer” figure is wrong. It came from a paper published in 2014 by a group of experts led by David Nutt – the former government drugs adviser famous in the UK for declaring that ecstasy and LSD were safer than alcohol, which led to his sacking.

“The Nutt paper had no evidence whatsoever. It was 12 guys who sat around and pulled that number out of the air,” said Glantz. “The most generous thing you can say about that paper is that it was much earlier in the process and there wasn’t a lot of evidence out there.” He believes the credibility of Nutt’s group has been undermined by revelations that they were part-funded by a consultancy called EuroSwiss Health, run by Delon Human, a South African doctor who has accepted funding from BAT for some of his ventures.

Nutt says that’s nonsense. The group comprised 12 world experts. “Has [Glantz] ever read the paper?” he said. “There are 14 variables in that paper [possible harms, such as death from cancer]. It looks at the effect of 12 different forms of nicotine on 14 variables. And I bet he wouldn’t actually disagree with any of them.” He gives an example. “Does he actually think that tobacco is not much more harmful than vaping on the likelihood of lung cancer?” The paper, he said, “comes up with an answer he doesn’t want. That’s why he thinks it’s bad science.”

Get the Guardian’s award-winning long reads sent direct to you every Saturday morning

Nutt, a professor in neuropsychopharmacology at Imperial College London, says he is “saddened” by Glantz’s attacks. “He was a hero of mine. He was one of the pioneers in demolishing the myth that tobacco wasn’t addictive and opposing the fraud and misinformation and lies of the tobacco industry. But the problem is he is still basically playing the same tune and we’re now in a different era.” It’s proven impossible to stop people selling tobacco, Nutt said. “So the anti-tobacco people have got to attack something else, because that’s what they do – they attack and they ban. So basically they’ve fixed their wagons against vaping because it is one thing they can ban, and they’re very successful. It’s laughable that in India people go to prison for selling vaping when the government allows advertising of tobacco.”

Glantz became an icon of the anti-tobacco movement after he received 4,000 leaked documents from Brown and Williamson, then the US’s third-biggest tobacco company, in 1994. They proved the industry knew that smoking caused cancer and had hidden it. Since then, Glantz has always objected vigorously to any compromise with the industry. In 1997, a deal was broached with the tobacco industry by Matt Myers of the Campaign for Tobacco-Free Kids. It would have brought in tight federal regulatory control of cigarettes, prohibiting the sort of advertising and marketing that is still ubiquitous in the US, as well as sales to children. But Glantz was opposed to any deal that would allow cigarette manufacturers to continue in business. His goal was to close them down altogether. Myers was stymied by his own allies, and the US still does not have the anti-tobacco regulations that are common in Europe.

Myers is calm and quietly authoritative. He is not against harm reduction, he says. Nor are other public health bodies such as the American Cancer Society or the American Heart Association. “All of us have said that under appropriate circumstances, if e-cigarettes are shown to actually significantly help smokers quit or switch completely, and that there are rules in place to prevent them being marketed in a way that doesn’t unduly impact youth, we would be supportive,” he said.

Up until now, on a national level, the US has had no regulatory control of e-cigarette sales, marketing, minimum age, or limits on nicotine content. “It’s the wild west,” said Myers. Tobacco-Free Kids has brought legal action against the FDA over its failure to regulate e-cigarette use and last year won a ruling from a federal judge that there was no excuse for further delay. The e-cigarette companies are irresponsible too, Myers added.

Myers says both sides in the argument interpret scientific studies according to their prior beliefs. He describes PHE’s “95% safer” figure as worthless, because not enough research has been done. “I have very little doubt that e-cigarettes under appropriate circumstances are significantly less harmful to a smoker. Do we know exactly how much less harmful? The answer is no, because we have no clue how much nicotine it is delivering, how pure they are, what else they’re putting in them. Comparing it to the most lethal product ever created, for public relations purposes, is not helpful.”

In March 2018, Myers and Glantz, as well as representatives of PHE and almost every other influential anti-smoking scientist or campaigner, attended the World Conference on Tobacco or Health in Cape Town – just up the road from BAT’s South Africa HQ. At this meeting, held every two years, activists can discuss the perfidy of the industry, celebrate their successes and plan for the future.

Nobody connected with the tobacco industry is allowed in the building. The WHO’s Framework Convention for Tobacco Control, which came into force in 2005, lays down recommended anti-smoking measures for governments, from taxing cigarettes to marketing controls and smoking bans in enclosed public places. It has been signed by 168 countries (the US is a notable exception). The Framework document says governments “need to be alert to any efforts by the tobacco industry to undermine or subvert tobacco control efforts” and must limit their contact to the absolute minimum. Activists now take this to mean there must be an impenetrable wall between themselves and anyone in any way connected with tobacco.

The man who did as much as anyone to establish the Framework Convention while an executive director of WHO, Derek Yach – originally from Cape Town but now based in the US – finds himself on the wrong side of that wall. Yach was pointedly excluded from the conference in his home city – and yet much of the conversation was about him. Yach had done the unthinkable: accepting almost $1bn over 12 years from Philip Morris, makers of Marlboro and other leading brands, to set up his Foundation for a Smokefree World in New York in 2017 to fund research into alternatives to tobacco.

As part of its commitment to a “smokefree future”, Philip Morris International is heavily promoting its e-cigarettes and Iqos, a cigar-shaped electronic device designed to heat, rather than burn, tobacco. “These products will one day replace cigarettes,” it says on its website, claiming to be moving on from tobacco products because “society expects us to act responsibly”. Iqos has taken off in Japan, where 3 million people regularly use them. Smoking there was declining by 2% a year before Iqos arrived, and is now dropping by 10% per year, for which Philip Morris International claims the credit.

Yach believes it’s in the tobacco companies’ interests to develop products that will leave conventional cigarettes behind. His erstwhile colleagues believe he is working for the devil. Why, they ask, does Philip Morris still sell cigarettes if it cares about the world’s health?

As rumours circulated that Yach was in the building or lurking outside, Michael Bloomberg, the billionaire former mayor of New York and funder of vast amounts of anti-tobacco science and programmes, was launching his own campaign. Striding down the hallway, flanked by purple banners emblazoned with the words Bloomberg Philanthropies, the financial backer of the conference and his entourage made their way to the TV cameras to discuss his Stop initiative. Bloomberg announced he has set aside $20m to counter the lies of the tobacco industry. Top of the watch list was Yach’s Foundation.

Facebook Twitter Pinterest Attorney general Letitia James announces a lawsuit by the state of New York against e-cigarette maker Juul. Photograph: Lucas Jackson/Reuters

Bloomberg helps pay for WHO’s work on tobacco. I was offered a five-minute audience – just long enough to ask if he thinks there’s a role for e-cigarettes in combating smoking. “It is like marijuana: one of the stupid things we’re doing is legalising it,” he said. “I think there’s no place for e-cigarettes. I think it’s a terrible idea.”

Yach claims that “A-grade scientists and researchers around the world” are picking up grants from his foundation and doing useful work, but few believe he can succeed. He admits he was taken aback by the strength of feeling. “I wasn’t completely naive about it, but I didn’t actually appreciate how harsh it would be, particularly from a pretty small bunch of people who have incredible influence at WHO,” he says.

Clive Bates, the former head of the UK’s Ash, who blogs and campaigns for e-cigarettes from his home in Nigeria – he is married to the British High Commissioner – said he was amazed by the anti-vaping anger in Cape Town. “It was like a cult, almost,” he said. “Particularly the attacks on the foundation. It’s quite an achievement for Derek to create an institution that has a worse reputation than Philip Morris.”

However, Robert West, a professor of health psychology and director of tobacco studies at UCL, says: “It is playing out beautifully [for Philip Morris]. [The Foundation] has got the tobacco control community arguing among itself and divided. Result.”

In the UK, a small number of prominent public health academics vehemently oppose e-cigarettes. Martin McKee, professor of European public health at the London School of Hygiene and Tropical Medicine, and Simon Capewell, professor of clinical epidemiology at Liverpool, believe the tobacco companies are using e-cigarettes as a route back to respectability. Sally Davies, until recently the UK’s chief medical officer (the most senior government adviser on public health matters), backs a ban on flavours that might attract children. She’s worried that we don’t know the long-term health effects of vaping, which she described as a “ticking time bomb”.

Ann McNeill says there is not much growth in vaping among kids in the UK. Her key concern would be a rise in vaping among young people who have never smoked, but there’s no sign of that. She thinks the key to reducing youth uptake is “getting adult smoking down”, because young people tend to imitate their elders’ behaviour.

She feels she and her colleagues at PHE have been unfairly attacked for saying vaping is 95% safer than smoking tobacco. She points out that they never said it was harmless. A 5% risk of harm, she insisted, “is not an insubstantial number”. She feels that the statement has been twisted as if PHE had said vaping was completely safe.

There is evidence that smoking is going down as e-cigarette use goes up, both in the UK and in the US. In 1942, 82% of British men smoked. By 2006, when e-cigarettes first appeared, 22% of adults in England smoked. The number of smokers is now at an all-time low of 14.7%, while 7% of the UK population are vaping regularly. Bans on smoking in public places, no-logo cigarette packaging with gruesome pictures of tumours, and the removal of cigarettes from sight in shops have all had an impact, but, McNeill says: “All the evidence, I believe, points to e-cigarettes playing a role.”

The best evidence that vaping helps people stop smoking comes from a study showing that e-cigarettes double the quitting success rate compared to gum or other aids. Peter Hajek from Queen Mary University of London and colleagues carried out the trial among more than 880 people who went to the NHS for help to give up smoking. Half were given nicotine replacement therapy in whatever form they wanted, such as patches or gum. The other half were given a starter e-cigarette kit and encouraged to buy their own when it ran out. The results, published in the New England Journal of Medicine in January 2019, showed that, at a year, the quit rate in the e-cigarette group was twice that of the nicotine replacement group, 18% versus 9.9%.

Of course, it did not satisfy the critics. When the researchers went back to the subjects after a year, most of the e-cigarette group were still vaping, whereas few of the ex-smokers in the other group were still chewing gum. So, they insisted, the study showed just how addictive e-cigarettes are.

Juul, once the leading e-cigarette, is struggling against public outrage and bad press. Sales have dropped and hundreds of staff have been laid off. The company said to be worth $38bn in 2018 was written down to $24bn by Altria in October last year, and valued at just $19bn by one of its major investors, Tiger Global Management, in December.

In the storm over child users, and anticipating an FDA ban, Juul stopped selling its fruit medley and mango flavours in the US in October and even dropped mint the following month. In January, the FDA acted, banning all flavours except tobacco and menthol from e-cigarettes that use a cartridge – as the Juul devices do. These are the devices kids use, said the FDA: flavours will not be banned from the refillable e-cigarettes that are more popular with adults. The FDA is also finally getting going on regulation, and has instructed manufacturers of e-cigarettes to apply for a right to trade by May.

The WHO, taking its cue from the US and from Bloomberg, has advised countries to control vaping, warning about the unknown impact on health and stating that e-cigarettes are risky for teenage brains, as well as for the foetus. That may leave the UK isolated, a lone bastion where (highly regulated) vaping is actually encouraged in the hopes of cutting smoking rates. Many public health experts in the UK believe they are witnessing an unnecessary tragedy, and that failure to promote the most promising method of helping people quit smoking is endangering the lives of millions.

McNeill insists public health experts in the UK do care about young people. But those whose lives are at risk are adults living in disadvantaged communities, who cannot kick their smoking habit. “I have lived with smokers and watched smokers die. You want them to do anything they can to prevent them from smoking.”

• This article was amended on 19 February 2020: to clarify that Altria is the parent company of Philip Morris USA and that Iqos is a product of Philip Morris International and to correct an error in attributing a quote from the WHO’s Framework Convention for Tobacco Control.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.