The tobacco industry has spent millions in pursuit of its Holy Grail – the cancer-free cigarette. But even if they succeed, smokers might not like it.

There exists, in some lawyer’s vault, somewhere in the United States, a secret memo that was authored by a tobacco company executive back in 1955. Written in the exclamatory patois of the time, it says, “Boy! Wouldn’t it be wonderful if our company was first to produce a cancer-free cigarette. What we could do to the competition.”

Just imagine it: a safe cigarette. One that doesn’t kill by causing cancer, or the diseases of the heart and lungs. One that can be consumed with pleasure to the smoker, benefit to the treasury and profit to the tobacco giants. To all the interested parties, this would indeed be the most tremendous invention. Ever since the most serious perils of smoking became suspected by scientists, back in the early Fifties, many hundreds of millions of pounds have been spent pursuing the impossible dream.

The early years of the quest were notable, chiefly, for the fact that they were rarely admitted publicly. The companies feared that public acknowledgement of a link between smoking, addiction and health would cause governments to regulate, sales to collapse and ruinous lawsuits to be brought. Now, a brave new era is upon us. All of the major tobacco companies are openly pursuing safer, “reduced toxicant” products. Large teams of scientists in labs across the world are working on finding new and fascinating ways of lessening the harm of the most fatal habit of all, even if they’re not entirely sure they’ll succeed.

It’s hard to imagine that the research and development laboratories of British American Tobacco (BAT) were once a secret. When approaching it on foot, through the Southampton suburb that surrounds it, the enormous, glinting complex looms into view at the end of a narrow street. In its vast concrete spaces, great machines manufacture experimental cigarettes and strange nicotine products. In laboratories, researchers in white coats work among corridors of pristine equipment, pouring liquids into test tubes and inserting cigarettes into robotic smokers.

“Whilst we have a fiduciary responsibility to make money for our shareholders, we also need to behave responsibly,” says chief scientific officer, Dr Chris Proctor, who sits in a conference room beneath a large perspex sign that reads “we welcome your comments”. “We’ve got consumers getting sick because they use our products.”

It’s a statement that previous BAT executives wouldn’t have dreamt of making. Incredibly, tobacco companies in the US were still denying the link between smoking, addiction and disease as recently as 2000 (in Britain, they began conceding the truth in the early Nineties). Just outside the room is a portrait of Proctor’s predecessor, Sir Charles Ellis, who surely would have been startled to hear what he’s been saying.

In 1962, the tobacco industry’s nightmare became real. A study by the Royal College of Physicians concluded that smoking causes lung cancer and urged the government to take “decisive steps” to combat the problem. In a speech to his colleagues that year, Sir Charles struggled to concede the devastating science, insisting that those who saw a link had “reached an emotional conclusion in which they believed passionately and sincerely… We know only too well that there are no conclusive proofs; that there are few, if any, cold scientific facts.” But, he added, if a link was ultimately proven, scientists would surely find a way to solve the problem. Cigarettes would be made safe.

The health risks were not to be admitted publicly, however. To protect themselves, special code words were used in internal documents, such as those that were zooming about BAT’s famous Southampton labs in 1957, which used “zephyr” for lung cancer and “borstal” for carcinogens.

One early answer was the filter. When Kent introduced its new filtered cigarettes, sales increased by more than $20 billion in a year. But there was a problem. “It was made out of asbestos,” says Mitch Zeller, a former associate commissioner of America’s Food and Drug Administration. “Sadly, all the people that were sucking down on those filters were sucking down asbestos fibres as well.”

In the early Seventies, the industry came up with another clever scheme: “light” or “low-tar” cigarettes that had tiny, laser-cut holes in their paper and would, in theory, dilute the smoke with fresh air, and thus reduce its harmful toxins. Once again, there was a problem: “Filter ventilation is probably the greatest fraud that has ever been perpetrated on consumers,” Zeller says. “All of the patents for this technology said that the holes were put 12mm from the mouth end of the cigarette. Why? Because when you place that cigarette in a smoking machine, which is used to measure tar and nicotine delivery, the ventilation holes are left open and that has a dilution effect. But when a human being smokes that cigarette, 12mm is where your lips and fingers go. So they block the holes.” A 1980 study found that when these holes are blocked, the toxic load of a “low tar” or “light” cigarette can be increased by up to 300 per cent.

It wasn’t until 1988 that the industry announced its next major advance, with the launch by US firm R J Reynolds of the ingenious “Premier” cigarette. Having cost $325 million to develop, the Premier helped solve the primary source of danger to any smoker: combustion.

It’s the act of burning, according to Robert West, professor of health psychology at University College, London, that triggers the release of hundreds of toxic substances. “Carbon monoxide is a poisonous gas that is produced whenever you burn any organic compound, whether it’s petrol, paper or tobacco leaf,” he says. “When you inhale it, it gets into the bloodstream, and that has a number of effects, one of which is to increase the hardening of the arteries. That contributes to heart disease. Then you’ve got the ‘tar’, which is just the name for a very large number of chemicals, many of which are known to be highly carcinogenic.”

The revolutionary idea behind the Premier was that it didn’t burn tobacco, but heated it instead. “You still got a lot of carbon monoxide, but you didn’t get anything like the tar exposure,” says West. But, yet again, there was a problem. The taste. The primary flavours of the Premier were sulphur and charcoal. Despite the company’s scientists attempting to mask it with hints of raspberry and chocolate, it didn’t stop test subjects describing the taste as being redolent of “burning plastic” and “s---”.

What they couldn’t know, back in the Eighties, was that their mission had been impossible from the start. Today, neuroscientists know cigarettes to be addictive in several different ways. One of its mechanisms involves nicotine triggering the release of dopamine in a part of the brain called the nucleus accumbens, where it acts as a teaching signal. “It tells the brain, ‘Pay attention to what you were just doing and the situation in which you were just doing it and the next time that situation comes along, do it again,’ ” explains West. The smoker remembers the act, ritual and experience of enjoying a cigarette, and develops a powerful urge to recreate it.

One major problem with genuinely low-tar cigarettes, such as the Premier, is that the tar is where most of the flavour lives. To the smoker’s addicted brain, it is simply delicious. “Nicotine trains you to smoke cigarettes with the kind of flavour that you’re used to,” says West. “And so even switching brands for smokers can be a bit of a thing, because it tastes different.” The Premier, then, was too different an experience ever to stand a chance. It lasted just a few months before it was withdrawn.

Deep in the smoking archives hides an altogether more curious tale. Early in the Sixties, a team of scientists led by the chemist James Mold announced to their bosses at the Liggett & Myers Tobacco Company that they had developed a truly “cancer-free” cigarette. Mold, now deceased, told a reporter in 1992, “We were the only ones in the whole damned world able to solve this problem.” After studying his data, the company apparently agreed. They stockpiled ingredients for Mold’s creation (it was to be coated with the metal palladium), and developed an advertising campaign for what they’d called “Tame” cigarettes. And then, it was killed. “I think they were concerned that they’d have everybody suing them because they’d be admitting they’d been making a hazardous cigarette,” said Mold.

Did the tobacco industry really conspire to destroy what it believed to be a truly cancer-free cigarette? “It wouldn’t surprise me,” says Zeller, who tussled with US tobacco companies for seven years in his time at the FDA. “Back then, when they were denying causality and addiction, I can see why that kind of decision would be made, for commercial and legal reasons. But I wonder how good the science was that showed it reduced the risk of cancer. I wonder what kind of evidence they had.”

Back at BAT, one of Proctor’s proposed solutions comes as a surprise. Not a cigarette at all, but a form of tobacco that you put into your mouth, where the nicotine kind of oozes out. It’s called “snus” and BAT has its own manufacturing plant just downstairs from where we’re sitting.

According to West, “Snus is a special form of tobacco that’s got practically no carcinogens in it. There are some questions about whether it might have a slight increased risk of heart disease, but it’s near enough safe.” Unfortunately, though, snus is illegal in Britain. “Which doesn’t make any sense at all,” says West. “There have already been two legal challenges in the European court that didn’t go anywhere, so I don’t see it changing any time soon.”

Even if the law could be overturned, Proctor’s experience promoting snus in markets where it is permitted suggests the task may be formidable. “We thought we’d have a go at snus,” he begins, brightly. “So we bought a small company in Sweden, which is now part of the group, and started trials in Japan, South Africa and Canada. In Japan it flopped. In South Africa, we tried for three or four years to get people to adopt the habit, but it was hard. And in Canada, I flew to Ottawa in 1998 to give a press conference about snus. There was me and our press officer sat in front of all these Canadian flags. No one turned up.”

One problem with snus is the perhaps insurmountable one: the dopamine in the nucleus accumbens, which teaches people to smoke exactly what they’ve always smoked, and not suck on a tea bag full of dead leaves.

Much of BAT’s research millions have been spent on developing a more traditional cigarette, which is known internally as the “RTP”, or “reduced toxicant prototype”. Using a combination of technologies – resins that absorb toxicants; tobacco that’s been specially washed and treated, then bulked out with a papery equivalent; chalk and seaweed added to the paper – they hoped to create a brand new product. Initial testing, though, has caused BAT to temper their ambitions. They’re now hoping to introduce some of their innovations into their standard smokes.

But even if they do manage to prove that their new creations lessen exposure to various toxicants, it’s another thing entirely to show that this reduction will lead to any significant health benefit. “How do you evaluate if any of this is going to be of useful effect?” asks Proctor. “That’s the story of a lot of our research effort for the last five or six years. It’s one of the most challenging things, figuring out if what you’ve done to the product actually reduces the risk.”

Zeller makes much the same point, although rather less optimistically. “At the end of the day, you’re still burning tobacco leaves and inhaling the smoke into your lungs. These products might make the difference between falling out of a 14-storey window and a 10-storey window.”

True to the spirit of Sir Charles, Proctor will press on. “It’s hugely challenging, but we’re going to continue on this journey of trying to reduce the toxicants in smoke and seeing whether they are beneficial,” he says. And does he believe that the ultimate prize, a cigarette that is both smokable and safe, will eventually be won? “No.”

This article also appeared in SEVEN magazine, free with the Sunday Telegraph. Follow SEVEN on Twitter @TelegraphSeven