If we search for the roots of climate denial it soon becomes apparent that they lie in the reaction of American conservatism to the fall of the Berlin Wall in 1989 and the collapse of the Soviet Union in 1991. As the threat of the 'red menace' receded, the energy conservatives had put into opposing communism sought other outlets. Islamism had for some time been building as a threat, as it seemed to challenge the achievements of the West and the inevitable march of its influence. But there was an internal enemy too. Since the 1970s 'neo-conservatism' had set itself against the influence of the 'new class' of liberal intellectuals who had betrayed the Western tradition with a sustained critique of its assumptions and achievements. Feminism, multi-culturalism and anti-colonialism not only sought to correct injustices, but uncovered oppressive structures buried deep within the foundations of Western civilisation.

Environmentalism posed a particular threat because it called into question the benign nature of the system not from the perspective of an oppressed group but from the perspective of science, the very basis of Western civilisation. In the emergence of the 'green scare' the Rio Earth Summit in 1992 was a critical moment, one that brought to a head three decades of rising environmental concerns around the world. Attended by 108 heads of state or government, it put environmentalism at the centre of global action and, among other important agreements, adopted the Framework Convention on Climate Change which to this day provides the architecture for international climate negotiations. The Earth Summit not only highlighted the growing body of science that identified environmental decline but signalled a marked shift in values.

President George Bush senior was well aware of the political dangers of the Rio Summit and instructed the US delegation to water down or block most diplomatic initiatives, including the Framework Convention. Bush and fellow conservatives recognised that after the Cold War a new threat to their worldview had emerged.

From the outset, environmentalism was seen as a threat to US national sovereignty. Before Rio, a senior Bush Administration official had expressed it this way: 'Americans did not fight and win the wars of the twentieth century to make the world safe for green vegetables.' This nationalistic framing of the issue has had a powerful and enduring impact in the United States. In his 2008 presidential campaign Barack Obama's climate policy emphasised greater energy efficiency in order to free the United States from the influence of 'foreign oil'. In defending the expansion of nuclear power, climate sceptic Frederick Seitz had put it more bluntly: 'We have more control over the cost of nuclear power. The Muslims can raise the price of oil to any level they want.' Among the 17,000 people attending the Rio Summit was Dixie Lee Ray, an influential conservative activist. A marine biologist with a doctorate from Stanford on the nervous system of a type of lanternfish, in 1973 Ray was appointed by President Nixon to chair the Atomic Energy Commission and was subsequently elected governor of Washington State. She co-authored Environmental Overkill, a 1994 book critical of environmentalism, and was closely associated with the heads of two right-wing think tanks, the Heritage Foundation and the Competitive Enterprise Institute, both active in denying climate science. At Rio Ray expressed alarm because the summit was sponsored by UN officials who, she said, were members of the 'International Socialist Party'. She saw the summit's Agenda 21 as designed to impose 'world government under the UN, [so] that essentially all governments give up their sovereignty, and that nations will be, as they said quite openly, frightened or coerced into doing that by threats of environmental damage'.

Ray was expressing one of the deepest fears of US conservatives, but their anxiety over national sovereignty was matched by the disquiet they felt at environmentalism's destabilisation of the idea of progress and mastery of nature. For conservatives, these beliefs define modernity itself. Their refrain that environmentalists want to take us back to living in caves reflects not just an inability to imagine a third path other than affluence and poverty, but their unquestioned identification of progress with unfettered growth. Any challenge to growth could only mean the end of progress, of civilisation and of the American way of life.

Yet within the collection of core ideas that defined conservative belief a contradiction had emerged. Science itself seemed to be saying that continued human advancement was inconsistent with endless growth and the desire to master the natural world.

The easiest resolution of the cognitive dissonance this generated was to reject the science that causes the discomfort. For some, the creationists, this was not difficult as a prior decision had been made to accept science conditional on its consistency with deeper beliefs. For more sophisticated conservatives, those who led the movement and privileged science over biblical literalism, the solution was not to reject science per se but to reinterpret some scientific practice with the claim that its objectivity had been corrupted by biases introduced by scientists themselves, those who had become infected by the values of liberalism that spread in the 1960s and 1970s.

These sentiments help explain why a handful of scientists with genuine climate science credentials broke from the bulk of their colleagues and joined the anti-environment movement in the 1980s. Myanna Lahsen has studied in detail the life experiences and beliefs of three prominent physicists who have participated in the conservative backlash against climate science. In the post-war decades Frederick Seitz, Robert Jastrow and William Nierenberg were physicists at the pinnacle of the profession, where they enjoyed the respect of society and the patronage of governments who understood scientific endeavour as the source of national power and prestige. Part of the nuclear science establishment with links to the defence effort, their influence reached a peak in the 1970s, just as the environment and peace movements began to challenge the benefits of nuclear technology and the undue power of the 'military–industrial complex'.

The social benefits of science and technology were no longer accepted uncritically and these challenges found expression in political demands for independent evaluation of science and technology. The scientific power and privilege of the elite went into decline. Lahsen reports that Seitz himself wrote of his depression over the new political environment and its attacks on the modernist program of progress through technological advance. 'Their discourses generally', writes Lahsen, 'reveal a pre-reflexive modernist ethos characterized by strong trust in science and technology as providers of solutions to problems ... an understanding of science and progress that prevailed during the first half of the twentieth century'. They do not see nature as fragile and they believe in the right of humans to use technology to exert mastery, and it is in respect to this supreme ability of humans that elite scientists such as themselves have a unique entitlement to shape opinion. They express outrage at those who challenge this view of science and progress, experiencing it as a personal affront, as a 'sense of violated entitlement'. Their intolerance of scientific ignorance can perhaps be forgiven, but how do they respond to those better informed? When asked why most scientists reject his sceptical views on global warming, Seitz (who has been president of the US National Academy of Sciences) opined: 'Most scientists are Democrats ... I think it's as simple as that.'

Among the characteristics of elite physicists like the trio is an intellectual arrogance that leads them to believe, as one close observer put it, that global environmental problems are 'trivia that can be handled by a good physicist on a Friday afternoon over a beer'. Being the stars of the sciences, with a rigour others want to emulate, gives them a sense of intellectual superiority and permission to be contrarian.

In 1984 Seitz, Jastrow and Nierenberg founded the George C Marshall Institute, a Washington think tank initially devoted to defending President Ronald Reagan's embattled Strategic Defense Initiative, or 'Star Wars' program, panned by most experts as unworkable and a massive waste. Although still campaigning on missile defence, in the 1990s the Marshall Institute's foremost activity became attacking climate change science. It is no surprise to find that Exxon began providing funding. Claiming its purpose is to counter the politicisation of science 'by providing policymakers with rigorous, clearly written and unbiased technical analyses', every paper on the subject of climate science it publishes or links to on its website aims to debunk the science.

The way in which conservative think tanks amplified the message of sympathetic scientists is well documented; given their status as a very small minority, it is a sign of their effectiveness that, in appearances before congressional hearings on climate change, representatives of conservative think tanks achieved virtual parity with scientists representing the consensus view.

Climate scepticism grew directly out of the conservative counter-movement against environmentalism. Its first task was to erode confidence in the science on which environmental concerns were based by arguing that the scientists had become politicised and were using their research, or allowing it to be used, to advance an anti-corporate political agenda. An analogy is sometimes drawn between those who have resisted the tide of scientific evidence on the dangers of climate change and those who once questioned the link between smoking and lung cancer in the face of overwhelming medical evidence. It turns out that the links between denialists in the climate change and smoking controversies go much deeper than mere analogy. In response to the 1992 report of the US Environmental Protection Agency linking passive smoking with cancer, Philip Morris hired a public relations company named APCO to develop a counter-strategy. Acknowledging that the views of tobacco companies lacked credibility, APCO proposed a strategy of 'astroturfing', the formation and funding of apparently independent front groups to give the impression of a popular movement opposed to 'overregulation' and in support of individual freedom. Foremost among the fake citizens' groups was The Advancement of Sound Science Coalition (TASSC). According to secret documents uncovered in a court case, and reported by George Monbiot, it was to be 'a national coalition intended to educate the media, public officials and the public about the dangers of 'junk science' ... Upon formation of Coalition, key leaders will begin media outreach, e.g. editorial board tours, opinion articles, and brief elected officials in selected states.'

The strategy was to link concerns about passive smoking with a range of other popular anxieties, including global warming, nuclear waste disposal and biotechnology, in order to suggest that these were all part of an unjustified social panic, so that calls for government intervention in people's lives were unwarranted. It set out to cast doubt on the science, to link the scare against smoking with other 'unfounded fears' and to contrast the 'junk science' of their opponents with the 'sound science' they promoted. As one tobacco-company memo noted: 'Doubt is our product since it is the best means of competing with the "body of fact" that exists in the mind of the general public. It is also the means of establishing a controversy.'

As the 1990s progressed and the rear-guard action against restrictions on smoking faded, The Advancement of Sound Science Coalition started receiving funds from Exxon (among other oil companies) and its 'junk science' website began to carry material attacking climate change science. Monbiot wrote that this website 'has been the main entrepôt for almost every kind of climate-change denial that has found its way into the mainstream press'. Having been set up by Philip Morris, TASSC 'was the first and most important of the corporate-funded organisations denying that climate change is taking place. It has done more damage to the campaign to halt it than any other body.' So the tactics, personnel and organisations mobilised to serve the interests of the tobacco lobby in the 1980s were seamlessly transferred to serve the interests of the fossil fuel lobby in the 1990s. Frederick Seitz had in the 1980s served as principal scientific adviser for cigarette-maker R.J. Reynolds, from which position he challenged the link between tobacco smoke and cancer. The task of the climate sceptics in the think tanks and PR companies hired by fossil fuel companies was to engage in 'consciousness lowering activities', to 'de-problematise' global warming by characterising it as a form of politically driven panicmongering.

As a result, climate denial and political conservatism have become, at least in the United States, entwined. Although some evangelical churches now encourage action to avert global warming as an expression of good stewardship of God's Earth, climate scepticism has become part of the worldview of some Christian fundamentalists. This stew of paranoia finds expression in figures such as Republican Congresswoman Michele Bachmann, who attacked House Speaker Nancy Pelosi for her 'global warming fanaticism ... She has said that she's just trying to save the planet. We all know that someone did that over 2,000 years ago.'

• This is an edited extract from Requiem for a Species by Clive Hamilton, published by Earthscan