Published online 9 June 2010 | Nature 465, 682-684 (2010) | doi:10.1038/465682a

News Feature

Spending on science is one of the best ways to generate jobs and economic growth, say research advocates. But as Colin Macilwain reports, the evidence behind such claims is patchy.

President Barack Obama says it. Francis Collins, director of the US National Institutes of Health (NIH), says it. University and research leaders elsewhere are saying it, too. The number one current rationale for extra research investment is that it will generate badly needed economic growth.

"Science is more essential for our prosperity, our health, our environment and our quality of life than it has ever been before," said Obama, addressing the National Academy of Sciences in Washington DC last year. Getting down to the details, Collins has recently cited a report by Families USA, a Washington DC-based health-advocacy group, which found that every US$1 spent by the NIH typically generates $2.21 in additional economic output within 12 months.

"Biomedical research has generally been looked at for its health benefits, but the case for it generating economic growth is pretty compelling," says Collins. In Britain, senior scientists have called on the government to support science as a means of helping the economy out of recession. Heeding such arguments, governments in Germany, Sweden, Canada and Australia, as well as the United States, have increased research spending as part of stimulus packages designed to aid their struggling economies.

Beneath the rhetoric, however, there is considerable unease that the economic benefits of science spending are being oversold. The Families USA study used a model developed by the Bureau of Economic Analysis at the US Department of Commerce to deduce the likely benefits of NIH spending in each state. Collins says he has been advised that the approach is "standard and considered reliable". But some economists question the basic assumption behind such models — that a certain amount of research input will generate corresponding economic outputs — or that those outputs can be quantified.

Costs or benefits

The problem, economists say, is that the numbers attached to widely quoted economic benefits of research have been extrapolated from a small number of studies, many of which were undertaken with the explicit aim of building support for research investment, rather than being objective assessments. The economics of health research, on which much analysis of costs and benefits has been focused, "has had very little money invested in it", says Martin Buxton, director of the Health Economics Research Group at Brunel University, UK. "And too much of what has been done, has been done as a process of advocacy."

“Too much of what has been done, has been done as a process of advocacy.”



Research leaders acknowledge that they need better tools. Collins says that the NIH held a workshop with economists in May to see whether it should invest some of its funds into economic outcomes. "We're very interested in tightening up the evidence base," he says.

Some of that evidence is already being collected. Under the programme STAR METRICS (Science and Technology in America's Reinvestment — Measuring the Effects of Research on Innovation, Competitiveness and Science), implemented after the US stimulus package was introduced, the Obama administration is seeking to trace the effect of federal research grants and contracts on outcomes such as employment, publications and economic activity (see Nature 464, 488–489; 2010). The programme's supporters say it will provide justification for the stimulus money — exactly what research agencies need as they come under pressure to show what recent investments have produced.

Economic arguments have always been used to make the case for science spending, particularly when times are tough. In the United States, these were strengthened by the publication of Rising Above the Gathering Storm, an influential 2006 report from the US National Academies, which called for the sharp expansion of publicly funded research and development to stave off competition from China and elsewhere. The 600-page report, written by a panel chaired by Norman Augustine, former chairman of Lockheed Martin, was put together by a large panel of senior scientists in a matter of weeks, to meet a tight congressional deadline.

In a section titled "Why are science and technology critical to America's prosperity in the 21st century?" the report reviews the literature that estimates return on investment (ROI) from research (see Table 2). This is illustrated by various graphs, including one showing steep declines in US death rates from heart disease between 1950 and 2000, inferring that this drop can be partly attributed to biomedical research.

Innovation drive

Gathering Storm recommended that federal investment in basic research should increase by 10% every year for seven years, and led Congress to consider spending increases of that order, mainly in the physical sciences and engineering. When the newly elected President Obama was hurriedly preparing a February 2009 bill aimed at stimulating economic growth, parts of these increases were thrown in, together with extra spending for the NIH. In the end, the American Recovery and Reinvestment Act included a further $21 billion of research spending, all justified by its supporters on the grounds that it would yield speedy economic returns.

Yet Stephen Merrill, executive director of the Board on Science, Technology and Economic Policy at the National Academies but who was not involved in the report, concedes that Gathering Storm doesn't, in itself, make a detailed case for the economic benefits of investing in research. For that, one has to look further back in the literature.

Economists have agreed for decades that a large component of modern economic growth has to be driven by 'innovation' — that is, the arrival of new ideas and technologies. "We have very good evidence that 50–70% of productivity growth arises from innovation," says Iain Gillespie, head of the Science and Technology Policy Division at the Organisation for Economic Co-operation and Development in Paris. Greater difficulty arises in determining what drives the innovation, though. Is it basic research, often publicly funded, as the science advocates contend? Or are other factors, such as the demands of consumers who buy, say, mobile phones or computer games, also involved? And even if scientific research does drive innovation, will more investment in science necessarily speed up the process? Unfortunately, economists concede, no one really knows.

In one of the bedrock papers in this field, Edwin Mansfield, the late University of Pennsylvania economist, estimated that academic research delivered an annual rate of return of 28% (E. Mansfield Research Policy 20, 1–12; 1991). The figure has been widely quoted ever since. But Mansfield reached this estimate by interviewing chief executives, asking them what proportion of their companies' innovation was derived from university research and, in effect, demanding that they come up with a number. "He was asking an impossible question," says Ben Martin, a former director of the Science and Technology Policy Research Unit at the University of Sussex, UK. "Methodologically, this was a dubious thing to do."

NIH director Francis Collin is exploring new ways to document the effect of research investment. A. HARRER/BLOOMBERG VIA GETTY

Whatever method economists have used since, measuring the ROI from research has proved tough, and has produced a wide range of values (see Table 2). Some look at the 'micro' level, asking things such as: what contribution did a dozen neuroscience grants received by the University of Cambridge in 1972 eventually make to drug development? Such efforts are complicated, however, by the difficulties of attributing credit for any given drug to the numerous research teams involved over time. Policy-makers are more interested in the 'macro' question, measuring the effect of combined research activities on a country's economic growth. According to Merrill, repeated efforts to pin down firm numbers here have also failed. "It is fair to say that this is an analytical dead end," he told attendees at the American Association for the Advancement of Science annual meeting in February.

Exceptional returns?

Martin says that for much of the literature, "there is some PR, rather than rigorous research involved". This influence derives in part from the activities of US medical research lobbyists. An example is the 2000 report Exceptional Returns: The Economic Value of America's Investment in Medical Research by Funding First, an initiative of the Mary Woodard Lasker Charitable Trust that advocated biomedical research spending. Pointing to work by various economists, the document estimated that the steep decline in cardiovascular deaths in the United States between 1970 and 1990 has an economic value of $1.5 trillion annually, and deduced that one-third of this — $500 billion a year — could be attributed to medical research that led to new procedures and drugs, a finding that was echoed in the Gathering Storm report. A plethora of studies in the United States and Australia followed through with similar claims.

Funding First has been disbanded, but Robert Topel, who studies labour economics at the University of Chicago and whose work was cited in the report, distances himself from some of its claims. "Probably only a little of the fall in the cardiovascular death rate has to do with surgery and beta-blockers," he says. "It is very hard to take changes in public health and attribute their cause." Topel also questions the report's implication that publicly funded biomedical research will create thousands of jobs in the pharmaceutical and biotechnology industries. Topel says that Mark Hatfield, the former Republican senator for Oregon who wrote the report's introduction, was constantly fishing for job numbers. "We kept telling Hatfield that jobs are a cost, not a benefit."

“It is very hard to take changes in public health and attribute their cause.”



The price of research

A key problem, says Topel, has been economists' inability to measure the costs of research as well as the benefits. These costs include the added expense of caring for elderly patients kept alive by new treatments, the costs of talented people doing research instead of something economically productive (such as running a technology company or an ice-cream van), and the cost of wayward outcomes, such as nuclear clean-up — a long-term 'outcome' of the research and development of nuclear energy and weaponry. Research agencies have no interest in assessing the costs of research fairly, says Barry Bozeman, a science-policy specialist at Georgia Institute of Technology in Atlanta. "Honest clients are in short supply" for research in this field, he says. "Most of them think they already have the answers, and want someone to find the numbers to prove them right."

The flaws in the 'exceptional returns' literature were thrown into sharp relief in a November 2008 study called Medical Research: What's it Worth? by the London-based Wellcome Trust and the UK Medical Research Council. In it, some UK health economists attempted to make rigorous estimates of the economic benefits of publicly and charitably funded medical research in Britain.

They estimated that every pound invested in cardiovascular disease and mental-health research brought about, through improved health, economic returns of 9% and 7%, respectively. Work in both fields generated an extra return of 30% through 'spillover' effects from research to the broader economy, such as training and industrial activity. But the report said that these findings were "at best tentative", and spelled out a long list of knowledge gaps.

Little is known about how long the economic benefits of research take to accrue; nor the extent to which the benefits of research done in one country or region are specific to that area, which is a central question for policy-makers. "Three-quarters of the benefits are in spillover, and that's where the evidence is weakest," says Jonathan Grant, president of RAND Europe in Cambridge, UK, and one of the study's main authors. Grant also questions the way that data on ROI gathered in one sector, such as agricultural research, have sometimes been applied to others. "Most of the empirical evidence in this area is (a) historical, (b) American and (c) from agriculture. How transferable is that? It's a big question in my mind," he says.

Efforts to strengthen the evidence are increasing. An $8-million-a-year grants programme at the National Science Foundation (NSF), for example, is supporting investigations by science-policy specialists and economists into various aspects of research economics, including several approaches to measuring the impact of Obama's stimulus package. The programme grew out of an initiative to build "a scientifically rigorous, quantitative basis" for research policy, launched in 2005 by John Marburger, then science adviser to President George W. Bush. As Marburger explains, "We need disinterested people — as opposed to the current situation, where everyone involved has an interest in the outcome."

Julia Lane, head of the NSF project, is also directing the related STAR METRICS programme. The first aim of the programme is to build a 'clean' database of all federally funded researchers in the United States — current records are confused, with conflicting information on names and affiliations — and estimate the number of people that they keep in employment. Later on, the plan is to track patents, citations and other metrics of the research's impact. Lane, like Marburger, suggests that researchers' use of the Internet to communicate and publish will enable STAR METRICS to track the creation and transfer of knowledge properly for the first time. "In the past, we haven't had the data infrastructure to do a full analysis," she says.

Tobin Smith, vice-president for policy at the Association of American Universities in Washington DC, is confident that the first STAR METRICS results in summer 2011 will help to show doubters that the stimulus-bill money has been wisely spent. "It will certainly help me," he says, "by telling our campuses how many people they are keeping in work — something universities have never been able to do." Like Smith, most research leaders and advocates seem assured that new data will reveal the healthy return on investment they have been touting all along.

ADVERTISEMENT

Not that this guarantees that the economic growth argument will continue to persuade. There are signs that a backlash against further research spending is already emerging. In May, the US House of Representatives decisively rejected a bill that would have authorized increased research funding for physical sciences agencies, and in Britain, research spending cuts by the newly elected government are widely anticipated. The pressure is building to show what earlier investments have produced.

As one former congressional staffer, who didn't want to be named, puts it: "If it turns out that all the stimulus has done is hire a load of foreign postdocs, there's going to be trouble."

See Editorial, page 665.