Published online 20 January 2010 | Nature 463, 291 (2010) | doi:10.1038/463291a

Column

Quantitative research assessment is a bad idea whose time has come, argues Colin Macilwain.

Is your work any good? Academic researchers will be asked this question with increasing frequency in this new age of austerity.

Next month, the Higher Education Funding Council for England (HEFCE) releases its final plans for the Research Excellence Framework (REF). This will be the seventh implementation of a systematic UK exercise to assess university research quality that is now the oldest and largest of its type in the world.

Successive Research Assessment Exercises (RAEs) at British universities have been widely lauded, at home and abroad, for helping to raise the quality of research. Their history is instructive because so many other countries are now following suit. Australia, the Netherlands and some Scandinavian nations are imitating aspects of the RAE directly. Extensive university reform is putting France and Germany on a trajectory towards similar approaches.

European Union research programmes, notably the new European Research Council, are energetically pursuing quantitative assessment of the people they support. And research assessment is set to come to the fore in the United States, as science spending slams into reverse after last year's short-term stimulus package. Such a change would mark the end of decades of successful resistance by US scientific leaders to the idea that the quality of their work can be measured quantitatively, like grain output.

The biggest lesson to be drawn from the UK RAE, which has been conducted six times since 1986, is social scientists' equivalent of the uncertainty principle: such exercises influence the behaviour of the observed, often in unforeseen ways. Whatever is measured becomes emphasized, probably at the expense of whatever is not. And as metrics change — as they must to stop institutions gaming the system — the process loses simplicity, transparency and credibility. When that happens, the exercise's direct influence on funding may be disrupted — enraging university staff, who spend so much time preparing for it.

High-impact performance

The UK RAEs have become the central measure of university success, even though the government funding apportioned on the basis of their results is just 6% of total annual university income — £1.5 billion (US$2.4 billion) out of £24 billion this year. When the Oxford-based Higher Education Policy Institute held a conference in London last October to discuss the topic with senior university leaders, the hall was packed to the rafters. A previous meeting, on the 'student experience', was only two-thirds full.

“The Research Excellence Framework could simply pit discipline against discipline in a race to tell the tallest tales.”



Since 2006, when a small deputation of vice-chancellors met with Gordon Brown — then Chancellor of the Exchequer — and convinced him that the RAE was too expensive and complex, HEFCE has been planning a successor that would have relied chiefly on citation analysis, as opposed to subjective peer review. That idea got dumped last spring in the face of widespread scepticism in disciplines ranging from engineering to English literature. Or almost dumped: citation data will still be gathered, but review panels will be given leeway on how to use them.

Instead, 'impact' has emerged as the signature aspect of the REF. That's because the Labour government, having invested very heavily in science over the past decade, is impatient for results. Not Nobel prizes, or papers in Nature, but something the taxpayer can actually eat or use to travel to work.

This new demand has already sent British academics scurrying off in search of a convincing narrative for their own work's influence on the economy, or on society, over the past 10 or 15 years. But given the paucity of an agreed understanding of how innovation really works, there are justifiable fears that this element of the REF will resemble an essay-writing competition, in which departments each submit 1,000-word yarns about how their work has changed the world. It isn't clear how the review panels will reliably assess these expositions.

This emphasis on economic and social impact has been publicly welcomed by university leaders, however apprehensive they may be in private. But it places growing, unrealizable expectations on university research as an economic driver — as Geoffrey Boulton, vice-principal of the University of Edinburgh, and Colin Lucas, former vice-chancellor of the University of Oxford, warned in a prescient 2008 paper (see http://go.nature.com/EEleZI). Since then, research lobby groups around the world have, rather desperately, pledged that research funding will help to secure economic revival. Little wonder that politicians are seeking data to back up these claims.

The right questions?

UK academics are now campaigning to slim down the 'impact' component from 25% to 20% of the REF, when its final form is released next month. (Sixty per cent of the REF would be determined by research quality, under the HEFCE proposal, and the remaining 15% by 'research environment'.) Allied to the uncertain status of citations data in the process, this leaves the REF looking less transparent, and potentially less effective, than the process it replaced.

The overall exercise, having grown steadily in influence and scale — the last RAE is estimated to have cost £60 million — is attempting to do too much. Formal research assessments can work well within specific disciplines on the basis of qualitative peer review — at least in science and engineering. It isn't too difficult for a panel of geophysicists to sit down and grade a set of submissions.

Sensible attempts can be made to broaden such analyses into an assessment of impact within science itself. The European Research Council, for example, is studying new techniques for assessing how its grant recipients participate within global networks of researchers, asking not only 'do they publish?' but also 'are they players?'. A group funded by the council and led by Maria Nedeva, a sociologist at the University of Manchester, UK, for example, is modelling the global research community — including conference organizers, professional societies, government advisers and highly cited authors — in certain disciplines to assess the respective roles of European and American contributors. Such approaches will, in time, grow in sophistication and effectiveness.

But the question of what a given advance in knowledge does for society as a whole is essentially unanswerable. In attempting to answer it by quantifying the impact of research outside the academy, the REF could simply pit discipline against discipline in a race to tell the tallest tales.

See http://go.nature.com/ILx8PC for more columns.

cfmworldview@gmail.com