Published online 13 August 2010 | Nature | doi:10.1038/news.2010.406

News

An analysis of over 50,000 Science papers suggests that it could pay to include more references.

For a well-cited paper, just add references? iStockphoto

A long reference list at the end of a research paper may be the key to ensuring that it is well cited, according to an analysis of 100 years' worth of papers published in the journal Science.

The research suggests that scientists who reference the work of their peers are more likely to find their own work referenced in turn, and the effect is on the rise, with a single extra reference in an article now producing, on average, a whole additional citation for the referencing paper.

"There is a ridiculously strong relationship between the number of citations a paper receives and its number of references," Gregory Webster, the psychologist at the University of Florida in Gainesville who conducted the research, told Nature. "If you want to get more cited, the answer could be to cite more people."

Although previous research has "suggested or shown" a relationship, Webster says, he believes that his study is the first to investigate the phenomenon comprehensively: he has looked at different journals and a large number of articles over a long timescale.

Webster has also found the effect, although to a lesser extent, in the the Journal of Consulting and Clinical Psychology and Evolution and Human Behavior — both important journals in their fields — with the results for the latter published last year1.

His latest study, presented at the International Society for the Psychology of Science & Technology conference in Berkeley, California, on 7 August, gathered data from the Thomson Reuters Web of Science database for all 53,894 articles and review articles published in the journal Science between 1901 and 2000.

A plot of the number of references listed in each article against the number of citations it eventually received reveal that almost half of the variation in citation rates among the Science papers can be attributed to the number of references that they include. And — contrary to what people might predict — the relationship is not driven by review articles, which could be expected, on average, to be heavier on references and to garner more citations than standard papers.

The study also looked at how the relationship has changed over time, finding that it had strengthened more than threefold over the 100-year period studied.

"By most metrics it is considered a pretty big effect," says Webster. "There was a small difference with review articles but, in fact, it was in the wrong direction. On average, review articles actually showed less of a relationship than standard articles."

Webster — who now wants to extend the analysis to include Nature papers, as well as interview scientists about their behaviour — says there is not yet enough evidence to say for sure that the relationship is causal. But he thinks that the psychology of working scientists may see them behave in an almost 'tit-for-tat' way that boosts their citation counts.

Relationships based on reciprocal altruism may bloom and fade but — over time — they might be driving the effect, Webster says. "Scientists are subject to social forces as much as anyone in any other profession."

ADVERTISEMENT

But others urge caution in interpreting the results. Jonathan Adams, a bibliometrics expert and director of research evaluation at Thomson Reuters, says that although the findings are "intriguing" they are "not surprising". At a global level, he adds, there are increasing levels of output and therefore referencing, which would, of course, increase gross citation rates.

Different subjects also have very different citation patterns — and lumping them together as one doesn't tell you very much. "Basic citation count must be contextualised against typical rates for the field," says Adams.

Webster says it's true that he didn't look at different disciplines — but that was not what he was interested in. "This study is just looking at the entire pattern," says Webster, "The research question I was really interested in is what do things look like in general on average in Science."

He says that while he agrees that the scientific enterprise has expanded over time, it shouldn't necessarily affect the relationship between citations and references. "They might both increase in tandem but this effect appears to be independent of that trend."