SCIENCE is, of course, a disinterested pursuit of the truth. Scientists, though, are only human. When it comes to research, that does not matter too much. Significant mistakes will be exposed by the experiments of others. When it comes to handing out the money that pays for research, though, the scientific method is not so easily applied to the business of error correction. Or is it?

One person who thinks it might be is John Ioannidis of Stanford University. Dr Ioannidis’s speciality is using the methods of science on the subject itself. He has, for example, looked at what fraction of published papers turn out, in the light of further research, not to be true—a surprisingly large proportion. Now, in collaboration with Joshua Nicholson of Virginia Tech, he has published a paper in Nature that asks whether America’s National Institutes of Health (NIH) awards its grants effectively, ie, to those most likely to make fundamental breakthroughs in their fields. The answer is perhaps not. Though the projects it does pay for are perfectly worthy, Dr Ioannidis and Dr Nicholson conclude that the highest scientific flyers of all are being kept away from the honeypot.

They came to this conclusion by studying what seem, on the face of things, to be the most influential medical-research papers of the past decade—those that have been cited in more than 1,000 subsequent papers. They then checked to see whether the principal researchers involved were (if working in American institutions) subsequently backed by NIH grants.

Taken for granted?

The database Dr Nicholson and Dr Ioannidis used contains more than 20m papers published between 2001 and 2012. Of those, 1,380 have received more than 1,000 citations. They sorted through these for studies that were biomedical in character (and thus within the NIH’s remit), and which involved primary authors working at American institutions. On papers that had more than one author, they defined the primary authors as those who came first and last in the author list. First authors are generally those (often young) who have done the bulk of the actual work. Last authors are frequently the heads of laboratories, who may or may not be the intellectual driving force behind a paper but who, by putting their names to it, take both credit and responsibility for what is in it. The result was a collection of 700 papers, with 1,172 qualifying primary authors to whom closer scrutiny was then applied.

NIH grants are handed out by groups called study sections, which are composed of experts in the fields in question. So Dr Ioannidis and Dr Nicholson first investigated the overlap between study-section membership, primary authorship of the high-impact papers they had identified, and the current recipients of NIH grants, as listed on the NIH website.

The upshot was that only 6% of the high-impact primary authors they identified as working at American institutions are members of NIH study sections. These people have done well for NIH grants: 89% of them receive such grants. For primary authors who are not members of study sections, though, things are rather different. In order to avoid a “coat-tail” effect, Dr Ioannidis and Dr Nicholson eliminated papers that had any primary author who is now a study-section member. That done, they found that only 40% of high-impact primary authors who are not part of study sections currently receive NIH grants.

All of which is rather worrying. So to try to work out what is going on, the two of them dug a little deeper and found two other things. One was that the successful NIH grant proposals of the high-impact primary authors who were study-section members were much more similar (measured by such things as the frequency of key words in them) to those of other successful proposals than were those of non-member high impactors. This, Dr Ioannidis and Dr Nicholson felt, meant either that study sections favour work similar to that done by their existing members or that they recruit members with similar interests to themselves. Indeed, it is possible that a feedback loop is operating here, because it is NIH policy to recruit into study sections those principal investigators who have previously been given grants.

The two researchers’ second additional finding was that, when they examined the publication records of 100 study-section members, chosen at random, although those members’ citation impacts were classed as “good” or “very good” using as yardsticks the number of citations of their most-influential paper and an overall-citation measure called the Hirsch index, they were not “exceptional”. Given that high-impact authors make up less than 1% of the membership of study sections, that is not surprising. But it is disturbing.

For its part, the NIH does seem willing to acknowledge that its grant-awarding process may not be perfect. Francis Collins, the agency’s boss, says he will “look very carefully at what is going on”, and also observes that the NIH has recently started several programmes specifically aimed at giving grants to rising stars, including one that will enable the best of those with newly minted doctorates—the scientific equivalents of apprenticeships—to skip the journeyman stage of a postdoctoral position and go straight on to being master (or mistress) of their own laboratory research groups. But he does also, in the spirit of the scientific method, draw attention to a few potential flaws in the newly published paper itself, such as the fact that some of the highly cited articles it draws on are reviews of the work of others, rather than pieces of original research (a weakness the authors themselves acknowledge).

With a bit of luck, then, the result will be a synthesis of thesis and antithesis, and an improvement in the grant-awarding process. And, of course, another, similar, study in a few years’ time can always check up to see whether this has happened, and thus how human scientists really are.