If I were to summarize the ideal scientific paper in four sentences, it would look like this:

Look at this cool thing we did.

This is how we did the cool thing.

This is the cool thing.

Wasn't that cool?

We like to think that the standard format (not to be confused with the Standard Model) was beautifully followed in days of yore. Nowadays, of course, it is not. Because things always get worse, right? In reality, scientific papers have always looked more like this:

Look at this cool thing we did, IT IS REALLY COOL, BE INTERESTED.

This is how we did the cool thing (apart from this bit that we "forgot" to mention, the other thing that we didn't think was important, and that bit that a company contributed and wants to keep a secret. Have fun replicating the results!).

This is the cool thing.

This thing we did is not only cool, but is totally going to cure cancer, even if we never mentioned cancer and, in fact, are studying the ecology of the lesser spotted physicist.

Call me cynical, but missing information in the methods section, as described in the parenthetical in item two, really, really bugs me. I think it bugs me more now than it did ten years ago, even though I'm no longer the student in the lab who's stuck with filling in the missing methods himself.

Maybe I'm angrier than I used to be, because, as I've gotten more experienced, it has gotten easier to spot the carefully camouflaged gaps.

Let's play the blame game

I blame publishers. In the good old days of print journals, each edition only held a finite amount of information, so paper lengths were limited. Although you may have needed 30 pages of close-spaced text to describe how you accomplished some arcane scientific feat, some journals only gave you half a column. Any scientific results that could not be communicated properly in a short format ended up in another journal that could accommodate them.

To keep papers short, many journals emphasize results and conclusions at the expense of methods.

This sometimes led to double publications: a short description of your results was published in one journal, while the extensive explanation of what you did appeared in a more technical publication.

Over time, short, direct articles have become more prestigious. Since university administrators are all about prestige, scientists now face increasing pressure to publish shortened forms of their research. The publishing houses, many of whom benefit from this pressure, are happy to accommodate.

To keep papers short, many journals emphasize results and conclusions at the expense of methods, often by moving them to the end and printing them in a font that requires a microscope. When I tried to report on a paper about adiabatic quantum computing recently published in Nature, I was dismayed to discover that all the useful information on methods wasn't in Nature at all, but in a separate document called supplementary information.

Supplemental

Supplementary information doesn't come in the print version of journals, so good luck understanding a paper if you like reading the hard copy. Neither is it attached to the paper if you download it for reading later—supplementary information is typically a separate download, sometimes much larger than the paper itself, and often paywalled. So if you want to download a study's methods, you have to be on a campus with access to the journal, use your institutional proxy, or jump through whatever hoops are required.

Right now, you might be saying to yourself, "Come on, Lee, you must have noticed that you needed the supplementary information when you first skimmed that quantum computing paper in Nature." But that's not how I work. When I skim, I concentrate on the introduction and results. Essentially, in my first read-through, I want to know two things: is this paper relevant to my work? And/or is it interesting to write about? If a paper doesn't pass that threshold, then I don't care about its methods.

For my day job, most of the journals I read do not publish articles with supplementary information. When writing at Ars, I often work about a month behind publication date, so I have time to read and, if necessary, download the supplementary information. But most of my paper reading occurs on trains where Internet access is spotty. I've abandoned more than one story because I discovered that I needed supplementary information that I couldn't download. (If you didn't know that it was possible to rage-quit writing, now you do.)

Papers should not be crippled in order to comply with arbitrary page limits.

The adiabatic quantum computing article was even worse. The paper was not published at that time, so I only had the press kit. Supplementary information is often provided to the press, but Nature apparently couldn't be bothered this time. Yes, the results looked pretty, but I couldn't even figure out whether they were experimental results or just calculations. Even when I decided they had to be experimental results, the one picture of the researchers' real set of qubits showed something that didn't look anything like what the results hinted at. The press kit extensively referenced supplementary information that would have clarified all of this, but of course I had no access to that. When I contacted the authors, they were kind enough to provide the supplementary information. Suddenly I had twelve more pages, and their paper finally made sense.

This is not how science works. Papers should not be crippled in order to comply with arbitrary page limits. And, even with the supplementary information, I don't believe that a competing group of researchers could reproduce the work described in this paper. There is simply not enough information to make that possible.

Unnecessary speed bumps

There is no reason an online research paper should be accompanied by a document called supplementary information. Everything should be in the main document, even if some things are relegated to an appendix. Just because publishers have chosen to make their print versions increasingly cryptic doesn't mean that they should cripple their online versions, too.

But more generally, methods should not be part of some appendix—they are a central pillar of any research report.

Consider the general reader. After all, general interest journals want to publish results that are compelling to a broad range of scientific specialists. Exactly how does Nature's quantum computing paper, and others like it, help? By itself, it is unreadable and meaningless (not the fault of the authors—if you just read the supplementary document before looking at the figures in the main paper, you have a clear, well-written paper).

The authors... were kind enough to provide the supplementary information. Suddenly I had twelve more pages, and their paper finally made sense.

Shouldn't a "general interest" paper satisfy interested parties that include physicists from further afield than me? And what about chemists, biologists, social scientists, and all the other scientific specialties? How do they benefit from such an incomplete paper? Apparently, journals think that general interest doesn't mean "write papers so that people both in and out of the field will be satisfied." They think general interest means "remove context so everything looks like gibberish." If you're an expert in the field, the paper itself is irrelevant—the only things worth knowing are in the supplementary information.

There's an obvious solution here: restrict supplementary information to things that really can't be included in a .pdf (video content, for instance) and make sure that a paper's .pdf includes all the relevant information. Or, at the very least, provide a single click link that downloads all the relevant content. This will still leave print readers confused, but I'm willing to make some sacrifices for a greater good.