One of the problems with much of science is that it's often a quasi-sedentary profession. Apart from those lucky few who have to trek through muddy fields in Vietnam or scale icy cliffs in Antarctica, the rest of us mostly took the job because it was indoors and required no heavy lifting.

That said, scientists get a lot of exercise. Usually, this involves energetically jumping to conclusions—there is nothing quite like seeing two data points, fitting a straight line and yelling, “We understand this COMPLETELY.”

The newest craze among the more energetic set is aquatic: shark jumping. Science is undergoing a cultural change at the moment, thanks to the Open Science movement, an idea that I am broadly supportive of. But some of the latest goals and statements seem a little… unrealistic to me.

Opening up science

It all began with open access: the idea that published scientific results should be open to anyone. I've always been supportive of open access, but I'm now much more in favor of the practice. I think that arXiv.org should be our model: any scientist can upload drafts of papers before they go through peer review, allowing anyone to read them and others in the field to review them informally. This is in contrast to a place like PLoS, which puts a paper through formal peer review before posting it online for anyone to see.

That's not to say that the PLoS stable of journals is deficient; I just happen to not have faith in peer review anymore. Most of the expenses associated with PLoS are centered on editorial functions, a submission process, and matching reviewers to manuscripts; the arXiv gets rid of all of this, making open access even cheaper.

To be clear, though, the arXiv has some absolutely bonkers papers on it, papers that are so bad that you have to wonder how the person writing it managed to even operate a computer. Peer review would filter some of those out.

Some, but not all. If you still believe that peer review works, I direct you to PubPeer, where you'll find plenty of examples of papers that are flawed or controversial in many ways. My point is that I don’t think the basic filtering of peer review is sufficient to justify a system that doesn’t actually manage to do anything beyond that.

Although I still have to publish in some closed access journals, I can see no downside to the entire scientific literature being open access. Indeed, I no longer see much of a downside to removing pre-publication peer review.

Looking behind the curtain

I'm also fully behind the idea that the data that underlies a paper should be available to readers. In fact, just this month, I’ve had students who attempted to get clarifications on figures and methods from papers, but the corresponding authors did not respond. This is really frustrating.

I have some more personal worries here, unfortunately. Almost all of my research is co-sponsored by industry. And as we all know, companies have their secrets. For instance, in two of our papers, we characterize a new type of waveguide and then show that you can do some really awesome stuff with it.

One of the companies we work with developed a process that allows it to make waveguides that no one else can make. The company has a competitive advantage, and it spent its own money to develop the techniques that allows it to make the waveguides. There is no desire or obligation on the company's part to release the details of the waveguide fabrication process. Sure, the paper broadly outlines the fabrication steps, but we certainly didn’t give enough details that someone could replicate it without a serious amount of work.

That limitation does not fit with the open data ideal. Everything we do is open. But the stuff we get from the company is not open, and we are not in a position to reveal their secrets. I worry, though, that this will not be enough to allow us to publish the work we do—if we purchase something from a company, especially when we also work in collaboration, we would not be able to publish unless we could give more detail on how that thing was made.

This could, in effect, set the boundary between science and not-science far too close to what I do for my comfort. To be fair, this is my problem and not one with the principle of open data. I can see a few disincentives in terms of the type of work that might be funded and the way we work with companies, but there is nothing that stops me from getting behind the idea.

For instance, in the case described above, the first paper would run afoul of open data principles, but the second would not. I don't like that, but I can live with it if I have to.

Jumping the shark

Now things are going a step beyond open data and papers. As seen in my example, many of the tools we use are not open. These aren't just physical tools. Most of my colleagues use Matlab or Mathematica, but I use Python, which is open. However, I’m not going to force my students to learn a new programming language, especially if the only reason is that some idealistic fool like me objects to not knowing how Matlab implements the Levenberg-Marquadt algorithm.

I happen to agree that there is a danger that an algorithm might not always function correctly due to invisible changes in proprietary software—we need to be aware of this. But frankly, that doesn't mean everyone must use open source products. It simply means that everyone should be skeptical of the results and run many test cases to build trust in their results. (Building fake data and running it through your analysis algorithm matters, irrespective of the licensing of the code.)

That said, I agree that algorithms are part of the data that should be shared with a publication: I don’t care if you wrote it in Matlab, Pascal, or machine code, the code should be available. You make it available; it's my problem to make it work for me.

The problem of collaboration and repeatability is exaserbated by everyone using their own tools. As the paper "Open source tools for large-scale neuroscience" says:

Solving these challenges will not only require new tools, but also a new culture. Most labs develop custom analysis strategies, using proprietary tools like Matlab that are poorly suited to collaborative development, inventing creative algorithms but only applying them to data from the lab in which they were developed, because they are hard to reproduce, require complex configuration, and barely run on single workstations.

I agree that work would be more replicable and collaboration would be easier if we all used the same tools. But this collaboration doesn’t depend on the openness of the software—it only depends on availability. I would love it if everyone I worked with used Python, and I suspect my students would all love it if I switched to Matlab. Neither are going to happen.

The subtext floating around in the open science movement is that if you don’t use open tools, you shouldn’t be able to publish because your work can’t be replicated. Frankly, I don’t think anyone's science should be rejected based on someone’s inability to purchase a Matlab license. But this is exactly where the argument being made against commercial software seems to be heading.

Jumping a whole school of sharks

If you take the algorithm argument to its logical conclusion, it becomes rather awkward: the full design of every lab instrument needs to be available. I’m sure that outfits like Bruker and Agilent will have no problem with opening up the designs of their atomic force microscopes.

There certainly can’t be any canny tricks that they’ve developed to give themselves a competitive advantage or differentiate themselves from each other, can there? So let's force them to spend huge amounts of money on patent protection rather than just keeping these innovations a secret. That will certainly make scientific instruments cheaper and increase access.

I’m not even entirely comfortable with sharing everything related to customized instruments. Take these scanning tunneling microscopes built by researchers at Leiden University, for example. They are based on commercial instruments, but the researchers have spent considerable time and effort customizing them so that they can do things far beyond the original design's capacities.

If I had to guess, I'd say those customization efforts were done in collaboration with the company that supplied the microscope. The deal was probably that the researchers get the microscope cheap and get a head start on everyone else in the field, and the supplier gets the details of the customization. In a couple of years, the customization is released as an accessory that is commercially available to everyone.

I see that as a win for everyone.

The alternative is that the scientists have to ask for much more funding to purchase the microscope at full cost. Which means less money for actually using the microscope to do science. They'd also have to make the modifications without the assistance of the company engineers who designed the ‘scope, and that process takes more time and is higher risk. And as the work is completely open, there is less motivation for the company to offer it as an add-on, since their competitors can also offer it. The likely outcome is that every researcher who wants to follow in Leiden’s path has to build it themselves.

I just can’t see what we’ve won in that situation. Sure, the manufacturer isn't getting a huge markup on an expensive add-on. But no one is getting that add-on unless they have the engineering skills to build it themselves.

If it’s open, that’s good

One counterargument is that if it is open, you can look inside and see where the flaws are and avoid reporting artifacts as real signals. I think that's a poor argument. If you are skilled enough to read the design files and use them to understand how your data might be an artifact, you are already skilled enough to know how a particular measurement may be an artifact and how to test for that without opening up the box. In fact, it's probably quicker to do that test than to go through the design files.

The alternative is that you are not skilled enough to recognize an artifact, in which case it's unlikely that you'll help yourself by going through the design files.

That said, I’m not against open instrumentation. If Bruker and Agilent choose to go that route, I will not rail against it. Given a choice, I will buy open instruments. Also, if new companies choose to go the open source route, I will not weep if they drive Agilent out of business. If the Matlab source code is released tomorrow, I’m not going to hold my head in despair.

On the other hand, I’m not going to put up with being shut out of publishing my work because I choose to use the instrument most suitable for my work, even if it is a closed instrument. And as long as that idea is the underlying threat, you can spot me over there: I’m the one giving the single-finger salute.