One of the keys to the treatment of disease is early and cheap detection. This is especially true for diseases that disproportionately affect developing nations, where high technology detection methods are often not affordable or even available. Cheap, sensitive, and simple detection kits for serious diseases could make a huge impact on health outcomes in the developing world.

Accordingly, I was very excited when a new protocol for detecting HIV was published in 2012. Diagnosis would be quick, relatively cheap, and involve observing a simple change of color. Even better, the sensitivity, even when tested on serum from infected users, was better than existing tests. It all sounded fantastic.

Fast forward six months, and things weren't so rosy. There seemed to be a lot of problems with the work, and people pointed these out to the people who developed the test. There are two ways these researchers could respond to this criticism: ignore it or refute it. Refuting it is, thankfully, the path being followed. But, the latest study still leaves some large holes to be filled.



Follow the chain of reasoning

The researchers in question here developed a protocol that offers a diagnosis via a solution changing color. The color change is due to the growth of gold nanoparticles. Nanoparticle growth is related to the kinetics—a chemist's way of saying speed—of the reactions that bring the gold out of solution and assemble it into nanoparticles. If the kinetics are fast, then the gold grows into individual spherical nanoparticles, which turn the solution red. If the kinetics are slow, then the nanoparticles have time to stick to each other, forming irregularly shaped aggregates that turn the solution blue.

The kinetics are controlled by the presence of hydrogen peroxide, which reacts with the gold ions in solution to produce nanoparticles: more hydrogen peroxide equals faster kinetics.

For the tests, the concentration of hydrogen peroxide is controlled by the presence of a catalyst that reduces hydrogen peroxide to water. The presence of the catalyst is, in turn, determined by the presence of a single protein found in HIV. This protein binds the catalyst to the surface of a test chamber.

To make this all work as a test, a sample of serum is mixed with a catalyst-containing solution and dropped into a prepared container. After some time, the solution is washed away. If the serum contained HIV, then some catalyst is left attached to the walls of the container; if there is no HIV, all the catalyst is washed away. A solution containing gold ions and hydrogen peroxide is then placed in the vessel. In the presence of the catalyst, the growth of nanoparticles is slowed, and the solution turns blue—go to your doctor, do not pass go. If the catalyst is all washed away, the solution turns red, and you can breathe easy.



Breaking the chain

It sounds great, but there were a number of aspects of this that raised questions within the research community.

First, the protocol appeared to work even when there might only be a single HIV protein in the vessel. That single protein would only bind a very small number of catalytic enzymes, which would then have to eliminate a lot of hydrogen peroxide. Even under optimistic calculations, this would require the most active catalyst known.

A second problem was that there was very little experimental noise in the results that were reported. At the lowest concentrations, the HIV protein was so diluted that there might not be any in some of the chambers, so you'd expect a few of them to turn blue at random. This didn't happen.

Finally, the hydrogen peroxide concentration range over which the color transition occurred was tiny. In fact, it was smaller than could reasonably be measured using standard lab equipment. The paper (and its follow up) reported the transition range without reporting any details on how it was measured. It appeared that you would need custom lab equipment to do the experiment, while the methods section only indicated that standard lab equipment was used.

These three points go directly to the heart of the validity of the researchers' interpretation of their experimental results and reflect a fundamental understanding of the chemistry at the heart of the protocol.

Reconstructing the chain

In the latest publication, some of these points have been clarified. The second point, about the lack of experimental noise, was the easiest to deal with. Indeed, in the original paper, the scientists reported that all three experiments showed a positive result at the lowest concentration. In the new paper, one of the experiments failed. The researchers used a statistical model to show that this sequence of results was not outside the bounds of possibility. It is a pity that they simply didn't do an additional 20 experiments—a trivial undertaking given their experimental setup—to confirm the statistical model. Nevertheless, it's a start.

The first point, sensitivity to single proteins, goes directly to the heart of the mechanism and has only been partially dealt with. The question of how such a small amount of catalyst can effect a macroscopic change is left unanswered. The results, however, are brought back into the realm of the reasonable by some additional details. The main point is that the color change is not permanent—at low concentrations of HIV protein, the solution starts blue but turns red over time.

That's because the catalyst changes the rate at which the reactions occur, but doesn't change the end result. That is, eventually, most of the mixtures will turn red. However, the rate at which it turns red is quite different and, according to the scientists, sensitively dependent on the concentration of hydrogen peroxide. The protocol, then, requires that the color is noted at a particular time, which depends on the starting concentration of hydrogen peroxide. This claim, according to the researchers, is also partially supported by results elsewhere. This puts the results on a more solid footing, though we will return to this point.

The last issue, regarding the catalyst's activity, is also answered by this, at least according to the authors. Although they report absolute concentrations over which the color change occurs, they state that, if their absolute concentration is off, they can simply change the time at which they observe. Or, more accurately, there is a sweet-spot of ~15 minutes, where HIV-free samples should be red and HIV-containing samples should always be blue. Therefore, only the difference in concentration matters. This seems a little dodgy to me, because it involves some judgement about the concentration being made by the person applying the test.

Are the questions answered?

In short, no, not all of them. But the arguments are a beautiful illustration of how science is done.

First, a little bit of information and a nice picture. The researchers claim that the transition between red and blue takes place between hydrogen peroxide concentrations of 119.95 microMolar and 120.00 microMolar. According to this paper, at a hydrogen peroxide concentration of 120 microMol, the reaction is complete at 11 minutes; for 110 microMol, the reaction takes 18 minutes.



At 11 minutes, the color difference between the two solutions is greatest, but, at 10 and 15 minutes, the color difference should still be visible. Unfortunately, the resolution of the picture from the paper isn't very good, so, I have taken a different approach. I used Photoshop to average the color of each solution and used each of these colors as the fill color for a circle. The top row is the color at 10 minutes, the bottom row is at 15 minutes, the left column is at 110 microMolar and the right column is at 120 microMolar. I think I could say with some confidence that the bottom row shows a difference.

That is for a 10 microMolar concentration difference. Compare that with the picture shown below, which shows the same color change occurring when the concentration is changed in steps of 0.05 microMolar. To give you an idea of how the averaging affects perception, I've included both the image from the paper and the Photoshop average. Despite the tiny concentration change, the transition looks the same: very, very sharp.

These two images are supposed to show the results of the same chemistry under the same conditions. No matter how we judge the research, there is clearly something going on that we simply don't understand.



Ars Technica has also obtained data from a third researcher who tried to replicate the basic hydrogen peroxide dependence of nanoparticle growth. In their work, the color change occurs over a much larger concentration range and over a vastly different time scale. Furthermore, their preliminary results hint that the reaction may not be as straightforward as first believed. The point being that this sharp transition actually has a fair amount of published (and unpublished) evidence against its existence.

Independent replication

There are also some people taking issue with the claim that it is an independent replication. The experiments were performed in a different lab, by different researchers. But the original scientists are still authors on the paper. On the face of it, I don't have too much of a problem with this, because the researchers state that all the actual lab work was done in the new lab by that lab's staff. To speed things up, however, they learned the protocol from the original researchers (and anyone with experience will tell you that replicating a protocol based on the text of a paper is chancy and time consuming). So, even though it is not ideal, this aspect of the study is not something I have a problem with.

The analysis is slightly more problematic. Since the analysis was done by both groups, any flaws are likely to have been replicated. On the other hand, the researchers developed a new statistical model to describe the color change at low HIV concentrations. This work was performed by the original scientists and deserves a place in the paper. Therefore, the original scientists are authors on a paper that claims independent verification. This is not usual practice, and I would have preferred a more independent approach. At some point, though, you have to trust that they actually did what they said they did.

This view, however, is not shared by all. One of the commenters at PubPeer contacted Ars Technica to say: "In physical sciences, the protocol should be sufficiently precise to allow replication by another group. This is a condition of publication in all journals. I agree that there is some difference between theory and practice, and, sometimes, you may need to email the authors to get some additional details that were forgotten in the original publication... An independent replication is not merely another person, chosen by the original authors, doing the pipetting under the coaching of the original authors, with their data analysed by the original authors. I really do not know of any other examples where such a thing was called an independent replication."

I think there are some very basic questions about the protocol that still need to be resolved. But the basic issues here—replication, physically plausible results—should get the blood of any scientist flowing. It is also especially important in this case, where time and precious resources may be invested in the development of a proper field trial. We don't need to know exactly how the underlying chemistry works, but we do need to be confident that we can understand it.

Nanoscale, 2014: DOI: 10.1039/C3NR06167A