Theranos is more likely than conventional labs to return blood work with unusual numbers, according to the first comparison study between the Silicon Valley start-up and two well-known commercial labs. The results suggest that doctors who use blood tests from Theranos may decide to treat patients for conditions they don’t have — or fail to treat patients who need their help.

Theranos’ results were compared to those from LabCorp and Quest Diagnostics, in the report from Mount Sinai, which was published in the Journal of Clinical Investigations. For multiple tests, Theranos’ results were more likely to fall outside the range that’s considered normal for a healthy adult compared with results from the other companies.

Theranos' results didn't match those from the other labs

In addition, Theranos’ results from certain tests often didn’t match up to those from Quest Diagnostics and LabCorp. For instance, the results on cholesterol tests (which doctors use to check up on heart health) were systematically lower than results returned by the other two labs, says Eric Schadt, a biologist at Icahn School of Medicine at Mount Sinai and a co-author of the study. So a doctor who uses Theranos' tests might choose not to treat a patient for high cholesterol — or recommend an inadequate treatment plan.

Theranos, a blood testing startup that's been valued at $9 billion, is known for its fingerprick blood tests, which it says can generate results for hundreds of tests using only tiny samples of blood. Until recently, the startup was thought of as a huge success story, both because of its technology and its charismatic CEO, Elizabeth Holmes. But reports from The Wall Street Journal published last year cast doubts about the accuracy of Theranos' technology. Then, in January, federal officials announced that Theranos' Newark, California, laboratory posed "immediate jeopardy to patient safety." Because of this, Theranos’ biggest partner, the drugstore chain Walgreens, shut down one of Theranos’ clinics; now, the drugstore giant is reportedly trying terminate its agreement with the startup. But it’s worth remembering that the controversy surrounding the company didn't start last October. From the start, Theranos dodged opportunities to validate its findings in peer-reviewed journals — even after claiming that it would do so last October.

Today’s comparison study was conducted without Theranos’s cooperation. Researchers attempted to contact Theranos through LinkedIn after receiving results from all three companies for each participant, but were unsuccessful, Schadt says. So the researchers say they aren’t sure Theranos' inventions were used to return results for blood samples drawn from patient's fingers; it’s possible that commercial machines were used instead. Theranos told The Verge that "all of the tests were run on Theranos technology." So today's report represents the public's first look at Theranos' real-life performance.

The scientists used LinkedIn to contact Theranos

In response to the study, Theranos wrote a letter to the Journal of Clinical Investigation saying the company was disappointed that a "flawed" study was published. Theranos told The Verge that the study's authors misrepresented some of Theranos' results and that the methodology they employed was wrong. The authors have responded to these allegations in a document they shared with The Verge. In it, they state that they had limited resources, but that they stand by their work. "We think our study results are of interest, and we hope will motivate others in the scientific community to conduct additional research to validate our findings and address any limitations of our approach," they write. "This is indeed the scientific approach."

Theranos also denies that the researchers attempted to contact the company before the study's publication. "The authors are making false claims to the media to inappropriately bolster their credibility and that of their study," the letter states.

In a phone call, Theranos spokesperson Brooke Buchanan said Theranos staff searched for emails and voicemails from the researchers, but could find nothing. They didn’t know the researchers used LinkedIn until today. "It's unfortunate that they chose to use that, rather than just call us directly," she said.

"The authors are making false claims."

To compare Theranos' tests to those conducted by commercial labs, all blood samples were taken on a single day, within a few hours, from 60 participants. The first blood samples were taken from participants’ arms at an outpatient clinic. Then, researchers drove participants to two different Theranos Walgreens Wellness Centers in Phoenix, Arizona, before taking venous samples from them again. Once the experiment was complete, the team wrote to Theranos CEO Elizabeth Holmes on LinkedIn. They also used LinkedIn to contact Sunny Balwani, Theranos' president and Patrick O’Neill, Theranos chief creative officer, to ask about the procedures the company used. These failed attempts were made on December 14th, Schadt says. Two screenshots of the messages can be viewed here and here; they show that the messages were never read.

The researchers found that there was a significant degree of variability between Quest and LabCorp as well — not just between those two and Theranos. But Theranos returned abnormal results more often than Quest Diagnostics or LabCorp. These problems are likely fixable, Schadt says.

As it stands, Theranos’ results may mean doctors will make bad decisions, says Eleftherios Diamandis, head of clinical biochemistry at the University of Toronto. Results that vary as much as these could mean doctors will treat a patient who doesn’t need care or miss a health condition that needs treatment. The finding suggests that Theranos’ fingerprick collection isn't as good as a standard vein draw. "What would I do if I were a patient?" he says. "I will leave the choice to the patients but for me, the choice is obvious."

In its letter to the journal, Theranos says that taking a blood sample from a person's arm before doing a fingerprick may make the fingerprick tests less reliable. The company also reprimands the researchers on their methodology: the study doesn’t say if more than one draw was conducted on each finger, for instance. "These fundamental problems problems with the conduct of the study reflect the lack of understanding of the study investigators concerning basic sample collection procedures..." the company states. In addition, Theranos lab director Daniel Young told The Verge that Theranos didn’t report abnormal results as often as the researchers claim in the study.

"Not at all clear why Theranos would ever list us as a 'potential competitor.'"

Theranos also claims that a conflict of interest wasn't disclosed in the study. Two of the study's authors, Schadt and Joel Dudley, have ties to a bioinformatics company called NuMedii that works in drug discovery, the company states in its letter to the journal, and that qualifies it as "potential competitor." "We are not currently working on drug discovery, but we have in the past and could again in the future," Young told The Verge on the phone. NuMeddii, for its part, told The Verge that the two companies have "distinct and different business models," so "it is not at all clear why Theranos would ever list us as a 'potential competitor.'" The researchers give a similar response in the document they shared with The Verge.

Schadt says the biggest problem the researchers faced while conducting this study was that they weren’t sure which machines Theranos used. All the Theranos samples were taken from participants' fingers, so it seems likely that the tests were conducted using Theranos' inventions, which is what the company says happened. But a WSJ report which suggested last year that Theranos has diluted blood samples so they could be analyzed on commercial machines, cast doubt in the researchers' minds.

If today's study is representative of Theranos' usual performance, it raises some significant questions about federal lab certification processes and inspections that are already in place, says Richard Horowitz, an emeritus professor of pathology at UCLA. "Testing disparities do occur," he says. But this study shows that there's "a need for more investigation of the factors that contribute to test variability."

The US government cited Theranos Newark Lab for putting patients at risk in January. Theranos says it has performed nearly six million tests in the past two years. The company could put qualms about its methods to rest if it were to publish its data — until then, studies like today’s represent the best information available about Theranos’ accuracy. In an email, Buchanan told The Verge that Theranos plans to publish data "in the near future. Stay tuned!"