A computer scientist at Amherst College has performed the first ever head-to-head speed test between a conventional and quantum computer — and, you’ll be glad to hear that the quantum computer won. But only just — and against a conventional computer that’s 6,000 times cheaper.

The quantum computer being tested was the D-Wave Two, which packs 439 quantum bits (qubits). Ever since D-Wave launched the first commercial quantum computer in 2011 — the 128-qubit, $10 million D-Wave One — the company has faced a lot of criticism from quantum physicists and computer scientists, who claim that D-Wave’s qubits aren’t actually quantum. Over the last year or so, thanks to peer-reviewed studies that have explored the D-Wave One’s inner workings, this criticism has faded.

This new study, from Catherine McGeoch of Amherst College, goes some way to confirming the D-Wave Two’s quantumness — but at the same time, her research shows that the D-Wave Two is nothing like the real, general-purpose quantum computers that should revolutionize the world we live in. D-Wave’s quantum computers use quantum annealing (a type of adiabatic quantum computation) to solve optimization problems — and really, only optimization problems. A true quantum computer, however, should use quantum entanglement.

As you probably know, quantum entanglement is incredibly finicky. The current state of the art doesn’t allow us to harness more than one or two entangled qubits for a few microseconds. Quantum annealing, however, can be performed with much noisier, lower-quality qubits — which is why D-Wave has managed to produce a 439-qubit system that works outside the lab, in a normal office setting. D-Wave’s qubit chips still need to be cooled to near-absolute-zero (0.02K, -273.13C), though, and the qubits (loops of niobium) are still so fickle (affected by external electromagnetic radiation) that each calculation is carried out 1,000 times to ensure its accuracy. Despite all this, there’s still no guarantee that the final solution will be optimal (but it usually is).

To perform the quantum vs. conventional computer speed test, three NP-hard optimization problems were carried out on a variety of systems: A D-Wave Two system containing a Vesuvius 5 chip (439 qubits), Blackbox (a Vesuvius 5/software hybrid), and three software solvers (CPLEX, METSlib Tabu, Akmaxsat) running on an Intel Xeon E5-2690 CPU (under Ubuntu Linux 12.04). The best example of NP-hard optimization is the traveling salesman problem, where you must devise the shortest route between a given number of destinations. Such optimization is computationally very difficult, but it’s theorized that quantum computers should be able to solve these problems much faster.

The results showed that, where the NP-hard problems could be executed directly on the hardware, the D-Wave system is around 4,000 times faster. When Blackbox had to be used, to break problems down into chunks that Vesuvius can understand, performance tied or bettered the software solvers. The study also briefly tested D-Wave’s newest chip, the Vesuvius 6, and found that it would be around 10,000 times faster than the software solvers.

These results probably pose more questions than they answer, though. We’re still not entirely sure how D-Wave’s chips actually work, and so we have no idea if we’re using them optimally. We also don’t know how the D-Wave chips compare to software solvers that use highly optimized simulated quantum annealing — it’s possible that the software would be just as fast as the hardware. Finally, we must remember that the software was running on a ~$1500 workstation, while the D-Wave Two, which was recently purchased by Lockheed Martin, has a price tag somewhere in the $10+ million range (6666 times more expensive). You could build a petaflop-class supercomputer for $10 million — and not only would it be faster than the D-Wave Two at NP-hard problems, but you could run normal software on it, too!

For more information on the testing methodology, and some (mostly) human readable background on what quantum annealing actually is, check out McGeoch’s research paper [PDF]. McGeoch will present “Experimental Evaluation of an Adiabiatic Quantum System for Combinatorial Optimization” at the Association for Computing Machinery (ACM) International Conference on Computing Frontiers next week, in Italy.

Now read: Chinese physicists measure speed of Einstein’s ‘spooky action at a distance’: At least 10,000 times faster than light