One of the most common questions we’re asked about hardware reliability is whether there’s a real difference between the various storage manufacturers. This information is typically locked up like Fort Knox, which is one reason why Backblaze’s ongoing storage reports have garnered widespread attention. Most drive reliability studies are either small scale, take place over limited time scales, or refuse to reveal individual vendor ratings. Backblaze, on the other hand, comes right out and says what it has been testing and what the failure rates look like.

I’ve criticized Backblaze in the past for using data sets that compared drives with wildly different ages, workloads, and storage conditions, but this latest data set appears to be more standardized as far as workloads are concerned. Let’s start with the big picture — which drives, from which manufacturers, fail the most (and least?)

Yes, the chart is to scale, and no, that giant Seagate column isn’t a typo. The failure rate on many Seagate drives is simply abominable, from 9.5% on certain 1.5TB drives to 23.5% on the old 7200.11 series, to a whopping 43.1% on the Seagate 7200.14 drive family of 3TB products. Age alone does not account for it — the 7200.11 drives are nearly 5 years old with a 23.5% failure rate, while the 7200.14 drives are half that age with a 43% failure rate. Many of Seagate’s problems, however, appear to be in the 1.5TB to 3TB range. At 4TB, Seagate is much better, with a failure rate of just 2.5% in 2014.

HGST (formerly Hitachi) remains the strongest and most reliable vendor of the three that Backblaze tracks, though they do note that data on Western Digital products is very scarce. Backblaze doesn’t source many WD drives because “Over the course of the last year, Western Digital drives were often not quoted and when they were, they were never the lowest price.”

According to Backblaze, the Seagate 4TB drives are already tracking much better than previous models, with an estimated annual failure rate of 2.6% in their first year, compared to the original failure rate of 9.3% for Seagate’s 3TB drives.

How much weight should consumers put on these results?

The Backblaze data is interesting, but it’s still shot through with problems that limit just how much weight I’m willing to put on it. Backblaze has redesigned its storage pods several times since it began gathering data in an attempt to limit vibration. The company has an admitted habit of sourcing the absolute cheapest drives it can find, which virtually guarantees that some of the products its stocking are going to be used or refurbished units. Its relentless focus on price above all other characteristics makes sense for its own operating environment, but the company’s use of consumer drives in enterprise-class deployments may create massive confounding variables.

It’s entirely possible that the cheapest HGST drives include superior vibration dampening technology to the cheapest Seagate drives. This isn’t a problem in consumer systems where there are rarely more than two physical discs, and those discs don’t usually spin at the same time. It could be a profound problem when 45 drives are stacked in an enclosure. Compounding this issue is the fact that Backblaze’s previous reports have acknowledged that different drives are put under different workloads, with apparently no regard for whether or not the stated workload matches the manufacturer’s intended ratings for the disk. Price, not workload, governs Backblaze’s decision process.

None of this is meant to imply that Backblaze’s work is wrong, as such, but it’s not at all clear how applicable it is to every day consumers and would-buy reliability hawks. We can be reasonably certain that Seagate’s 3TB and 4TB drives don’t fail at anything like 25-40% in the real world, or else the entire internet would be on fire with self-reported problems. We checked, and it isn’t. Tweaktown wrote an article discussing many of these issues last year; it’s worth a read if you want to explore them in more detail.

Backblaze could substantially address these concerns by pruning their data sets to only include drives that were housed in the storage pods and ran the same workloads. While this wouldn’t resolve the fundamental mismatch between the manufacturer-intended use environment for the HDD and the demands of Backblaze’s business model, it would at least give us enough information to make an apples-to-apples comparison.

Since the company seems unlikely to do that, I’m at least willing to say that HGST appears to come out of this comparison with a reliability gold star. If you’re shopping for a new drive, I’d keep that in mind.

Now read: Why are some hard drives more reliable than others?