How does the testing process work? Which products took the most work to test? Any outlandish tales?

By the time we're testing a handful of items for a particular product category, we've already put in dozens of hours of research and interviews in order to even identify what to test. We really go through each product category with a fine-toothed comb and identify what an ideal product might look like, if such a thing were to exist, before we even begin to narrow down what's actually available on the market to test.

Each product gets a different testing procedure, but we try to put together a plan that does not replicate what other reviewers have already done. So, for example, if it turns out that a site like Anandtech or CNET already ran all the benchmarks you can possibly run on a new tablet, we aren't going to run those same benchmarks again just for the sake of doing so. We're going to name and cite them, and talk about their results, and possibly use that data to help formulate different tests.

We also try to make our tests as "real world" as possible. We recently published a piece on surge protectors where we had an electrical engineer use a noise generator to send voltage spikes through each of our testing candidates to see how each surge protector clamped down on the spike. No one else is doing real world tests like that—you just read the box and you know that this surge protector claims to be able to clamp down 600 volts over a period of 10 years, but does it really? With our tests, we were able to actually see—through the data—how much capability each surge protector loses every time it has to clamp down on a giant spike like that. (Did you know they can only do that so many times before failing, and the only way to know your surge protector is on its last legs is when it finally blows out?)

I would say one of the products that took the most work to test was bike locks. We spent a huge number of hours—seriously, it's terrifying—interviewing and working with a professional bike thief to learn about all the ways in which most bike locks fail, and how thieves usually get around even the best bike locks. We followed this guy around and he showed us exactly how things work and what to look for, and he performed all the testing for us when we narrowed things down to the final candidates. Sometimes, testing isn't so much about the technical details (like with surge protectors) and more about how real humans are going to interact with the item. In the case of bike locks, the things you have to think about the most are actually human factors, with the technical details coming second.

Have there been results that were particularly surprising? Any top-of-the-line products that failed miserably? Unknowns that triumphed?

To be honest, we find surprising things pretty often during testing. For example, we're working on a piece for Sweethome right now that evaluates pepper mills/grinders, and not one, but two of our final testing candidates ended up burping out metal shards along with the ground pepper. And this is after eliminating hundreds of others based on our research and other reviews, so these are not bottom-of-the-barrel pepper grinders. I mean, can you believe that? Two of the "best" pepper grinders that you can buy from reputable brands basically cover your food with metal shards? Wouldn't you like to know that as a consumer? We also found one popsicle mold that put metal shards into the popsicles themselves, so this isn't just limited to things that grind.