bezos.JPG

Amazon founder Jeff Bezos has found that many of his innovations have had unintended consequences.

(The Associated Press)

For years, undergraduates wielded astonishing -- and mostly secret - power in this country. The reason: college students were the dominant group serving as test subjects in academic research.

The various studies conducted by universities -- in disciplines ranging from economics to health to psychology -- play a key role in determining government policy and even shaping our view of ourselves. That meant the responses of 19- and 20-year-old across the land cumulatively carried weight.

But over the past decade a new group of test subjects has risen up to increasingly take the place of college kids and others who participate in academic studies. And this might not be a good thing.

They're called "Turkers." Jenny Marder of PBS Newhour has produced a fascinating, must-read report about them and how they could be skewing the results of important academic research.

We're talking about the destruction of an old order here, and about unintended consequences, so that means we're talking about Amazon. Ten years ago Jeff Bezos' company launched Mechanical Turk, an online "Marketplace for Work" that allows "requesters" to crowdsource minor, routine tasks such as transcribing audio or making phone calls.

Sounds like a good idea, right? Academic researchers certainly thought so, and they rushed in to take advantage of it. Using Mechanical Turk, they found they could easily acquire all the feedback they needed for their studies at pennies per response.

The problem is that Mechanical Turk, like Amazon's straightforward consumer site, has developed a small cadre of super-users. These are people who are essentially making their living by filling out questionnaires all day long, day after day, month after month, year after year.

Researchers began turning to Mechanical Turk because it promised a bottomless well of participants. But it turns out they're simply getting the same few people over and over again. A Turk super-user apparently participates in more studies every week than traditional participants do in their lifetimes. There's also the fact that researchers cannot control the environment for their study when their subjects are coming from the Internet. "As the use of online crowdsourcing in research continues to grow," writes the Newshour's Marder, "some are asking the question: How reliable are the data that these modern-day research subjects generate?"

In short, we just don't know.

David Rand, a Yale University psychology professor who was an early advocate for the Amazon forum, calls the output of the Turk super-users "a big problem."

It's not just the issue of having a sample group that's too small. Many studies rely on what's called "gut responses." But if you're participating in hundreds of studies every week, your gut becomes numb. "You kind of just lose that freshness," one Turker told the Newshour. Another way of putting it: the study subjects have figured out the researchers' tricks, and this means they can't be counted on to produce honest, uncynical responses.

Marder ultimately doesn't reach a conclusion. "Are Turkers misleading scientists and corrupting their science? Or, in a universe of imperfect study participants, is Mechanical Turk the best option available?" she asks.

It's an important question, and in this case, a gut response won't be enough.