This post details how to setup and use Amazon’s Mechanical Turk with the SERP Turkey tool. If you’re not familiar with SERP Turkey, or want instructions on that tool then you should first read the SERP Turkey introduction.

If you’ve never heard of mTurk then there are good introductions here and here. Basically mTurk provides a huge online pool of workers – people at home who get paid anywhere between a few cents to a few dollars to complete simple tasks online. Anyone can join the workforce as a ‘worker’, and people in the US can become ‘requesters’ that set tasks. Here I describe how to become a requester and how to setup tasks that bring workers to your SERP Turkey tests. Don’t despair if you are not in the US – there is a method for you to get Amazon Turk workers with an extra step – see the section below.

Once you’ve registered as a requester (which may take a day for your first payment to go through), and created your template (10 mins) you’ll be able to create new SERP Turkey tests and get the live in SERP Turkey in less than 5 minutes.

Registering as a requester

So – lets get started. Head over to the mTurk request start page, and fill in your email and press the button to proceed. You’ll be asked for your name, email confirmation and to create a password. Next you’ll be asked for your company name and US address and phone number. Check the box to agree to the terms and proceed to the next screen. So far so good!

Ok, if you’d like to take the tour you’re offered now, go for it and I’ll see you on the over side. Otherwise just press ‘Get Started’ and start setting up your first task.

But some credit

mTurk works on a prepayment system and you have to buy credit to start. This is simple – just goto ‘Account Settings’ and then click the ‘Prepay for Mechanical Turk HITs’ link.

You’ll need a US registered credit card (though I’ve heard some people got through without one) to make the payment. be aware that your first payment might take 24-36 hours to get approved.

Creating a task (HIT) for SERP Turkey

mTurk tasks are called HITs (Human Intelligence Task) and are generally created using re-usable templates. Templates will define:

HIT Title, Description & Keywords

Payment per task (per worker)

How many workers you want

Time allotted per assignment

How long before the task expires?

Worker required qualifications

Templates will also define how you wish the question of the task to be presented to the user.

So, lets get going… You’ll be asked what sort of work you want the user to do:

Select ‘Categorization’ and you’ll be presented with a list of pre-built template types:

Select the Search Relevance template, which we’ll customise.

Here are the settings I’ve been using:

Title: Click the best result from a set of Search Results

Description: Simply click on the result that you’d select yourself for that search.

Keywords: search, evaluate, opinion, google, quick

Time allotted per assignment: 2 minutes

HIT expires in: 7 days (generally I’ve not been fussed about needing the results before a deadline – at $0.05 per HIT you should allow 24-36 hours per 100 users)

HIT Approval rate: >=90% or >=95% depending how quickly I wanted the results. Obviously >=95% gets slightly improved results but a little slower.

Number of HITS approved: >100 or >500, once again depending on speed vs. quality

Reward per assignment: $0.05 – I experimented with $0.10 but didn’t see any real benefit in speed or quality. I’d rather twice the number of users!

Number of assignments per hit: 200 – this is entirely up to you and your budget and the number of variants you want. The more the better! I’d say ~100 per variant is an absolute minimum. If you’re experiment has subtle changes only between the variants then you’ll need even more. Note that each worker can only do a HIT once.

If you want to read more about your configuration options then look at page 10 of this mTurk guide.

Once you’ve chosen your settings press the ‘Design Layout’ button to proceed. We’ll overwrite the template provided – press the “Edit HTML Source” button you see at the top of the text area. Paste in the code here:



<h3 style="font-weight: normal; ">Pick a Search Result</h3>

<p style="font-weight: normal; ">Imagine you searched for "<b>${term}</b>" in Google.</p>

<p style="font-weight: normal; ">When you click the link you will see (in new window) a list of <i>dummy</i> search results for that search; please click the result that you’d deem most relevant:</p>

<p><b><span class="Apple-style-span" style="font-weight: normal; "><b><a target="_blank" href="${result_page}">${result_page}</a></b></span></b></p>

<p>Please paste the result code on the second screen into the answer box below. Thanks!</p>

<p><textarea name="comment" cols="80" rows="3"></textarea></p>



Click the ‘Edit HTML source’ button again to toggle back to display mode and you should see something like this (click to enlarge):

The ${term} and ${result_page} markers you see will be filled in when you create a task from this template. This makes the template re-usable for multiple SERP Turkey tasks. We ask the user to paste the response code they get back from mTurk as we have to ask them a question (mTurk rules), and this response code encodes whether their click was recorded or not (if they timed out by taking longer than 5 minutes, or fiddled with the page somehow they’ll get a failure code). Only you will know which codes are a success for a test (shown in your SERP Turkey dashboard).

Hit the “Preview and Finish” button and you’re template will be saved and ready to use. Woo!

Creating a SERP Turkey task in mTurk

Now you have a template, you need to create a HIT when you want to run a SERP Turkey test.

Step 1: Create your test in SERP Turkey. If you’re not sure how, then read the SERP Turkey introduction blog post I wrote on SEOmoz.

Step 2: On the SERP Turkey Dashboard page you’ll find a download link for an mTurk input file; download it. This input file will fill in the ${term} and ${result_page} variables we created in our template.

Step 3: Login to mTurk and hit the chubby “Publish” button at the top. Select whichever template you want (you probably just have the one we just created if this is your first time).

Step 4: Upload the input file you downloaded (or select a previously uploaded one).

Step 5: Now you’ll see a preview of how the HIT will look to workers. If everything looks in order press “Next” to see the final screen.

Step 6: You can now give this batch of HITs a name, review how much it is likely to cost etc. If everything looks good then hit “Publish”.

Now you’re test is live and out there in the world!

Monitoring results

You can now monitor (and download) the results in realtime via the SERP Turkey interface. You can also view the “Manage” tab in mTurk to see how many workers have recorded completing the job on your end.

You should periodically review the results in mTurk and approve the workers work, so they get paid (you want to establish a good relationship as a requester so try to pay within a couple of days – you can automate payment in your template settings). If you’re so inclined you can see what response code each worker entered and compare this to the ‘success’ response code in SERP Turkey’s dashboard to see whether their click was counted or not – they are only not counted currently if they took longer than 5 minutes from viewing the SERPs to clicking a result. You should be able to safely ignore this and just pay all the workers.

Cancelling a job

If you’re monitoring the results and realise that the test isn’t going how you planned, there is a problem or you think you already have enough data then you can cancel a job at any time in the Manage page of mTurk.

Outside of the US?

If you are outside of the US and cannot access mTurk directly, worry not. There is another solution: Smartsheet Crowdsourcing actually leverages Amazon’s Mechanical Turk, but you don’t need to be in the US to use it. You do have to pay a $30 monthly subscription but then you can leverage Turk. If someone wants to write a Smartsheet SERP Turkey post I’ll happily add a link in here and on the SERP Turkey page!

Future

IF SERP Turkey proves popular I will look at adding more features, and either way I will soon make it so you can download all the raw click data for your tests to do your own more detailed analysis. I just haven’t had time!

Wrap up

That’s pretty much all there is to it! 🙂

If you have any feedback or suggestions then contact me by email at [email protected] or via twitter at @TomAnthonySEO, or leave a comment here.

Happy Turkeying!