MTurk is a great way to get fast, reliable responses to your survey or study. Today, we’re going to talk about how to manage survey results hosted on Qualtrics. You can also host your survey on directly on MTurk.

To create an account on MTurk visit requester.mturk.com.

Getting Started

To post a survey hosted on another service such as Qualtrics on MTurk, you’ll want to start from the Survey Link template. This enables you to post a task (HIT) on MTurk that directs Workers to your survey.

You will be prompted to provide a name, description, and keywords for your survey, be as descriptive as possible and include an estimate of how long your survey will take. You can then specify how much you will pay for each response and the number of respondents (Workers) you want to participate in your survey.

Select Auto-Approval settings for your Project. This value tells MTurk how long to wait before automatically approving a Worker’s response. If you set this value too short, you may not leave yourself enough time to fully review Workers’ responses before they are automatically approved. If you set this value too long, Workers may become frustrated at the delay in receiving payment upon completion of the survey. Customers conducting surveys tend to find that three days is a good balance, but please adjust to suit your survey requirements.

Tracking responses

If you’re using the Survey Link template and hosting your survey on another website such as Qualtrics, SurveyMonkey, or your own website, you’ll need a way to match responses to the Worker information in MTurk. You can do this by either by providing Workers with a unique completion code or collecting a Worker’s ID in the survey itself. If your survey tool allows you to provide a unique completion code when complete you can then direct Workers to provide it in the MTurk task when they are done. You can then link the survey response to the Worker in your results. Instructions for doing this with Qualtrics are below.

The other option is include a question in your survey that asks Workers for their Worker ID as part of the survey. Workers can generally find this on their Dashboard or in the upper left corner of the Worker website. When you get your results you’ll be able to match the Worker IDs. Note that it is a violation of MTurk policies to ask for a Worker’s email address.

Assigning Random IDs to Respondents

You may need to create a random ID for each of your respondents. This is useful if you are running a study with MTurk where you ask respondents to prove their completion with an ID. To generate a Random ID there a few steps. First select Survey Options in the menu for your survey.

Generating a Completion Code in Qualtrics

To generate a unique completion code there a few steps. First select Survey Options in the menu for your survey.

Click Add a New Element Here and select Embedded Data, then name the Embedded Data Random ID.

Click Set a value and click the blue dropdown arrow and highlight Insert Piped Text, then highlight Random Number. Type your minimum and then your maximum. Both should be the same number of digits, (for example, for a 5-digit ID, your minimum is 10000 and your maximum is 99999). Click Insert. Then, click and hold down Move to drag the Embedded Data to the top of the Survey Flow.

Finally, click Save Flow. Note that a number randomly generated using piped text will not automatically be saved in your data. To capture the random number in your data, you need to set up an Embedded Data field in your Survey Flow where the value is set equal to the random number generator as outlined in the steps above.

Then, return to your survey editor and pipe the Embedded Data into your question to display it to the respondent as an MTurk Completion Code.

In order to Display the Embedded Data in a question click the question to edit it. Click Piped Text then highlight Embedded Data.

It is important you select Embedded Data. If you select Random Number instead, you will create a NEW random number that will not be saved with the response data. Type in the name of the Embedded Data that holds your IDs. This is the name saved in the Survey Flow (Random ID). Click Insert.

When respondents complete the survey, they will see their Random ID.

We recommend that you provide instructions on what the respondent needs to do with the ID. Because the survey is ending on a Descriptive Text question, we also want them to click the Next button, or their response may not be submitted.

Test!

You’ve completed all the steps, but just to be safe before putting it in front of Workers, let’s try publishing it to just a few Workers first. This will give you a chance to take a look at the responses and make sure you’re getting what you’re looking for from your study and validate that your embedded data is populating as expected.

Checking results

At any point after you publish your survey you can start checking your responses. If your survey is hosted entirely on MTurk, you can find all of your responses by viewing and downloading your Results on the Manage tab. If you hosted your survey elsewhere, you’ll need to download them from there and then match them up with your MTurk results. This can be done in Microsoft Excel or Google Sheets by using the VLOOKUP() function and matching on either a completion code or Worker ID.

VLOOKUP() allows you to look up a value in one worksheet from another worksheet. To get started, download your results as CSV (comma-separated value) files from both MTurk and Qualtrics and open them in Excel. If you provided each Worker with a unique ID that they entered in their MTurk task you will typically find it in column AB of your MTurk results and the WorkerID will be in column P. To link the WorkerID into your survey results file, first copy column P to column AE so that it’s to the right of the surveycode.

Now you can add a WorkerID column to your survey results. Add a new column heading called WorkerID and type =VLOOKUP( and you will be prompted to add four attributes. The first should be a cell in the same row in a column containing your survey code. The second should be a reference to the columns in your MTurk results file containing the surveycode and WorkerId, it this case they are AB:AE. The third value should be the number of columns between the surveycode and WorkerId including the columns themselves, in this case that’s 4. The final attribute should simply be FALSE. The result will look something like this:

=VLOOKUP(R2,Batch_2801044_batch_results.csv!$AB:$AE,4,FALSE)

Approving work

On the results tab in MTurk you’ll be able to see the responses for each Worker and Approve their work so that they get paid for completing it. Prompt approval is a great way to build trust with the Worker community and maintain a positive reputation.

Those are the steps to manage your survey responses with Qualtrics. We hope you found it useful.

If you have any questions, please post a question to our MTurk forums. To become a Requester, sign up here. Want to contribute as a Worker customer? Get started here.