Today, we’re going to talk about how to manage the results of surveys you host on Qualtrics. You can also host your survey on MTurk itself but some customers choose to use Qualtrics, SurveyMonkey, or a number of other survey tools they may be more familiar with.

Getting Started

To post a survey on MTurk that is hosted on another service such as Qualtrics you’ll want to start from the Survey Link template. That will let you post a task on MTurk that directs Workers to your survey.

On the first screen of this template you will be prompted to provide a name, description, and keywords for your survey. It’s always best to be descriptive as possible and include an estimate of how long your survey will take. You can then specify how much you would like to pay Workers for each response and the number of Workers you want to participate in your survey. This is described as Assignments in MTurk.

Make sure you note the Auto-Approval settings for your Project. This tells MTurk how long to wait before automatically approving a Worker’s response if you don’t do it yourself first. If you set this value to be too brief, it is possible that you may not leave yourself enough time to fully review Workers’ responses before they are automatically approved. If you set this value to be too long, Workers may become frustrated at the delay to receive their payment for completing your HIT. Customers performing surveys tend to find that 3 days is a good balance, but please adjust to suit your situation.

Tracking responses

If you’re using the Survey Link template and hosting your survey on another website such as Qualtrics, SurveyMonkey, or your own website, you’ll need to have some way to match responses to the Worker information in MTurk. You can do this by either by providing Workers with a unique completion code or collecting a Worker’s ID in the survey itself. If your survey tool allows you to provide a unique completion code when complete you can then direct Workers to provide it in the MTurk task when they are done. You can then link the survey response to the Worker in your results. Instructions for doing this with Qualtrics are below.

The other option is include a question in your survey that asks Workers for their Worker ID as part of the survey. Workers can generally find this on their Dashboard or in the upper left corner of the new Worker website. When you get your results you’ll be able to match the Worker IDs. Note that it is a violation of MTurk policies to ask for a Worker’s email address.

Generating a Completion Code in Qualtrics

To generate a unique completion code there a few steps. First select Survey Options in the menu for your survey:

Change your Survey Termination from Default to Custom and create a new message option in your Library called MTurk Completion Code:

The body of this message should be something similar to the following:

Thank you for completing our survey.
Your response has been recorded.

Your MTurk completion code is:
${e://Field/MTurkCode}

Next go to the Survey Flow tab and add a new element:

Choose Web Service and enter:

http://reporting.qualtrics.com/projects/randomNumGen.php

Select Test URL and when prompted select the check box in front of randomand click Add Embedded Data. Click Add parameter to web service and add two values, a min of 0 and a max of 99999999. This will tell the service to generate a number between 0 and 9,999,999,999 so it’s unlikely that two Workers will get the same code. You can now change the name of your new field to MTurkCode. It’s important that this name match the field name you included in the completion message defined earlier. After this is done, it should look something like this:

You can now Preview your survey to confirm that it generates a unique code each time it’s completed. Visit the Data & Analysis tab for your survey and select Export to download a CSV file containing the responses and the random code that were generated for each.

Test!

You’ve completed all the steps, but just to be safe before putting it in front of Workers, let’s try publishing it to just a few Workers first. This will give you a chance to take a look at the responses and make sure you’re getting what you’re looking for from your study. After all, nobody wants to spend hundreds or thousands of dollars on a study only to discover they made minor mistake and will need to run it again.

Checking results

At any point after you publish your survey you can start checking your responses. If you hosted your survey entirely on MTurk, you can find all of your responses by viewing and downloading your Results on the Manage tab. If you hosted your survey elsewhere, you’ll need to download them from there and then match them up with your MTurk results. This can be done in Microsoft Excel or Google Sheets by using the VLOOKUP() function and matching on either a completion code or Worker ID.

VLOOKUP() allows you to look up a value in one worksheet from another worksheet. To get started, download your results as CSV (comma-seperated value) files from both MTurk and Qualtrics and open them in Excel. If you provided each Worker with a unique ID that they entered in their MTurk task you will typically find it in column AB of your MTurk results and the WorkerID will be in column P. To link the WorkerID into your survey results file, first copy column P to column AE so that it’s the the right of the surveycode.

Now you can add a WorkerID column to your survey results. Add a new column heading called WorkerID and type =VLOOKUP( and you will be prompted to add four attributes. The first should be a cell in the same row in a column containing your survey code. The second should be a reference to the columns in your MTurk results file containing the surveycode and WorkerId, it this case they are AB:AE. The third value should be the number of columns between the surveycode and WorkerId including the columns themselves, in this case that’s 4. The final attribute should simply be FALSE. The result will look something like this:

=VLOOKUP(R2,Batch_2801044_batch_results.csv!$AB:$AE,4,FALSE)

Approving work

On the results tab in MTurk you’ll be able to see the responses for each Worker and Approve their work so that they get paid for completing it. Prompt approval is a great way to build trust with the Worker community and maintain a positive reputation.

Those are the steps to manage your survey responses with Qualtrics. We hope you found it useful.

This tutorial was originally published on Happenings at MTurk.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses XenWord.