Amazon Mechanical Turk (MTurk) lets Requesters find Workers to help with tasks of all sizes and shapes. In a previous tutorial, we showed how to create a Project with lots of HITs. Later, we shared a tutorial on how to review the results. Today, we will show a quick way to ask a single, quick question to a large audience of Workers.
In this tutorial, we will learn create a single HIT and ask 1,000 Workers to answer it. This might be useful if you are conducting a study and want several opinions on a single topic.
For our tutorial, imagine you want to research how people commute to work each day. We’ll ask Workers to choose from two answers: car or mass-transit. Let’s get started.
Create a HIT
Sign up and log in to the MTurk Requester Website. Click the “Create” tab and then click on the “Create HITs Individually” in the sub-heading to get started.
Let’s take a moment and walk through each section of the Create page:
1. Describe your HIT
Here you define the basic attributes of your HIT.
- Title and Description: Conveys to Workers what your work is about. The title for our HIT will be “How do you commute to work every day?” and we will add a brief description of the task.
- Keywords: These help Workers looking for certain types of work to discover your HIT. Workers can search by any combination of Title, Description, and Keyword attributes. Selecting thoughtful keywords helps make it easier for you to help Workers discover your work.
- Expires in: This is the length of time your HITs will be available for Workers to discover and complete in the MTurk marketplace. If the HIT isn’t worked on by a Worker by this time, it will be automatically expired. We are on a tight timeline so we will set our HIT to expire in two days.
2. Provide Detailed Instructions
- Instructions: Here we explain to Workers how to best complete our HIT
- Workers will have  to complete this HIT: This is the amount of time a Worker has to complete your HIT after accepting it. If a Worker does not complete the HIT in this amount of time, the HIT is automatically returned and another Worker is eligible to accept and work on it instead. Since our HIT is a quick question, it should take no more than a few seconds to complete. It shouldn’t take an entire hour, but to give Workers the flexibility to take up to an hour to submit their answer to ensure they aren’t rushed.
If your HIT requires viewing an image, listening to a piece of audio, or viewing a video, you can click the button “Add Picture,” “Add Audio,” or “Add Video” to continue. As our HIT does not require any of these options, we will leave this section blank.
3. Select an Answer Format
Our question has two options for Workers to choose between: car and mass-transit, so we will select “Single choice — text”
Now let’s modify the choices Workers can choose between. If your question requires more choices, simply click “Add a choice” to add another selection for Workers to choose from.
4. Pay For Your HIT
Here we set how many Workers we would like to answer our question and how much we plan to pay for their response.
Since are asking 1,000 Workers to answer our question, we set the Assignments to 1,000. We will provide a $0.02 reward for Workers to answer this single question.
Click “Preview Your HIT” to continue.
Preview Your HIT
We now see a preview of HIT. The blue rectangle is how our HIT will appear within the list of available HITs in the MTurk marketplace. When a Worker previews and accepts our HIT, they will see the contents in the black outlined box. This is the individual Assignment of our HIT,
Now, click “Create HIT” to publish the HITs to the MTurk marketplace.
Review Submitted Assignments
A day or two after we published our HIT, we return to gather results. Once again, log in to the MTurk Requester Website. Click the “Manage” tab and then click on the “Manage HITs Individually” in the sub-heading to get started.
We can see our completed HIT in the list.
Click “Review submissions” to view the answers Workers submitted.
Unlike when you create a Project, here all HITs are manually approved or rejected. To approve the submitted Assignment, click the checkbox of the row containing the Assignment you would like to approve. Any assignments that you do not approve or reject will be automatically approved after 30 days.
You may click more than one checkbox at a time. Then click “Submit” at the bottom of the list. This will immediately approve this Assignment and pay the Worker the reward you set when creating your HIT. Please note that the act of approving an Assignment cannot be reversed. In other words, once approved, an Assignment cannot later be “unapproved” or rejected. Rejecting work is handled the same way as approving work.
After approving the Assignments, you can download the results.
To do this, click “Download Results” under the Assignments heading.
Then confirm your download by clicking “Download Results” again to download the CSV file.
Upon opening the CSV file, scroll to the far right to see Workers responses. Notice that MTurk also includes identifiers for the Worker, HIT, and the Assignment in the file for your reference.