Qualifications as Blocks

An issue that commonly arises on the MTurk developer forum (see here) or StackOverflow (see here) is the issue of how to prevent workers from completing a new HIT if they have completed a HIT in the past. One reason this might come up is that a requester has posted a first HIT and received completed work, then changes in the HIT dramatically and reposts it as a new HIT.

How do you achieve this? Well, the MTurk Requester User Interface supplies no mechanism for handling this. A common suggestion from AWS in the past was to block the workers who completed the first HIT (using, for example, BlockWorker) but this has proven to be less than ideal solution. For one, it doesn't prevent workers from seeing the HIT. Instead, when they try to accept the HIT they're told they've been blocked. Technically this should have no impact on them, but the message shown is a bit threatening and typically creates bad reactions on the worker forums, in TurkOpticon ratings, and via email.

The current best practice is to instead use a Qualification Requirement to only allow new workers to complete the new HIT. This technically requires several steps:

1. Create a QualificationType that automatically grants a particular score (e.g., 100)
2. Assign this Qualification to all workers from the first HIT with a score below the automatically granted score.
3. Attach the Qualification as a QualificationRequirement to the new HIT.

With this method, workers will not be able to complete the new HIT and new workers simply need to request the Qualification in order to complete the HIT. The code to achieve this is annotated below.

1. Create New QualificationType

First, you simply need to create a QualificationType using CreateQualificationType:

thenewqual <-
CreateQualificationType(
    name="Auto-Granted Qualification to Prevent Retakes",
    description="This qualification is for people who have worked for me on this task before. It is granted automatically to new workers.",
    keywords="Worked for me before",
    auto = TRUE,
    auto.value = 100)

2. Assign Qualifications

Next, you can use AssignQualification to give all workers from your previous HIT a score below the threshold you'll use on the new HIT. In the previous step we set the Qualification to automatically grant a score of 100. So we'll set a score of 50 here, but you could use anything.

w <- c('Worker1','Worker2','etc.') # a vector containing WorkerIds
AssignQualification(
    qual = thenewqual$QualificationTypeId,
    workers = w,
    values = "50")

This will return a dataframe specifying the scores assigned to each worker, so you don't need to store the result.

3. Create a New HIT with a QualificationRequirement

Finally, when you create the new HIT, you simply need to specify the QualificationType you created as a Qualification Requirement:

# generate a QualificationRequirement object
req <- 
GenerateQualificationRequirement(thenewqual$QualificationTypeId,"==","100")

# create HIT
newhit <- 
CreateHIT(
    title="Survey",
    description="5 question survey",
    reward=".10",
    duration=seconds(days=4),
    keywords="survey, questionnaire",
    hitlayoutid = "23ZGOOGQSCM61T1H5H9U0U00OQWFFU",
    qual.req = req)

Now just wait for responses to roll in without having to worry about workers retaking the HIT.