Qualifications as Blocks

An issue that commonly arises on the MTurk developer forum (see here) or StackOverflow (see here) is the issue of how to prevent workers from completing a new HIT if they have completed a HIT in the past. One reason this might come up is that a requester has posted a first HIT and received completed work, then changes the HIT dramatically and reposts it as a new HIT but wants to avoid retakes by workers from the earlier HIT.

How do you achieve this? The MTurk Requester User Interface (prior to July 2014) supplied no mechanism for handling this. Three common strategies in that time were the following:

1. One suggestion from AWS in the past was to block the workers who completed the first HIT (using, for example, BlockWorker) but this has proven to be less than ideal solution. For one, it doesn't prevent workers from seeing the HIT. Instead, when they try to accept the HIT they're told they've been blocked. Technically this should have no impact on them, but the message shown is a bit threatening and typically creates bad reactions on the worker forums, in TurkOpticon ratings, and via email.
2. You can include a note in the HIT that workers retaking the HIT will have their assignments rejected. This puts all of the burden on the worker, who often does not have access to a list of the HITs they've completed in the past. If you choose this option, you need to supply a list of WorkerIds that are not eligible to complete the HIT or supply the worker with some way of checking whether they've previously completed this specific task. The MTurk worker interface supplies no such information.
3. Finally, you can use javascript to "soft block" workers. Essentially, once they accept the HIT, you can use javascript to prevent the worker from seeing the HIT or clicking a link to an off-site page. The relevant javascript code is available here.

However, better than any of these, is to use a Qualification Requirement to only allow new workers to complete the new HIT. There are two ways to do this, one that is a positive requirement (i.e., only workers with the Qualification, which can be granted automatically to new workers, are able to complete the HIT) and one that is negative (i.e., only workers who do not have a Qualification can complete the HIT). I walk through both methods (which are similar) below.

The Positive Qualification Method

1. Create a QualificationType that automatically grants a particular score (e.g., 100)
2. Assign this Qualification to all workers from the first HIT with a score below the automatically granted score.
3. Attach the Qualification as a QualificationRequirement to the new HIT.

With this method, workers will not be able to complete the new HIT and new workers simply need to request the Qualification in order to complete the HIT. The code to achieve this is annotated below.

1. Create New QualificationType

First, you simply need to create a QualificationType using CreateQualificationType:

thenewqual <-
CreateQualificationType(
    name="Auto-Granted Qualification to Prevent Retakes",
    description="This qualification is for people who have worked for me on this task before. It is granted automatically to new workers.",
    status = 'Active',
    keywords="Worked for me before",
    auto = TRUE,
    auto.value = 100)

2. Assign Qualifications

Next, you can use AssignQualification to give all workers from your previous HIT a score below the threshold you'll use on the new HIT. In the previous step we set the Qualification to automatically grant a score of 100. So we'll set a score of 50 here, but you could use anything.

w <- c('Worker1','Worker2','etc.') # a vector containing WorkerIds
AssignQualification(
    qual = thenewqual$QualificationTypeId,
    workers = w,
    value = "50")

This will return a dataframe specifying the scores assigned to each worker, so you don't need to store the result.

3. Create a New HIT with a QualificationRequirement

Finally, when you create the new HIT, you simply need to specify the QualificationType you created as a Qualification Requirement:

# generate a QualificationRequirement object
req <- 
GenerateQualificationRequirement(thenewqual$QualificationTypeId,"==","100")

# create HIT
newhit <- 
CreateHIT(
    title = "Survey",
    description = "5 question survey",
    reward = ".10",
    duration = seconds(days=1),
    keywords = "survey, questionnaire",
    expiration = seconds(days=7),
    assignments = 50,
    hitlayoutid = "23ZGOOGQSCM61T1H5H9U0U00OQWFFU",
    qual.req = req)

The Negative Qualification Method

1. Create a QualificationType with no automatic score or test.
2. Assign this Qualification to all workers from the first HIT with any score value.
3. Attach the Qualification as a QualificationRequirement to the new HIT using the "DoesNotExist" comparator.

With this method, previous workers will not be able to complete the new HIT and new workers simply need to accept the HIT in order to complete it.

1. Create New QualificationType

First, you simply need to create a QualificationType using CreateQualificationType:

thenewqual <-
CreateQualificationType(
    name="Qualification to Prevent Retakes",
    description="This qualification is for people who have worked for me on this task before.",
    status = 'Active',
    keywords="Worked for me before",
    auto = FALSE)

2. Assign Qualifications

Next, you can use AssignQualification to give all workers from your previous HIT any score on the Qualification. The value doesn't matter because we will (later) just check to see if workers have the Qualification at all, but not what their score is.

w <- c('Worker1','Worker2','etc.') # a vector containing WorkerIds
AssignQualification(
    qual = thenewqual$QualificationTypeId,
    workers = w,
    values = "50")

This will return a dataframe specifying the scores assigned to each worker, so you don't need to store the result.

3. Create a New HIT with a QualificationRequirement

Finally, when you create the new HIT, you simply need to specify the QualificationType you created as a Qualification Requirement with the "DoesNotExist" comparator:

# generate a QualificationRequirement object
req <- 
GenerateQualificationRequirement(thenewqual$QualificationTypeId, "DoesNotExist", "")

# create HIT
newhit <- 
CreateHIT(
    title = "Survey",
    description = "5 question survey",
    reward = ".10",
    duration = seconds(days=1),
    keywords = "survey, questionnaire",
    expiration = seconds(days=7),
    assignments = 50,
    hitlayoutid = "23ZGOOGQSCM61T1H5H9U0U00OQWFFU",
    qual.req = req)

Now just wait for responses to roll in without having to worry about workers retaking the HIT.