Skip to content
This repository has been archived by the owner on May 24, 2019. It is now read-only.

Surveys

Jump to bottom
Thomas J. Leeper edited this page Jun 11, 2014 · 11 revisions

One of the most common social science applications of MTurk is as subject recruitment for survey (experimental) research. While common, the integration of MTurk and a survey questionnaire presents a number of challenges, which this tutorial aims to address.

How to write questionnaires

There are (at least) three different ways to write questionnaires in MTurk: natively in the MTurk application, in an off-site tool linked from an MTurk HIT, or in an off-site tool as an ExternalQuestion HIT.

Questionnaires natively in MTurk

One can write questionnaires natively in MTurk using either the proprietary QuestionForm markup or as standard XHTML.

"Link and code" method

A much more common approach, and certainly the easiest of the three methods, is simply to write a basic HIT in HTML that includes a link to the off-site survey and a single text form field for the worker to submit a completion code that they retrieve from the end of the survey.

The HTML can be setup using the "Survey Link" project example or by simply creating an HTML <form> block such as the following:

<h3>Study title</h3>

<p>Description of study</p>

<p>Survey link: <a href="http://www.linktomysurvey.com" target="_blank">http://www.linktomysurvey.com</a></p>

<p>Provide the survey code here: <input id="code" name="code" size="10" type="text" /></p>

The link redirects the worker to the survey in a new window or tab. When they've completed the survey, the survey tool should provide a code that they can copy and paste into the form field.

The easiest way to setup this HIT is to create the survey link page in the Requester User Interface and then pull the HITLayoutId for that setup (the code can be retrieved by clicking on a project name on this page). That code can then be used directly and simply in MTurkR:

newhittype <- 
RegisterHITType(title="10 Question Survey",
                description="Complete a 10-question survey about news coverage and your opinions",
                reward=".20", 
                duration=seconds(hours=1), 
                keywords="survey, questionnaire, politics")
CreateHIT(hit.type=newhittype,
          hitlayoutid="23ZGOOGQSCM61T1H5H9U0U00OQWFFU")

ExternalQuestion method

The link and code method is easiest to setup, but invites problems with workers entering fake codes, workers completing the survey but not entering the code (and thus not getting paid), and other human error possibilities. By contrast, the ExternalQuestion method provides a seamless link between the MTurk interface and an off-site tool. Instead of showing the worker an HTML page with a link, configuring an ExternalQuestion actually shows the worker the survey directly inside an <iframe> in the MTurk interface. This eliminates the need for the requester to create codes and for workers to copy them. But, it requires a somewhat sophisticated survey tool (e.g., Qualtrics) that can handle redirecting the worker back to the ExternalQuestion submit URL for their assignment.

The submit URL has to take the form of:

https://www.mturk.com/mturk/externalSubmit?assignmentId=workerspecificassignmentid&foo=bar

The foo=bar part is required by MTurk (it requires the assignmentId and at least one other field).

Note: The above link is for the live server. For testing in the requester sandbox, the base URL is:

https://workersandbox.mturk.com/mturk/externalSubmit

Creating the HIT is quite simple. It simply requires registering a HITType, building the ExternalQuestion structure (in which you can specify the height of the <iframe> in pixels), and using it in CreateHIT:

newhittype <- 
RegisterHITType(title="10 Question Survey",
                description="Complete a 10-question survey about news coverage and your opinions",
                reward=".20", 
                duration=seconds(hours=1), 
                keywords="survey, questionnaire, politics")
eq <- GenerateExternalQuestion(url="https://www.example.com/myhit.html",frame.height="400")
CreateHIT(hit.type=newhittype, question=eq$string)

Some important notes:

  1. The site you're hosting the question on should be able to serve HTTPS, otherwise some browsers will treat the MTurk <iframe> as unsecure content.
  2. On the first page of your survey, you may want to remind workers to accept the HIT before completing the survey. With an ExternalQuestion, workers can preview the entire survey before accepting the HIT (indeed, they can complete the entire survey) but will be prevented from submitting the HIT back to MTurk. Reminding them to accept the HIT will save requester and workers some unnecessary hassle.

Survey Software Tips

Below are some tips for using different online survey tools with MTurk.

Qualtrics

Qualtrics is a power survey (and experimental) tool and I highly recommend it over free alternatives if using MTurk for any kind of large project. Of course, its features are pricey and it is not reasonable to buy an individual license. Institutional subscriptions, however, are relatively affordable.

Qualtrics can be used with either the "link and code" method or the ExternalQuestion method, but the ExternalQuestion method will provide the most seamless experience for the worker.

This requires:

  1. Create the Qualtrics survey
  2. Setup the Qualtrics survey to extract embedded URL parameters (see documentation), which MTurk will attach to the Qualtrics survey link. You need to extract the assignmentId field. It is also helpful to extract the workerId field to embed it in the survey dataset as a unique identifier for each respondent.
  3. Setup the Qualtrics survey to redirect (at completion) to the ExternalQuestion submit URL, using the assignmentId (the embedded data field) and at least one other survey field. See Qualtrics documentation for details. For the live server, this URL should look like: https://www.mturk.com/mturk/externalSubmit?assignmentId=${e://Field/assignmentId}&foo=bar and for the sandbox it should look like: https://workersandbox.mturk.com/mturk/externalSubmit?assignmentId=${e://Field/assignmentId}&foo=bar
  4. Create an ExternalQuestion HIT via MTurkR using the survey link from Qualtrics, as above.

This tutorial is a nice demonstration of (some of) this workflow.

Survey Monkey

It is not possible to use the ExternalQuestion with Survey Monkey because it is not capable of redirecting the worker to a custom URL, but even basic Survey Monkey plans allow you to pass a "custom data field" that can be used to match completions in MTurk to survey responses. With platinum and enterprise plans, multiple fields can be sent from MTurk into the survey instrument (e.g., to track assignmentId, workerId, hitId, and possibly other variables such as randomized content shown to the worker in the MTurk HIT interface).

The use of the custom data field is described in the Survey Monkey documentation here and the use of more advanced custom variables for platinum accounts is described here.

In short, the survey link can extract the workerId (or assignmentId) from an MTurk assignment and pass that as a custom field. The HTML and JavaScript necessary to do this is shown in detail here: https://github.com/leeper/MTurkR/blob/gh-pages/javascript/mturk-urlpipe.html. A minimum example of the relevant javascript to extract URL parameters in MTurk and attach them to a survey link is as follows:

function turkGetParam( name ) { 
  var regexS = "[\?&]"+name+"=([^&#]*)"; 
  var regex = new RegExp( regexS ); 
  var tmpURL = fullurl; 
  var results = regex.exec( tmpURL ); 
  if( results == null ) { 
    return ""; 
  } else { 
    return results[1];    
  } 
}

var surveylink = new String("http://www.example.com/?" + "assignmentId=" + assign + "&hitId=" + hit + "&workerId=" + worker);

/* the `surveylink` javascript variable can then be displayed to the worker using any javascript method */

Google Forms

Google Forms are one of the easiest and cheapest ways to create a simple survey. As with Survey Monkey, however, they cannot redirect to a custom URL and therefore cannot be used for ExternalQuestion HITs. But, you can configure pre-populated form fields based on the URL of the form, which will allow you to pre-fill the WorkerId and/or other fields in the survey form for purposes of tracking completed assignments. To do this:

  1. Create your form
  2. Follow the directions [in the Google Forms documentation for creating pre-populated form fields](pre-populated form fields). You'll have to supply a dummy WorkerId (and any other fields) in order to generate the pre-filled URL.
  3. Copy the pre-filled URL from Step 2. Remove the dummy WorkerId (and any other field values).
  4. Plug the pre-filled URL into the javascript code shown immediately above (in the Survey Monkey example), allowing WorkerId (and any other fields) to be plugged in as appropriate.

This will create an automatically generated WorkerId field in your form, but you may still want to supply completion codes to workers to enter back into the HIT.

.

Clone this wiki locally