AMT vs. Prolific

[Joshua Hartshorne]

For people who have used both, what has your experience been? Do you prefer one to the other? Why?

[Joshua Hartshorne]

Bumping this up.

[Dasa Zeithamova]

I have not used Prolific, but my experience with AMT was poor. It's been a few years but we encountered three issues.

1. Setting up an account turned out more complicated than we anticipated, and included request from mTurk that my lab manager sends a copy of her US passport to prove we are US-based. We have very innocent memory tasks, no sensitive information collected, but somehow got flagged as super-suspicious.

2. The HIT system meant that we had to pay a little to just ask a basic question for eligibility (to match our IRB), before even getting on to experiment. Not sure if it was bots, but those tiny payments ended up adding up, so it was essentially more expensive to get one usable subject online as it was in the lab.

3. We ran the same experiment in parallel in the lab and online. The online performance, even after we excluded those clearly not even trying, was still 1SD below what we were getting in the lab.

I'm sure these don't happen to everyone and there may be "user error" somewhere in there, but the point is it wasn't nearly as straightforward as we hoped it would be.

We are now using Pavlovia and it was way easier to set up and get data comparable to those collected in the lab.

[Paul Compensis]

I have used both for Qualtrics-based acceptability studies (hence surveys). I personally prefer Prolific for a couple of reasons although it is a bit more expensive in comparison to AMT (I also have the impression that rewards that seem to be acceptable for participants on the platforms are a bit higher in Prolific).

1. I often conduct studies on smaller languages and need to control closely for some demographic features. The pre-screening in Prolific is way more elaborated and updated esp. concerning linguistic and psychological dimensions but also more fine-grained socio-economic variables. So it is easier to target realiably a certain sample of people.

2. Handling the organization of the study is also more intuitive in Prolific in my opinion. This might seem to be trivial but in the year 2020 this should be a matter of course:  The user interface in Prolific has a more user-friendly and "beautiful" appearence. In AMT, I always have the impression to work on a 2000s webpage :D Esp. for early stage researchers and grad students this makes it easier to run an experiment.

3. I don't know about AMT but costumer's service/ help desks works really great with Prolific.

4. The target groups differ: People in AMT are used to micro-tasking that can be easily done while waiting for the bus, waiting in a queue etc. This can really be a problem if you have a typical 10 min survey/ experiment. In Prolific, there is much more academic research represented, so participants are used to this (of course, to some extent this brings us back to the classical WEIRD problem and exam-biased participants we have in the labs). But still, response to my studies was much faster in Prolific (30 participants within the first 1-2 hours is quite common, even with my smaller languages) than in AMT.

To be fair, there are some disadvantages I must mention:

1. AMT offers a bigger sample in total (particularly for English-speaking participants). Because of the complex pre-screening it takes longer to register as a worker/ participant in Prolific. So we see a clear quantity/ quality of information trade-off concering the sample.

2. Prolific is more expensive. But for me, the great service and pre-screening partly justifies that.

3. Microtasking can be an advantage! AMT can be great for smaller tasks, e.g. translating experimental stimuli quickly into another language, checking stimuli data by several people beforehand (in the sense of pilot studies), recording a bunch of audio stimuli etc etc.