MTurk workers who did not finish the experiment get paid a month after the HIT

[Geoffrey Castillo]

Hi all,

We're seeing weird payment in our MTurk transactions.

A month after we pay a HIT, some extra MTurkers receive their show-up fee.

Some of them appear in our data and they did not finish the experiment.

The others are completely unaccounted for.

In the `settings.py` file the expiration is set to `'expiration_hours': 7 * 24`, so I don't think that's the problem.

We're wondering if this has something to do with people accepting the HIT on MTurk without completing the work in oTree, and getting paid automatically a month later by MTurk.

Did that also happen to you? Are we missing something?

Best,

Geoffrey

[Chris @ oTree]

I found this in the MTurk docs. It says HITs are auto-approved after 30 days: https://www.mturk.com/help#work_submitted_by_workers

Maybe someone submitted after you approved all assignments, but before the HIT expired? (Note, you can manually expire a HIT through the oTree admin interface.)

[Geoffrey Castillo]

I see, but then isn't it a potentially big problem with the way oTree handles HITs now?

Say I choose a really short expiration time and I expire the HIT as soon as I have the number of people I want.

However it's always possible that someone submitted the HIT on MTurk without ever starting the task on oTree (this could even be the very first person to see the HIT).

Since they didn't reach oTree I can't reject them because they are not recorded, but because on MTurk side they've taken a spot and submitted there they'll get paid no matter what.

The problem is not so small either; on a 30-subjects HIT we see about 5 people got automatically paid a month later.

I'm surprised no one realised before?

[Geoffrey Castillo]

I think the only real way to prevent that is to use completion codes.

We didn't use completion codes because I was left with the impression after reading the doc (https://otree.readthedocs.io/en/latest/mturk.html#making-your-session-work-on-mturk) that it wasn't really needed:

"On the last page of your study, give the user a completion code. For example, you can simply display: “You have completed the study. Your completion code is TRUST2020.” If you like, you can generate unique completion codes. You don’t need to worry too much about completion codes, because oTree tracks each worker by their MTurk ID and displays that in the admin interface and shows whether they arrived on the last page. The completion code is just an extra layer of verification, and it gives workers a specific objective which they are used to having."

but completion codes are actually absolutely necessary if you don't want MTurkers to submit without doing anything and be paid automatically a month later.

[Chris @ oTree]

Hi, I think I see what you are saying. I will take a look at this ASAP. (Although I don't think a completion code would help with this, since people could still enter a bogus code.)

Thanks for raising this.

[Chris @ oTree]

Hi, you are right about this. I was not aware of this auto-approval rule, and I can't find any way to disable auto-approval. But I can fix it by auto-rejecting workers who submit the assignment without even opening the oTree link. Anyway I have coded a fix to this locally. Each time the payments tab is loaded, it checks for any submissions that are not found in the oTree database and rejects them. I will release soon unless anyone has an alternate suggestion.

Thank you for raising this issue! That was a good catch.

[Geoffrey Castillo]

Hi Chris,

Thanks a lot for looking into this!

It's surprising that no one realised before!

I think the solution you propose would work fine and would automatise a lot of the process.

Another solution, probably the simplest, would be that the 'submit HIT' button on MTurk only works if the completion code is correct---but I don't think that's possible.

Another possibility (and it's something that's been annoying us a bit) is to have the HITs created by oTree appear on the Requester interface of MTurk.

Right now when we publish a HIT through oTree the HIT is nowhere to be seen. It only appears in the transactions.

If it were to appear in the interface, we could at least check by hand who really started the experiment and reject them through the interface.

I imagine oTree HITs do not appear on the Requester interface because they've been created through the API, but if there was a way to making them appear we'd super happy because sometimes it's a bit confusing---you have to completely trust oTree that the HIT is really there and you cannot check on MTurk.

[Chris @ oTree]

OK good I will publish the fix soon.

On Wednesday, July 22, 2020 at 10:22:45 PM UTC+8 Geoffrey Castillo wrote:

Another solution, probably the simplest, would be that the 'submit HIT' button on MTurk only works if the completion code is correct---but I don't think that's possible.

Unfortunately this would be tricky to arrange, due to technical limitations (CORS restrictions, etc).

Another possibility (and it's something that's been annoying us a bit) is to have the HITs created by oTree appear on the Requester interface of MTurk.

Right now when we publish a HIT through oTree the HIT is nowhere to be seen. It only appears in the transactions.

If it were to appear in the interface, we could at least check by hand who really started the experiment and reject them through the interface. 

I imagine oTree HITs do not appear on the Requester interface because they've been created through the API, but if there was a way to making them appear we'd super happy because sometimes it's a bit confusing---you have to completely trust oTree that the HIT is really there and you cannot check on MTurk.

Agreed it's unfortunate. MTurk removed HITs from their UI a while ago. But there is a tool called https://manage-hits-individually.s3.amazonaws.com/v4.0/index.html#/credentials that someone created to replace it. Currently, MTurk UI only shows batches. There is no way to create a batch through the API. 

[Geoffrey Castillo]

Perfect!

I'd maybe make it a button rather than rejecting people when the payment tab is loaded.

I can see the following happening: some MTurker submits the HIT on MTurk with the intention of doing the work just a bit later; if we reload the payment tab in the meantime they'd be rejected, even if they actually wanted to do it.

Although maybe this is a very special case...

[Chris @ oTree]

Yes I agree it feels a bit bold to auto-reject assignments. But I think it is the worker's responsibility not to submit an assignment until they have done the work. Especially since the default UI template has a required completion code. They would need to intentionally type in a completion code in order to submit.

Currently my implementation is quite simple, just a few lines of code. Even just adding a button would actually require more rearrangement of the code in a few places, and I don't want to do that unless there is a compelling use case. So my bias is to first try this out and see if people have issues with it.

[Chris @ oTree]

Also, we show the assignment IDs that got rejected, so if the experimenter wants, they can undo the rejection (using MTurk CLI or other tools).

[Geoffrey Castillo]

Yeah. Then it's fine!

On a related note, is there a way to check the completion code MTurkers submit?

[Chris @ oTree]

The completion codes will show up on the payments page.

[Matthew Walker]

I would second this - if workers intentionally submit a false completion code after accepting the HIT but before completing the work I think that this is valid grounds for rejection.

Matt 

[Chris @ oTree]

Thank you. And by the way, the rejection message says that it was auto-rejected and the reason why. By the way this is now available in the newest oTree version.

[Daniel Frey]

Hi Chris

One question related to this topic: Are the assignment-IDs of auto-rejected HITs stored somewhere? 

The auto-rejected assignment-IDs were shown the first time I opened the payments-tab, but unfortunately I did not save them manually...

A worker asked if it's possible to take away the rejection for their approval-score.