How to Test Campaigns in Sandbox Mode

[Cody Wild]

Hello there, 

I'm currently trying to get the campaigns feature working for a MTurk app hosted on Heroku, using a local dashboard (with `ad_url_domain` set to my Heroku app link).

It seems like I've created a test campaign in Sandbox mode (see below)

However, I can't figure out how to confirm that it's working properly, in the sense that (1) HITs are actually being created, and (2) they're linking to the right (Heroku) app. 

Should it be the case that, if things are working properly, I would see HITs created in the "HITs" tab of the dashboard? Or are HITs not actually created by Campaigns in Sandbox mode because obviously no workers are actually going to able to gradually complete them? If that's the case, is there any way to make sure that a campaign would create HITs that point to the right place, or would I have to create a campaign in live mode to test that? 

Thanks! 

[Cody Wild]

Ah, gotcha, thanks! I've now successfully gotten HITs to start. One thing I am noticing is that the HITs seem to be created in a slightly confusing-to-me way. I started a campaign requesting 20 total assignments, with 2m between rounds. 

- I got the first HIT created at 3:37, with 9 max assignments 

- The second HIT was created at 3:39, with 9 max assignments (18 total) 

- The third HIT was created at 3:41, with only 1 assignment, getting the total up to 19 

When I check the "Tasks" tab, I keep seeing the "Next Run Time" value pushed back 2 minutes and then another 2 minutes into the future, which makes me think there's still one HIT outstanding for that final assignment, but that it's waiting for some condition to be fulfilled. 

Does this sound like expected behavior? I was wondering if this was some weird quick of running a campaign in a Sandbox setting, and thus a setting where you'd never actually reach the point of any of the assignments being completed. 

On Tue, May 11, 2021 at 3:34 PM Dave Eargle <da...@daveeargle.com> wrote:

No, the hits should actually get created, and you should see the campaign listed as one of your `tasks`. However, if you don't have `do_scheduler` set to true though, the campaign will never run, nor will it appear in your tasks pane. That's a "poor communication bug" that I have on my todo to fix. I'm also going to make it possible to schedule campaigns from any instance, but not run them unless the instance is set to `do_scheduler`.

--
You received this message because you are subscribed to the Google Groups "PsiTurk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psiturk+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/311818c7-15ef-4fd3-bef4-3e0b3b542f0en%40googlegroups.com.

[da...@daveeargle.com]

There's no special logic for sandbox mode -- it sounds like a straight logic error on my part. The logic is here: https://github.com/NYUCCL/psiTurk/blob/b8025ea29c0e853f2f20abfb8ade75a48909e57e/psiturk/tasks.py#L20-L59

[Cody Wild]

I think I'm getting a similar issue to this again, but this time the issue is that too many assignments are being created. I started a HIT at :30, requesting 30 assignments, with a 2m buffer in between. I came back (after a time away from my computer, not monitoring things) to see the following sizes of HITs having been created: 

• :30 9 
• :32 9 
• :34 9 
• :36 3 
• :46 1 
• :48 1 
• :50 3 
• :54 1 
• :00 4 
• :08 1 
• :10 2 

So ultimately creating 43 assignments rather than 30. 

My suspicion is that the issue is coming from here; it only counts "Available" and "Pending" as "maybe will complete," and doesn't seem to count assignments that are in the "Submissions Needing Action" state. 

[Dave Eargle]

Hmm, there isn't actually any state called "submissions needing action" -- that is a psiturk-computed field based on number pending, number available, number completed, and the max for a given hit, see https://github.com/NYUCCL/psiTurk/blob/d56d26c1822599dafc3c79433ac704f0cbc06d80/psiturk/amt_services.py#L161-L165

I'll have to think through the logic, but 43 is way higher than what I would have thought. I have never had a campaign go over. Perhaps it's a clash between your grace time being greater than your round interval?

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/0dd70b79-dae6-48ca-a6be-cf421e513e37n%40googlegroups.com.

[Cody Wild]

Grace time is denoted in seconds, right? I'm pretty sure my grace time is 5 seconds and my round interval time is 2 minutes. 

[Dave Eargle]

Oh, right. Yes, seconds. Still, maybe it's possible that your hits are taking a really long time to run, I guess. The log would tell.

[Cody Wild]

By "run" do you mean "start," or are you talking about the duration of the HIT itself? 

[Cody Wild]

The full log, from when I started the campaign until when I killed it to stop new HITs being added. It seems like things are able to be started successfully meaningfully before the next round begins? 

```

2021-05-26 15:30:21,285 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:32:19 UTC)" (scheduled at 2021-05-26 22:30:19.711634+00:00)
2021-05-26 15:30:23,248 Posting total of 9 assignments this round.
2021-05-26 15:30:23,997 Response(Operation: create_hit | Status: success)
2021-05-26 15:30:23,997 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:32:19 UTC)" executed successfully
2021-05-26 15:32:20,433 Looking for jobs to run
2021-05-26 15:32:21,154 Next wakeup is due at 2021-05-26 22:34:19.711634+00:00 (in 119.277577 seconds)
2021-05-26 15:32:21,155 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:34:19 UTC)" (scheduled at 2021-05-26 22:32:19.711634+00:00)
2021-05-26 15:32:22,089 Posting total of 9 assignments this round.
2021-05-26 15:32:22,801 Response(Operation: create_hit | Status: success)
2021-05-26 15:32:22,802 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:34:19 UTC)" executed successfully
2021-05-26 15:34:20,433 Looking for jobs to run
2021-05-26 15:34:21,163 Next wakeup is due at 2021-05-26 22:36:19.711634+00:00 (in 119.276740 seconds)
2021-05-26 15:34:21,164 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:36:19 UTC)" (scheduled at 2021-05-26 22:34:19.711634+00:00)
2021-05-26 15:34:22,102 Posting total of 9 assignments this round.
2021-05-26 15:34:22,851 Response(Operation: create_hit | Status: success)
2021-05-26 15:34:22,852 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:36:19 UTC)" executed successfully
2021-05-26 15:36:20,442 Looking for jobs to run
2021-05-26 15:36:21,168 Next wakeup is due at 2021-05-26 22:38:19.711634+00:00 (in 119.268615 seconds)
2021-05-26 15:36:21,169 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:38:19 UTC)" (scheduled at 2021-05-26 22:36:19.711634+00:00)
2021-05-26 15:36:22,118 Posting total of 3 assignments this round.
2021-05-26 15:36:22,890 Response(Operation: create_hit | Status: success)
2021-05-26 15:36:22,890 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:38:19 UTC)" executed successfully
2021-05-26 15:38:20,438 Looking for jobs to run
2021-05-26 15:38:21,167 Next wakeup is due at 2021-05-26 22:40:19.711634+00:00 (in 119.272381 seconds)
2021-05-26 15:38:21,168 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:40:19 UTC)" (scheduled at 2021-05-26 22:38:19.711634+00:00)
2021-05-26 15:38:22,130 Posting total of 0 assignments this round.
2021-05-26 15:38:22,130 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:40:19 UTC)" executed successfully
2021-05-26 15:40:20,441 Looking for jobs to run
2021-05-26 15:40:21,164 Next wakeup is due at 2021-05-26 22:42:19.711634+00:00 (in 119.269281 seconds)
2021-05-26 15:40:21,165 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:42:19 UTC)" (scheduled at 2021-05-26 22:40:19.711634+00:00)
2021-05-26 15:40:22,299 Posting total of 0 assignments this round.
2021-05-26 15:40:22,300 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:42:19 UTC)" executed successfully
2021-05-26 15:42:20,435 Looking for jobs to run
2021-05-26 15:42:21,161 Next wakeup is due at 2021-05-26 22:44:19.711634+00:00 (in 119.276042 seconds)
2021-05-26 15:42:21,162 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:44:19 UTC)" (scheduled at 2021-05-26 22:42:19.711634+00:00)
2021-05-26 15:42:22,113 Posting total of 0 assignments this round.
2021-05-26 15:42:22,114 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:44:19 UTC)" executed successfully
2021-05-26 15:44:20,440 Looking for jobs to run
2021-05-26 15:44:21,678 Next wakeup is due at 2021-05-26 22:46:19.711634+00:00 (in 119.268013 seconds)
2021-05-26 15:44:21,679 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:46:19 UTC)" (scheduled at 2021-05-26 22:44:19.711634+00:00)
2021-05-26 15:44:22,713 Posting total of 0 assignments this round.
2021-05-26 15:44:22,713 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:46:19 UTC)" executed successfully
2021-05-26 15:46:20,950 Looking for jobs to run
2021-05-26 15:46:21,677 Next wakeup is due at 2021-05-26 22:48:19.711634+00:00 (in 118.760641 seconds)
2021-05-26 15:46:21,678 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:48:19 UTC)" (scheduled at 2021-05-26 22:46:19.711634+00:00)
2021-05-26 15:46:22,591 Posting total of 1 assignments this round.
2021-05-26 15:46:23,335 Response(Operation: create_hit | Status: success)
2021-05-26 15:46:23,336 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:48:19 UTC)" executed successfully
2021-05-26 15:48:20,441 Looking for jobs to run
2021-05-26 15:48:21,172 Next wakeup is due at 2021-05-26 22:50:19.711634+00:00 (in 119.269349 seconds)
2021-05-26 15:48:21,173 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:50:19 UTC)" (scheduled at 2021-05-26 22:48:19.711634+00:00)
2021-05-26 15:48:22,131 Posting total of 1 assignments this round.
2021-05-26 15:48:22,879 Response(Operation: create_hit | Status: success)
2021-05-26 15:48:22,880 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:50:19 UTC)" executed successfully
2021-05-26 15:50:20,443 Looking for jobs to run
2021-05-26 15:50:21,177 Next wakeup is due at 2021-05-26 22:52:19.711634+00:00 (in 119.266851 seconds)
2021-05-26 15:50:21,178 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:52:19 UTC)" (scheduled at 2021-05-26 22:50:19.711634+00:00)
2021-05-26 15:50:22,250 Posting total of 3 assignments this round.
2021-05-26 15:50:22,983 Response(Operation: create_hit | Status: success)
2021-05-26 15:50:22,984 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:52:19 UTC)" executed successfully
2021-05-26 15:52:20,445 Looking for jobs to run
2021-05-26 15:52:21,176 Next wakeup is due at 2021-05-26 22:54:19.711634+00:00 (in 119.264684 seconds)
2021-05-26 15:52:21,176 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:54:19 UTC)" (scheduled at 2021-05-26 22:52:19.711634+00:00)
2021-05-26 15:52:22,286 Posting total of 0 assignments this round.
2021-05-26 15:52:22,286 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:54:19 UTC)" executed successfully
2021-05-26 15:54:20,442 Looking for jobs to run
2021-05-26 15:54:21,675 Next wakeup is due at 2021-05-26 22:56:19.711634+00:00 (in 119.268088 seconds)
2021-05-26 15:54:21,676 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:56:19 UTC)" (scheduled at 2021-05-26 22:54:19.711634+00:00)
2021-05-26 15:54:22,820 Posting total of 1 assignments this round.
2021-05-26 15:54:23,552 Response(Operation: create_hit | Status: success)
2021-05-26 15:54:23,553 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:56:19 UTC)" executed successfully
2021-05-26 15:56:20,946 Looking for jobs to run
2021-05-26 15:56:21,673 Next wakeup is due at 2021-05-26 22:58:19.711634+00:00 (in 118.764430 seconds)
2021-05-26 15:56:21,674 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:58:19 UTC)" (scheduled at 2021-05-26 22:56:19.711634+00:00)
2021-05-26 15:56:22,792 Posting total of 0 assignments this round.
2021-05-26 15:56:22,793 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:58:19 UTC)" executed successfully
2021-05-26 15:58:20,441 Looking for jobs to run
2021-05-26 15:58:21,671 Next wakeup is due at 2021-05-26 23:00:19.711634+00:00 (in 119.269236 seconds)
2021-05-26 15:58:21,672 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:00:19 UTC)" (scheduled at 2021-05-26 22:58:19.711634+00:00)
2021-05-26 15:58:22,887 Posting total of 0 assignments this round.
2021-05-26 15:58:22,887 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:00:19 UTC)" executed successfully
2021-05-26 16:00:20,942 Looking for jobs to run
2021-05-26 16:00:21,655 Next wakeup is due at 2021-05-26 23:02:19.711634+00:00 (in 118.768259 seconds)
2021-05-26 16:00:21,656 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:02:19 UTC)" (scheduled at 2021-05-26 23:00:19.711634+00:00)
2021-05-26 16:00:23,169 Posting total of 4 assignments this round.
2021-05-26 16:00:23,870 Response(Operation: create_hit | Status: success)
2021-05-26 16:00:23,871 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:02:19 UTC)" executed successfully
2021-05-26 16:02:20,424 Looking for jobs to run
2021-05-26 16:02:21,150 Next wakeup is due at 2021-05-26 23:04:19.711634+00:00 (in 119.286146 seconds)
2021-05-26 16:02:21,151 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:04:19 UTC)" (scheduled at 2021-05-26 23:02:19.711634+00:00)
2021-05-26 16:02:22,231 Posting total of 0 assignments this round.
2021-05-26 16:02:22,231 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:04:19 UTC)" executed successfully
2021-05-26 16:04:20,437 Looking for jobs to run
2021-05-26 16:04:21,161 Next wakeup is due at 2021-05-26 23:06:19.711634+00:00 (in 119.273082 seconds)
2021-05-26 16:04:21,162 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:06:19 UTC)" (scheduled at 2021-05-26 23:04:19.711634+00:00)
2021-05-26 16:04:22,236 Posting total of 0 assignments this round.
2021-05-26 16:04:22,237 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:06:19 UTC)" executed successfully
2021-05-26 16:06:20,436 Looking for jobs to run
2021-05-26 16:06:21,165 Next wakeup is due at 2021-05-26 23:08:19.711634+00:00 (in 119.274568 seconds)
2021-05-26 16:06:21,166 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:08:19 UTC)" (scheduled at 2021-05-26 23:06:19.711634+00:00)
2021-05-26 16:06:22,260 Posting total of 0 assignments this round.
2021-05-26 16:06:22,260 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:08:19 UTC)" executed successfully
2021-05-26 16:08:20,441 Looking for jobs to run
2021-05-26 16:08:21,165 Next wakeup is due at 2021-05-26 23:10:19.711634+00:00 (in 119.269670 seconds)
2021-05-26 16:08:21,165 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:10:19 UTC)" (scheduled at 2021-05-26 23:08:19.711634+00:00)
2021-05-26 16:08:22,407 Posting total of 1 assignments this round.
2021-05-26 16:08:23,115 Response(Operation: create_hit | Status: success)
2021-05-26 16:08:23,116 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:10:19 UTC)" executed successfully
2021-05-26 16:10:20,435 Looking for jobs to run
2021-05-26 16:10:21,158 Next wakeup is due at 2021-05-26 23:12:19.711634+00:00 (in 119.274848 seconds)
2021-05-26 16:10:21,159 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:12:19 UTC)" (scheduled at 2021-05-26 23:10:19.711634+00:00)
2021-05-26 16:10:22,265 Posting total of 2 assignments this round.
2021-05-26 16:10:22,980 Response(Operation: create_hit | Status: success)
2021-05-26 16:10:22,981 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 23:12:19 UTC)" executed successfully
2021-05-26 16:11:12,312 Removed job campaign-4
``` 

[Dave Eargle]

Did your hits have a really short lifetime? I extracted the posting rounds below. Were there 43 assignments available at the same time? I see that it posted 30 in the first four rounds, then nothing for the next 4 rounds, then 1, then 1, then 3... this is possible without there being a bug if not all of the original four HITs completed your task by the time the hit expired (in which case, they wouldn't be able to ever submit to mturk, and therefore would never be counted as completed). Then the campaign would continue to post new rounds until the completed number got up to 30. I see at the bottom of your log that eventually it decided that it had completed.

However! With the way the task counts how many have completed the task, if they weren't able to actually submit to mturk even though psiturk thought they did, then the count would be off. Let's see... that would overcount the "number completed", and post not enough assignments, which isn't the problem here.

Another possibility is that there's some kind of lag in mturk's reporting of how many assignments are available / pending for a given HIT. I know there is a lag for other operations, like approving workers, which is a huge pain that I haven't tracked down. But we could easily add more logging to the task to see how many it thinks are outstanding for any given round.

The order from your log below:
9
9
9
3
0
0
0
0
1
1
3
0
1
0
0
4
0
0
0
1
2

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/5e226f34-dcf3-4e96-9f04-e2ea3e782763n%40googlegroups.com.

[Dave Eargle]

One more thing -- this is really stupid, but mturk counts an assignment as "pending" even if someone is only previewing it. But we have to rely on the pending count to know whether any workers have begun the task or might begin the task. If 9 people are previewing a HIT with 9 assignments, then the task is unassignable even if none of them end up even starting the task. In this case, the HIT's assignments become assignable again as soon as they stop previewing it. So yeah, what did your count of completed workers look like?

[Cody Wild]

> I see at the bottom of your log that eventually it decided that it had completed.

That was me manually stopping the campaign because I was worried about it adding too much financial liability of workers who had started tasks; it didn't get to the point of terminating itself. 

> I see that it posted 30 in the first four rounds, then nothing for the next 4 rounds, then 1, then 1, then 3... this is possible without there being a bug if not all of the original four HITs completed your task by the time the hit expired 

The hit duration was 30m, but had a normal/default expiration window of 24hrs, so I don't think the HIT could have expired (other than me expiring it manually) during the time new HITs were being created. 

> Another possibility is that there's some kind of lag in mturk's reporting of how many assignments are available / pending for a given HIT. I know there is a lag for other operations, like approving workers, which is a huge pain that I haven't tracked down. But we could easily add more logging to the task to see how many it thinks are outstanding for any given round.

Yeah, that seems worth looking into! 

[Dave Eargle]

Do you remember how many had "completed" and how many the dashboard reported were "available" by the time you cancelled the campaign?

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/1c5a1519-020b-40f4-b28a-8d250b782651n%40googlegroups.com.

[Dave Eargle]

Also, what is your intention in only a 2 minute gap between rounds, with only 9 per round? I use the spacing to do things like spread out rounds of 20, one per hour, to get workers at different times of day, to improve generalizability of findings. But what does 2 minutes do -- why not just post 30 per round, with something like a 1 hour gap or a 24 hour gap (since your HIT is 30 minutes)? That way, if one of your early hits expired before all workers took it, the campaign would post another round of (the number missing).

If you're thinking that the max a round can have is 9, that's not correct -- the posting task will post several hits with up to 9 each until the round goal is met.

[Cody Wild]

Oh, yeah, I'd assumed that a round was equivalent to starting a HIT, and so selecting > 9 would induce the higher cost. 

I'd also assumed from the structure of there being different rounds that it was recommended to have some spacing between them, so I picked a spacing that was small but enough that there hopefully wouldn't be errors due to the server being hit too much. 

[Dave Eargle]

Hmm my thought for the server being hit too much would be because of workers coming in in droves, not for the posting of the hits. I could describe all this better in the docs.

To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/0f367282-f5f5-403e-83e6-840c85e9ad42n%40googlegroups.com.

[Cody Wild]

Yeah, tbc, I'm not making an argument that I think this is a relevant server concern, I was just Chesterton's-Fence-assuming it was because I misunderstood the purpose of rounds