Possible Campaigns Race Condition

[Cody Wild]

 I'm trying to run a campaign in live mode, and it seems to not be able to start any HITs (after 8 minutes, no HITs have been created per the Dashboard)

Starting a campaign with a 2 minute interval between HITs, server.log for relevant time period: 

```

2021-05-26 15:00:32,055 Added job "do_campaign_round" to job store "default"
2021-05-26 15:00:32,137 Looking for jobs to run
2021-05-26 15:00:32,866 Next wakeup is due at 2021-05-26 22:02:31.273320+00:00 (in 119.135006 seconds)
2021-05-26 15:00:32,867 Run time of job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:02:31 UTC)" was missed by 0:00:01.594264
2021-05-26 15:02:32,003 Looking for jobs to run
2021-05-26 15:02:32,730 Next wakeup is due at 2021-05-26 22:04:31.273320+00:00 (in 119.268667 seconds)
2021-05-26 15:02:32,731 Run time of job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:04:31 UTC)" was missed by 0:00:01.457810
2021-05-26 15:04:32,000 Looking for jobs to run
2021-05-26 15:04:32,717 Next wakeup is due at 2021-05-26 22:06:31.273320+00:00 (in 119.272278 seconds)
2021-05-26 15:04:32,718 Run time of job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:06:31 UTC)" was missed by 0:00:01.444992

2021-05-26 15:06:31,991 Looking for jobs to run
2021-05-26 15:06:32,712 Next wakeup is due at 2021-05-26 22:08:31.273320+00:00 (in 119.281770 seconds)
2021-05-26 15:06:32,712 Run time of job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:08:31 UTC)" was missed by 0:00:01.439598

```

It seems like maybe the wakeup and runtime code is in some sort of race condition, where we don't wake up until slightly after the desired runtime, and then fall back to trying again 2m later, etc?

[Cody Wild]

From this, it looks like we might want to set the `misfire_grace_time` to be larger, but I'm not sure how exactly to do that within Psiturk. 

https://stackoverflow.com/questions/30235604/next-run-time-missed-by-some-seconds-in-job-scheduling-in-apscheuler

[Cody Wild]

My actual server and experiment logic is running on Heroku, but `/dashboard` doesn't seem to work on Heroku, so I'm using a local dashboard on a local Psiturk server to create HITs that direct people to a remote experiment server, by setting `ad_url_domain` and `database_url` accordingly. This seems to have worked for the creation of a single HIT - I created the HIT via the local server, and was able to monitor the resulting assignment in the dashboard through Pending, Needing Action, and Completed. 

I'll try monkey-patching my local installation to use a larger `misfire_grace_period` in the location you linked, and see if that helps! 

On Wed, May 26, 2021 at 3:23 PM Dave Eargle <da...@daveeargle.com> wrote:

Would be here https://github.com/NYUCCL/psiTurk/blob/d56d26c1822599dafc3c79433ac704f0cbc06d80/psiturk/models.py#L320-L327

--
You received this message because you are subscribed to the Google Groups "PsiTurk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psiturk+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/3d5ceca8-dde1-4b2a-b213-40f9990f6e70n%40googlegroups.com.

[Dave Eargle]

Setting misfire_grace_time would be done here https://github.com/NYUCCL/psiTurk/blob/d56d26c1822599dafc3c79433ac704f0cbc06d80/psiturk/models.py#L320-L327

I'm still reading about it

On Wed, May 26, 2021 at 4:18 PM Cody Wild <cody...@berkeley.edu> wrote:

--

[Dave Eargle]

It looks like the default misfire grace time is only 1 second! https://github.com/agronholm/apscheduler/blob/e77e9339c675fcc30a9f6ae85dbd0b1a7b13427f/apscheduler/schedulers/base.py#L707-L711 If gunicorn is busy serving another thread, this is pretty likely to be missed.

Probably a sane misfire_grace_time would be the specified spacing between hits for the campaign.

[Cody Wild]

Yeah, empirically, it does seem like setting `misfire_grace_time` to 5s has caused me to be able to successfully create HITs through the scheduler :

```

2021-05-26 15:30:20,482 Added job "do_campaign_round" to job store "default"
2021-05-26 15:30:20,564 Looking for jobs to run
2021-05-26 15:30:21,284 Next wakeup is due at 2021-05-26 22:32:19.711634+00:00 (in 119.146860 seconds)
2021-05-26 15:30:21,285 Running job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:32:19 UTC)" (scheduled at 2021-05-26 22:30:19.711634+00:00)
2021-05-26 15:30:23,248 Posting total of 9 assignments this round.
2021-05-26 15:30:23,997 Response(Operation: create_hit | Status: success)
2021-05-26 15:30:23,997 Job "do_campaign_round (trigger: interval[0:02:00], next run at: 2021-05-26 22:32:19 UTC)" executed successfully

```

Notably, it looks like the job schedule for 22:30:19 got run at 22:30:21, so that would be outside the 1s default 

You received this message because you are subscribed to a topic in the Google Groups "PsiTurk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/psiturk/diE8-rld6Rk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to psiturk+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psiturk/CAKL%3D07R7w1behJcMKz-hPske3P22vzwoZcPgcX5R%3DiGu7y6r2A%40mail.gmail.com.

[Dave Eargle]

Great! Thanks for reporting back!