[CRITICAL] WORKER TIMEOUT

[Jeff Eilbott]

Hello,

We've been experiencing issues with an experiment, often getting a gunicorn "[CRITICAL] WORKER TIMEOUT" in server.log.

I've checked to make sure the server isn't under too much load (memory, CPU, bandwidth) and that's not the case.

It seems to occur sporadically throughout the experiment in maybe 10-20% of runs. There are two calls to preload images (about 15 MB each) and it doesn't appear to coincide with these calls.

Other than recording trial data, there are no other calls to the server (or any external service) in the experiment, so I'm not sure what could be causing the timeout.

I'd appreciate any help. Our lab is unable to replicate when testing the experiment ourselves.

Thanks,
Jeff

[Jeff Eilbott]

After doing some more research, the only thing I can think of causing the timeout is the psiTurk.preloadImages call, specifically when participants have a slow internet connection, which is why we can't replicate. I read in a GitHub issue this could be due to the synchronous nature of gunicorn and a solution would be to put nginix in front? Is that a reasonable first step for troubleshooting? Any tips?

Thanks!

[Jeff Eilbott]

Further, I've been looking at the network traffic and throttling bandwidth to 50kbps with Firefox developer tools and haven't been able to recreate. Some requests are even taking > 60 seconds and not triggering a "WORKER TIMEOUT" on the server.

[Jeff Eilbott]

Also, for context, I'm running two experiments on the same server on different ports. Is there any way that could be responsible?

Thanks!

[Dave Eargle]

Sorry, I missed all of these. You're right that WORKER TIMEOUT is due to gunicorn workers (threads) going too long. Yes, you absolutely should put nginx in front to cache with images that big, or with a high number of images. I found an old gist I wrote with an example nginx conf: https://gist.github.com/deargle/5d8c01660a77b8090a2cd24efcda2c59#file-psiturk-use-ad-server-conf

And no, each of your two experiments would use different gunicorn workers. 

[Avinash Vaidya]

Hi Dave,

I am having some trouble parsing the gist you have written for the example nginx. I am trying to deal with a very similar problem as Jeff where we have an experiment with lots of images and would want to put nginx in from of our experiment which is being run in PsiTurk 2.3.12 (we are still using the PsiTurk Ad server). It looks like the last example in the gist is applicable to this problem but I am not sure where this code is supposed to go...? 

Thanks,

Avi