tgext.asyncjob not working with uwsgi as service

[Bastien Sevajol]

Hello,

We have troubles with tgext.asyncjob and uwsgi. tgext.asyncjob worker didn't receive job when project is started with uwsgi as a service. To resume when it's working or not:

(Debian 8.5)

• Project started with gearbox: working
• Project started with uwsgi cli command and following ini config: working
• Project started with uwsgi service and following ini config: NOT working
• Project started with uwsgi service and following ini config AND lazy-apps option: working
  

Note: We can't use lazy-apps uwsgi option because breaking some parts of our project.

The uwsgi ini config is the following:

[uwsgi]
plugins = python3
chdir = /home/bastien/Projects/tracim/tracim
home = /home/bastien/.virtualenvs/tracim_dev
module = wsgi
callable = application
socket = /var/run/uwsgi/app/algoo.trac.im.local/socket
enable-threads = true

During our debugging, wa can affirm this: Code who request job correctly reach tgext.asyncjob.queue.AsyncJobQueue#perform (and make correctly the queue.Queue#put). But in tgext.asyncjob.worker.AsyncWorkerThread#run, following line:

msg = self.queue.get()

still waiting. It's like this queue is not the same in main program and asyncjob thread.

Do you hav any idea about this bug ?

Best Regards,

Bastien.

[damien accorsi]

Hil all,

Anyone is using tgext.asyncjob with uwsgi ? The behavior is very strange. It's like uwsgi does not allow threads to share a common queue.
Note : asyncjobs were working through apache wsgi ; now that we work with nginx/uwsgi, the asyncjob thread does not communicate with web threads anymore...

Damien

[Alessandro Molina]

I usually don't deploy under uWSGI, so I can't confirm. 

But what you are describing would make sense if uWSGI "workers" are subprocesses.

In that case, by the description of "lazy-apps" option I would guess that if that option is False, uWSGI will fork() with the app already loaded (and by consequence the multiprocessing.Queue already existing). As multiprocessing.Queue is in fact implemented over a pipe the file descriptor will be cloned in sub processes. This is probably causing some odd condition in multiprocessing.Queue which breaks it.

Solution is to run things with lazy-apps or try to find a way to recreate the queue within the new processes.

I saw uwsgi has a "close-on-exec" option to close all inherited file descriptors in forked subprocesses, maybe that will help? If you are lucky enough the multiprocessing.Queue might recreate a new pipe if the current one is closed (or might just crash :D didn't check in multiprocessing source code)

It's all guessing here, but it's a reasonable guess.

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.
To unsubscribe from this group and stop receiving emails from it, send an email to turbogears+unsubscribe@googlegroups.com.
To post to this group, send email to turbo...@googlegroups.com.
Visit this group at https://groups.google.com/group/turbogears.
For more options, visit https://groups.google.com/d/optout.

[Alessandro Molina]

On Tue, Jan 10, 2017 at 9:14 AM, Alessandro Molina <alessand...@gmail.com> wrote:

In that case, by the description of "lazy-apps" option I would guess that if that option is False, uWSGI will fork() with the app already loaded (and by consequence the multiprocessing.Queue already existing). As multiprocessing.Queue is in fact implemented over a pipe the file descriptor will be cloned in sub processes. This is probably causing some odd condition in multiprocessing.Queue which breaks it.

Came to my mind that asyncjob actually uses threads, so the queue is not multiprocessing.Queue() but queue.Queue(), which is an in-memory deque with a mutex and not a pipe. So the file descriptors thing doesn't apply.

Still it applies that the reason probably lies on the fact that it forks after having loaded the app, so lies somewhere as a side effect of memory copying.