I'm building a web application with Flask, gevent-websocket and Celery. The workflow I'm trying to support is:
- a client requests a job via a websocket being listened to by gevent
- the web app submits the job to celery
- when the job is done, the result from celery is written back into the web socket
some psuedo-ish code:
@app.task
def handle_message(message):
...
def handle_request(socket):
while True:
message = socket.receive()
job = handle_message.delay(message)
result = job.get()
socket.send(result)
where gevent is in charge of that loop and celery is passing off that handle_message call through a job queue.
My question is if that job.get() call will mess with gevent's cooperative multitasking by blocking and not letting other greenlets execute because there isn't a gevent.sleep() call or something to yield control to another greenlet. If that's the case, is there something I can do to make job.get() yield control to other greenlets?
I was using rabbitmq as the backend so that I could do non-polling job.get() calls, but it does occur to me that I could use redis or another polling backend to do something like:
while job.status not in (celery.states.FAILURE, celery.states.SUCCESS)
gevent.sleep(1)
to ensure I'm playing nice with gevent, but I'd prefer not to have to poll my result backend if I don't have to.
Thanks, and sorry if this is a silly question or one that would be better placed on the Celery mailing list -- I'm new to many of these technologies. Other suggestions on how to architect a system like this would also be welcome if I appear to be doing something silly.
Thanks again!