Celery, gevent and AsyncResult.get()

[Walt Askew]

I'm building a web application with Flask, gevent-websocket and Celery.  The workflow I'm trying to support is:

- a client requests a job via a websocket being listened to by gevent

- the web app submits the job to celery

- when the job is done, the result from celery is written back into the web socket

some psuedo-ish code:

@app.task

def handle_message(message):

   ...

def handle_request(socket):

   while True:

        message = socket.receive()

        job = handle_message.delay(message)

        result = job.get()

        socket.send(result)

where gevent is in charge of that loop and celery is passing off that handle_message call through a job queue.

My question is if that job.get() call will mess with gevent's cooperative multitasking by blocking and not letting other greenlets execute because there isn't a gevent.sleep() call or something to yield control to another greenlet.  If that's the case, is there something I can do to make job.get() yield control to other greenlets?

I was using rabbitmq as the backend so that I could do non-polling job.get() calls, but it does occur to me that I could use redis or another polling backend to do something like:

while job.status not in (celery.states.FAILURE, celery.states.SUCCESS)

    gevent.sleep(1)

to ensure I'm playing nice with gevent, but I'd prefer not to have to poll my result backend if I don't have to.

Thanks, and sorry if this is a silly question or one that would be better placed on the Celery mailing list -- I'm new to many of these technologies.  Other suggestions on how to architect a system like this would also be welcome if I appear to be doing something silly.

Thanks again!

[Shahaf Abileah]

I don't have an answer for your quesiont, but here's some related info you may find useful.

We have a very similar setup at my company.  We wanted to use RabbitMQ as the storage backend for the same reason - to avoid polling.  But we also wanted the ability to "broadcast" the result, in case multiple listeners care to have the same result, and Rabbit isn't set up for that (as far as we can tell).  So we ended up using Mongo as the store, and we do poll, which is a shame.  But overall the system works pretty well.

Anyhow, I'm curious to hear what other people say about task.get(), whether it's blocking for gevent.

--S

--
You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

http://twitter.com/shahafabileah

[AM]

If you have monkey patched everything and job.get makes a network
request (which I believe it will do in this case) you should be good to go.

A more robust solution would be use a redis backend and use its pub sub
functionality to 'subscribe' to celery events.

HTH
AM

> --
> You received this message because you are subscribed to the Google
> Groups "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gevent+un...@googlegroups.com

> <mailto:gevent+un...@googlegroups.com>.