Aync consumer?

[Patrik Iselind]

Hi,

I'm following http://aio-pika.readthedocs.io/en/latest/rabbitmq-tutorial to see if aio-pika will fit in an upcoming project.

One thing that strikes me is that the on_message function isn't a coroutine. I couldn't find any documentation on how to handle this.

My use case is something along these lines.
My app will get work from a RabbitMQ queue. The work involved is mostly IO bound but could take quite a while (up to say an hour or so) to complete, hence i want to use aiohttp and Motor (https://motor.readthedocs.io/en/stable/) with friends in the on_message function as well.

Most examples i find of using asyncio know the work to be performed in advance. I've yet to discover the examples where you don't know the work in advance.

Do you have any suggestions?

Regards,
Patrik

[Michael Klishin]

RabbitMQ client libraries need a function to call for each delivery. From there you can use any

concurrent dispatch mechanism you want as long as you avoid

 * Sharing channels between threads for publishing

 * Double acknowledgements

 * Acking on a channel different from that the delivery was on

As far as the protocol goes, most other scenarios are entirely an implementation detail.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Patrik Iselind]

Den måndag 14 augusti 2017 kl. 10:21:48 UTC+2 skrev Michael Klishin:

RabbitMQ client libraries need a function to call for each delivery. From there you can use any

concurrent dispatch mechanism you want as long as you avoid

 * Sharing channels between threads for publishing

 * Double acknowledgements

 * Acking on a channel different from that the delivery was on

As far as the protocol goes, most other scenarios are entirely an implementation detail.

On Mon, Aug 14, 2017 at 11:14 AM, Patrik Iselind <patri...@gmail.com> wrote:

Hi,

I'm following http://aio-pika.readthedocs.io/en/latest/rabbitmq-tutorial to see if aio-pika will fit in an upcoming project.

One thing that strikes me is that the on_message function isn't a coroutine. I couldn't find any documentation on how to handle this.

My use case is something along these lines.
My app will get work from a RabbitMQ queue. The work involved is mostly IO bound but could take quite a while (up to say an hour or so) to complete, hence i want to use aiohttp and Motor (https://motor.readthedocs.io/en/stable/) with friends in the on_message function as well.

Most examples i find of using asyncio know the work to be performed in advance. I've yet to discover the examples where you don't know the work in advance.

Do you have any suggestions?

Regards,
Patrik

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

I'm not sure you got my question, either that or i don't understand your answer.

In short, there is nothing in the documentation stating if the on_message can be a coroutine or not. According to the code it seems that it's possible.

If on_message can indeed be a coroutine, and that coroutine takes 'quite a while' to complete. Does that get in the way of consuming the next message in any way? I'm not quite into using asyncio, yet.

// Patrik

[Michael Klishin]

I'm not a Python expert by any stretch but here's how consumer delivery

callback is invoked:

https://github.com/pika/pika/blob/01adac6449105a3808b889587c44b8592cf9be56/pika/channel.py#L1124

so, it is invoked as a function/callable. If my reading of

https://docs.python.org/3/library/asyncio-task.html

is correct, await or yield need to be used for the runtime to guarantee corouting execution. Pika doesn't

do that.

https://github.com/pika/pika/blob/3f472a6e75975705448d2bc612a9ee27cd0fa0bf/pika/connection.py#L1455

https://github.com/pika/pika/blob/3f472a6e75975705448d2bc612a9ee27cd0fa0bf/pika/connection.py#L2042

whether consumer dispatch operations may block I/O event loop is an adaptor-specific thing.

Pika does not use a thread pool or similar to dispatch consumer-specific methods (e.g. basic.deliver)

like some other clients do, so that's certainly possible but perhaps async-io accounts for that?

It should be easy enough to test with a delivery handler that sleeps for a few minutes ;)

See also http://www.rabbitmq.com/confirms.html and tutorial 2.

HTH.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.