Re: [carrot-users] Kombu performance with librabbitmq vs pika > 0.9.5 ?

[Ask Solem]

On Jan 23, 2013, at 12:57 AM, Matteius Davis <matt...@gmail.com> wrote:

> Greetings,
>
> I hope this posting finds some people well, as I have some questions, we have some Python RabbitProducer classes including an RPC version of them, and they use the pika library to implement the RabbitMQ communications. I know this is not a pika mailing list (though I did not see one), this also pertains to kombu for which I am planning a trial conversion tonight to see if this will even work in kombu.
>
> Well we have been running this pika at 0.9.5 for a while with stability and reasonable response times, but recently we upgraded to 0.9.6 and 0.9.8 to implement request time-outs when doing RPC callbacks, and this included a lovely new feature *sarcasm* of a SOCKET_TIMEOUT witha default value of 0.25 seconds, so now all of a sudden where we never had time-outs, we now have a new kind of timeout trigering errors. When we raised this timeout to 10 seconds our application time shot up to 70 seconds per request, dropping to .5 seconds it now adds 3 seconds to the request. Since we decided the overall timeout should still happen due to nginx, well we have downgraded back to 0.9.5 pika for performance reasons.
>
> Pika appears dead in the water at this point, this issue seems well under documented and yet some timeout stuff exists on the gthub issue tracker though perhaps unrelated. Has anyone else noticed a performance degredation upgrading the recent pika version? Or timeout issues?

I'm not sure, I guess the issue tracker would be the canonical source. It does sound like a bug, maybe easily fixed, but I don't know, I'm not involved in the development of it.

>
> Well now I want our implementation to work for Kombu, not using pika, but using the probably much faster C based librabbitmq. Can anyone verify that kombu with librabbitmq is much faster than pika, and less error prone with the timeouts?

librabbitmq is written in C, it does not yet support Consumer Cancel Notifications or heartbeats, but this is
something I'm working on and it will be ready soon[1].

Note that benchmarks can be regarded as a game at best, you should never trust them to give
you anything useful, e.g. usually applications will not be sending and receiving messages most of the time.

I've never benchmarked librabbitmq against pika.
I have seen drastic improvements when comparing librabbitmq to amqplib, but that doesn't mean
it applies to pika.


Kombu does not use pika by default, it uses another library (py-amqp: http://pypi.python.org/pypi/amqp).
This is a fork of amqplib with many improvements. There have been plans to switch to pika, and even some experimentation, but it never got to an acceptable state, and I feel that the pika API changes too frequently so it would at least have to wait for a pika 1.0 release.

[1] py-amqp supports these so you can revert to the pure-python implementation if you require them.

>
> I am also curious for someone to verify if the default engine of kombu is pika unless librabbitmq is installed? I am teriffied that pika is at the root of all the Python, RabbitMQ client/producer libraries, but there is hope that kombu can run stable and without depending on pika. I got worried when I saw pika class types defined in the kombu documetnation.
>
> Please advise,
>
> --
> You received this message because you are subscribed to the Google Groups "carrot-users" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/carrot-users/-/hASaqm9JUpoJ.
> To post to this group, send email to carrot...@googlegroups.com.
> To unsubscribe from this group, send email to carrot-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/carrot-users?hl=en.



--
Ask Solem
twitter.com/asksol