RabbitMQ workers got stuck, the messages are unacked but the connection is alive

[Manol Trendafilov]

I am developing scalable system with RabbitMQ. I write the application in Symfony2 ( PHP ) and the php workers had problems and they lose connection.

I was inspired from this article: https://blog.vandenbrand.org/2015/01/09/symfony2-and-rabbitmq-lessons-learned/

to setup my workers and to solve issues with the PHP workers.

The default setup is with worker written in Go and it passes the data to a PHP command line handler script. Actually it works very well, but when I performed a load test with 15k messages and 6 workers the system got stuck after 6-10 hours of work and the messages status was 'unacked'. The connection between RabbitMQ and the workers is alive.

This is the go worker which I use ( it behave as a wrapper ): https://github.com/ricbra/rabbitmq-cli-consumer

This is the configuration of this worker:

    [rabbitmq]
    host = HOSTNAME
    username = USERNAME
    password = PASSWORD
    vhost=/
    port=5672
    queue=fetch_product_queue
    compression=Off

    [prefetch]
    count=1
    global=Off

    [exchange]
    name=send_product
    type=direct
    autodelete=Off
    durable=On

    [logs]
    error = /tmp/fetch_product_error.log
    info = /tmp/fetch_product_info.log

I belive it's not a development issue but it's more likely to be a RabbitMQ configuration issue since it works well with several thousand messages.

The Go worker uses this library: streadway/amqp which supports AMQP 0.9.1. The RabbitMQ server version is 3.6.1

The server operates in DigitalOcean with 2 cores, 2 GB RAM.

The workers are placed on 6 DO instances with 512 MB RAM and 1 core. Thank you for your help in advance.

[Michael Klishin]

It is really hard to suggest much without seeing your code.

If you have unacknowledged messages, it is not a RabbitMQ configuration issue.

Rather your consumer code needs to be reviewed as it does not acknowledge messages.

See tutorial 2 [1]. RabbitMQ logs could give a clue. The Go client you use very intentionally does not provide

automatic connection recovery and no client recovers from channel errors (because those can indicate

application issues). 

1. http://www.rabbitmq.com/getstarted.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ