long live consumer(work queues) seems not consuming message

[Chen Wei Hsu]

I use the example from tutorial-two-php to construct a work queue. It looks fine when I first test it. However, it looks like the consumer stops responding after about few hours or a day. I use a daemon and run the consumer as a worker, and I make the daemon as a service in ubuntu 16.04. When checking service status, the service(daemon) and worker are both running. But it is just not responding to the new message I send to the queue. I am not using any timeout or heartbeat setting, and it's just basic examples like the tutorial. What could possibly make this happen?

I now use a workaround to timeout the consumer like "$this->channel->wait(null, false, mt_rand(self::TIME_LIVE_MIN_SEC, self::TIME_LIVE_MAX_SEC));", so I won't have a very long living consumer which I consider as a bad thing in php. Any suggestion to avoid long living consumer would be welcome. Thank you

[Michael Klishin]

Hi Chen,

Per-operation timeouts is a common thing in messaging-based systems. You seem to be using a PHP client and it has its own

specific limitations due to PHP's programming model.

However, I believe you may be chasing the wrong goose. Are you sure that your daemon didn't e.g. lose connection to RabbitMQ?

As far as I know phpamqp-lib does not support automatic connection recovery. A channel exception is another event

that would halt all activity on a channel since the channel is then closed. Does your application code handle such scenarios, or at least logs them?

It's quite easy to see total number of connections, list them, explore consumers on a queue and so on via RabbitMQ management UI [1].

See also [2] as it is a highly relevant topic.

1. http://www.rabbitmq.com/management.html

2. http://www.rabbitmq.com/heartbeats.html

On Sun, Sep 17, 2017 at 10:26 PM, Chen Wei Hsu <cwhs...@gmail.com> wrote:

I use the example from tutorial-two-php to construct a work queue. It looks fine when I first test it. However, it looks like the consumer stops responding after about few hours or a day. I use a daemon and run the consumer as a worker, and I make the daemon as a service in ubuntu 16.04. When checking service status, the service(daemon) and worker are both running. But it is just not responding to the new message I send to the queue. I am not using any timeout or heartbeat setting, and it's just basic examples like the tutorial. What could possibly make this happen?

I now use a workaround to timeout the consumer like "$this->channel->wait(null, false, mt_rand(self::TIME_LIVE_MIN_SEC, self::TIME_LIVE_MAX_SEC));", so I won't have a very long living consumer which I consider as a bad thing in php. Any suggestion to avoid long living consumer would be welcome. Thank you

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Chen Wei Hsu]

Hi MK,

My daemon does not keep any connection, it only spawns a worker to make a connection and wait for messages. The works are forked process from the daemon. I don't remember seeing any channel exceptions but 

I will definitely keep an eye on it when I use the channel without timeout next time. Thanks for the suggestion!

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Chen Wei Hsu]

Though timeout a worker works just fine, I still want to try the long-live worker. After a few hours of test, I can see the errors when the workers are read.

rabbitmq log:

=WARNING REPORT==== 25-Sep-2017::07:55:47 ===

closing AMQP connection <0.13338.15> (172.31.28.127:46818 -> 172.31.28.231:5672, vhost: '/', user: 'guest'):

client unexpectedly closed TCP connection

php log:

Exception from 15091: Broken pipe or closed connection"

And from what I saw in the log, my long-live workers always die after about an hour. After some experiments, I think the root cause is that I have a haproxy to proxy the message to message queue, and the timeout tunnel is set to 3600s which is just the case when I see the worker dead. I think I just find the root cause of my dead worker.

[Michael Klishin]

Thanks for reporting back! http://www.rabbitmq.com/heartbeats.html mentions

heartbeats' effect on certain aspects of the proxy behavior, so while it's not their

real reason for existence, they can be useful in this case — make sure to enable them

with a value of, say, 10 seconds.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.