idle_since updated for all queues every 2 minutes since 3.8.1

[Jesse Young]

Hi,

After upgrading to the current version of the RabbitMQ server (3.8.5) I noticed that the "idle_since" property returned by the management API (GET /api/queues/vhost)  is updated every 2 minutes for every queue. 

Our application was using the idle_since property to determine which queues have been idle for a long time and delete them to avoid using too many file descriptors. After upgrading to 3.8.5, this approach no longer works because no queues are ever idle for more than 2 minutes.

I tried various different versions of RabbitMQ and determined that this issue only occurs on 3.8.1 and above. 3.8.0 as well as 3.7.26 work fine.

The issue can be reproduced by running the official Docker image (rabbitmq:3.8.1-management), creating a queue, then hovering over the "idle" state on the queues page of the management UI (or using the API). The "idle since" time will update every 2 minutes.

Looking at the changes between 3.8.0 and 3.8.1, I'm guessing it may have been caused by https://github.com/rabbitmq/rabbitmq-server/pull/2143 , which checks the state of queues during the garbage collection process.

Is there a workaround, or would it be possible to restore the previous idle_since behavior?

Thanks,

Jesse

[Luke Bakken]

Hello,

idle_since has been discussed on this list already -

https://groups.google.com/d/topic/rabbitmq-users/HOWrfrJySWE/discussion

https://github.com/rabbitmq/rabbitmq-management/issues/766

Thanks,

Luke

[you...@telerivet.com]

Hi Luke,

Thanks for the links! I didn't find those discussions when searching this list.

I tested fetching message_stats.publish_details and message_stats.deliver_get_details to determine the approximate last time that the queue was active. However, fetching these stats for all queues is very slow when the server has a large number of queues (the API response time depends on the number of samples requested). Also, since the samples are returned at fixed increments, determining the approximate last time a queue was active requires making multiple API requests with different intervals, like msg_rates_age=3600&msg_rates_incr=60 / msg_rates_age=86400&msg_rates_incr=3600 / msg_rates_age=2592000&msg_rates_incr=86400. It's not really an ideal approach.

It seems that the current position of RabbitMQ developers is that idle_since "does not and never has meant consumer activity". If RabbitMQ developers expect users to know this, I'd suggest adding it to the API documentation. RabbitMQ's management API documentation doesn't actually describe the meaning of the values returned by API calls (including the idle_since property), so users are forced to make assumptions. For years, it worked fine to assume that idle_since is only updated when something external interacts with the queue.

We are planning to revert to an earlier version of RabbitMQ before 3.8.1 for now. I'm hoping that if we ever need to upgrade, RabbitMQ's management API might have some way to get timestamps for the last publish/consume/get activity on a queue. It seems that there are at least some other users that would also find this useful.

Thanks,

Jesse

[Michael Klishin]

We will document this field as a debugging metric that does not try to represent consumer activity

(although for certain workloads, it effectively will) and something we'd not recommend for most users

to use to assess consumer activity. Application-level metrics would be much more accurate, representative and would give

both developers and operators much more more control.

[you...@telerivet.com]

Hi Michael,

Documenting this field would be great.

I'm not sure what you mean by application-level metrics. Does that mean our application using another mechanism (e.g. database/cache) to keep track of the last time it published or consumed from each queue?

Do you think it would be possible to keep track of the last publish/consume time for each queue in a RabbitMQ plugin?

Thanks,

Jesse