Monitoring Streams

[Ryan Riley]

Hello,

I'm really excited about Streams and appreciative of this community's contributions! A few questions around monitoring...

I'm trying to figure out how to manage streams, such as monitoring the difference from a consumer's offset to the queue's offset to get an idea of how deep the "backlog" is in a given stream. 

I saw in a comment that there are entries under the per-object metrics endpoint. We have 2000-4000 queues in our rabbit cluster, so this ends up being an untenable number of time-series for Prometheus to handle for us. We have used the /metrics/detailed endpoints with success, but I don't see anything for Streams.

Further, in the Management UI, it seems you can only see the "Ready" messages which is ever-increasing. What is the use-case for this? I'm not sure how to make this data useful. It skews our alarms for queued messages as well, as there seems to be no way to discriminate between classic queues and stream queues. Would adding a Prometheus label for Queue type (classic/quorum/stream) be a reasonable feature request?

In essence the question is: what is the expected way to monitor how far behind consumers are on a given Stream? 

[Arnaud Cogoluègnes]

There should be the offset_lag metric. It represents the lag of a given consumer on a given stream.

[Ryan Riley]

Hi Arnaud,

Thanks, but where do I find this metric? Is it exclusively in the per-object metrics? In the default Prometheus metrics I see a rabbitmq_stream_consumer_max_offset_lag, which is helpful, but in the Detailed Metrics documentation I don't see that offset_lag metric. Is there a way to acquire this in the detailed metrics view?

As I mentioned, using per-object metrics is prohibitively resource intensive for us because of the number of queues our cluster has.

[Michal Kuratczyk]

As far as I can tell, it is indeed not currently available in /metrics/detailed. I will look into adding it there. For the time being, you can get it from /metrics/per-object.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/rabbitmq-users/207355d7-9f3a-422f-968d-16c061d07a1bn%40googlegroups.com.

--

Michal

RabbitMQ Team

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[Michal Kuratczyk]

Correction: my bad, it is available:

$ curl -s 'localhost:15692/metrics/detailed/?family=stream_consumer_metrics' | rg s1
rabbitmq_detailed_stream_consumer_max_offset_lag{vhost="/",queue="s1"} 149384782

The problem is that this family of metrics is not mentioned in https://www.rabbitmq.com/docs/prometheus#detailed-endpoint

[Ryan Riley]

Aha! Excellent! Thanks for finding this, very helpful. I see it has a `rabbitmq_detailed_stream_consumer_max_offset_lag` on a per-queue level, which is exactly what I was looking for.