Query to alert on queue size increase

[Pete Leese]

I have a legacy system then requests sometimes get setup with a threshold of 50 currently but sometimes can take hours to reach the aforementioned threshold and the damage is already done.

Is there a method of creating an alert that will notify me only if the queue continues to increase over a 10 minute period for example?

Sometimes the queue does drop down after building within 10 minutes but it’s the scenario where it continues to raise over a period of time I wish to be notified upon.

Thanks

Pete

[Matt Palmer]

On Sun, Aug 26, 2018 at 04:05:55PM -0700, 'Pete Leese' via Prometheus Users wrote:
> Is there a method of creating an alert that will notify me only if the queue continues to increase over a 10 minute period for example?

https://prometheus.io/docs/prometheus/latest/querying/functions/#deriv()

- Matt

[von Neudeck, Nicolai]

Hi Pete!

It was not completely clear to me, what queue you were talking about, so please forgive me, if I misunderstood and my answer won’t be helpful to you.
An increasing queue over a certain time should be queryable by taking the first derivative over a graph. In Prometheus that is for practical purposes most of the time rate(), although there is deriv(). So make an alert for the rate() of the qeue length being greater than x for y minutes.
E.g. write an alert to fire when query "rate(qeue_length) > 2" for 10m.
In general it is often very helpful to have a look at curve sketching basics when unsure about how to write alerts.

Don’t give up. Almost everything can be expressed in PromQL, but it can be a bit fiddly sometimes.

Nicolai von Neudeck

-----Ursprüngliche Nachricht-----
Von: 'Pete Leese' via Prometheus Users <promethe...@googlegroups.com>
Antworten an: Pete Leese <petel...@googlemail.com>
Datum: Montag, 27. August 2018 um 01:05
An: Prometheus Users <promethe...@googlegroups.com>
Betreff: [prometheus-users] Query to alert on queue size increase

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9e63f51f-5e83-4bbe-b2d5-1e9f76ec838d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Brazil]

On 27 August 2018 at 09:33, von Neudeck, Nicolai <nicolai.v...@sap.com> wrote:

Hi Pete!

It was not completely clear to me, what queue you were talking about, so please forgive me, if I misunderstood and my answer won’t be helpful to you.
An increasing queue over a certain time should be queryable by taking the first derivative over a graph. In Prometheus that is for practical purposes most of the time rate(), although there is deriv(). So make an alert for the rate() of the qeue length being greater than x for y minutes.
E.g. write an alert to fire when query "rate(qeue_length) > 2" for 10m.

rate() is for use with counters, and deriv() is for use with gauges. A queue length is a gauge as it can go up and down, so deriv(queue_length[10m]) > 0 is what you want.

Brian

 

In general it is often very helpful to have a look at curve sketching basics when unsure about how to write alerts.

Don’t give up. Almost everything can be expressed in PromQL, but it can be a bit fiddly sometimes.

Nicolai von Neudeck

-----Ursprüngliche Nachricht-----

Von: 'Pete Leese' via Prometheus Users <prometheus-users@googlegroups.com>

Antworten an: Pete Leese <petel...@googlemail.com>
Datum: Montag, 27. August 2018 um 01:05

An: Prometheus Users <prometheus-users@googlegroups.com>

Betreff: [prometheus-users] Query to alert on queue size increase

    I have a legacy system then requests sometimes get setup with a threshold of 50 currently but sometimes can take hours to reach the aforementioned threshold and the damage is already done.

    Is there a method of creating an alert that will notify me only if the queue continues to increase over a 10 minute period for example?

    Sometimes the queue does drop down after building within 10 minutes but it’s the scenario where it continues to raise over a period of time I wish to be notified upon.

    Thanks

    Pete

    --
    You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
    To post to this group, send email to prometheus-users@googlegroups.com.

    To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9e63f51f-5e83-4bbe-b2d5-1e9f76ec838d%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7AD0D90C-E0DC-4713-A6EF-AB810D133426%40sap.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Pete Leese]

Thanks for the answers.

I do keep meaning to buy Brian’s book and get some courses booked but you know how it is :)

Cheers

Pete.

[Nicolai von Neudeck]

Thanks for correcting me Brian.
I knew about the deriv for gauges, but not that rate was only for counters.
And I also forgot to add the time in the query.
I should have taken more care when answering the question.
That was very helpful, thank you very much.

Nicolai

Am Montag, 27. August 2018 10:43:13 UTC+2 schrieb Brian Brazil:

On 27 August 2018 at 09:33, von Neudeck, Nicolai <nicolai.v...@sap.com> wrote:

Hi Pete!

It was not completely clear to me, what queue you were talking about, so please forgive me, if I misunderstood and my answer won’t be helpful to you.
An increasing queue over a certain time should be queryable by taking the first derivative over a graph. In Prometheus that is for practical purposes most of the time rate(), although there is deriv(). So make an alert for the rate() of the qeue length being greater than x for y minutes.
E.g. write an alert to fire when query "rate(qeue_length) > 2" for 10m.

rate() is for use with counters, and deriv() is for use with gauges. A queue length is a gauge as it can go up and down, so deriv(queue_length[10m]) > 0 is what you want.

Brian

 

In general it is often very helpful to have a look at curve sketching basics when unsure about how to write alerts.

Don’t give up. Almost everything can be expressed in PromQL, but it can be a bit fiddly sometimes.

Nicolai von Neudeck

-----Ursprüngliche Nachricht-----

Von: 'Pete Leese' via Prometheus Users <promethe...@googlegroups.com>

Antworten an: Pete Leese <petel...@googlemail.com>
Datum: Montag, 27. August 2018 um 01:05

An: Prometheus Users <promethe...@googlegroups.com>

Betreff: [prometheus-users] Query to alert on queue size increase

    I have a legacy system then requests sometimes get setup with a threshold of 50 currently but sometimes can take hours to reach the aforementioned threshold and the damage is already done.

    Is there a method of creating an alert that will notify me only if the queue continues to increase over a 10 minute period for example?

    Sometimes the queue does drop down after building within 10 minutes but it’s the scenario where it continues to raise over a period of time I wish to be notified upon.

    Thanks

    Pete

    --
    You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

    To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
    To post to this group, send email to promethe...@googlegroups.com.

    To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9e63f51f-5e83-4bbe-b2d5-1e9f76ec838d%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7AD0D90C-E0DC-4713-A6EF-AB810D133426%40sap.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io