Alert Query

[sri L]

Hi all,

I am trying to create an alert rule for pod unreachable condition. Below expression I used but alert was triggering whenever new pod got created, we want alert only when the previous state of a pod was in the ready state and then went to unreachable/terminating/pending states.

kube_pod_status_ready{condition="true"} == 0

Please suggest if we have any suitable alert expression for the above requirement.

Thanks

[Brian Candler]

I guess it goes through non-ready states while it's starting up.

A simple approach is to put "for: 3m" on the alert so that it doesn't fire an alert until it has been in the down state for 3 minutes.

Another approach would be:

kube_pod_status_ready{condition="true"} == 0 and max_over_time(kube_pod_status_ready{condition="true"}[10m]) == 1

This will fire if the pod was ready at any time in the last 10 minutes, but is not ready now. This does mean that the alert will clear after 10 minutes of error condition, though.

[sri L]

Thanks Brian Candler.

I am thinking of combining two conditions.

((kube_pod_status_ready{condition="true"} == 0 and max_over_time(kube_pod_status_ready{condition="true"}[10m]) == 1) or (max_over_time(kube_pod_status_ready{condition="true"}[10m]) == 0))

Expecting this expression to alert if pod was up in the last 10 mins and currently unreachable or pod is unreachable from last 10mins or more.

Please correct if there is any better way

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7728c736-797f-4771-b809-24e5f6b3931dn%40googlegroups.com.

[Brian Candler]

(max_over_time(kube_pod_status_ready{condition="true"}[10m]) == 0))

That won't solve your problem, because if there were only one initial sample with value zero, it would trigger immediately, just as you do today.  You would need to make this more complex, for example with count_over_time as well.

Be very careful with "and" and "or" operators. They do not work like booleans in normal languages. They are combining vectors, matching the label sets of each element in the vector.

I recommend you go for the simplest expression which meets your needs sufficiently well.