Correlating different alerts to produce single alert

[radhamani...@gmail.com]

I want to send alert by  doing some correlation based upon multiple alerts.For eg: if podA,podB,serviceA are all 100% down in two different namespaces(namespace1,namespace2),then  I want to send alert like ApplicationA is down. Is this possible? How to do the correlation between different alerts?

[Christian Hoffmann]

Hi,

There is a special metric called ALERTS which exists for every active
(pending or firing) alert. You could cross-reference this.

However, there might be better ways which are more readable. I'm not
sure either if it would be a good idea to use ALERTS in other alerts. :)

In general this sounds like you could go with a simple up-based alert?

If you need to distinguish based on namespace, you might be looking for
the count or sum functions along with some by(namespace)?

Kind regards,
Christian

[radhamani...@gmail.com]

Thankyou for the response..

I think I didnt explain my question properly..

If pods in two different namespaces go down,then we need to send a alert as an appA is down..

Can I simply write expr as Kubepoddown_in_namespaceA and Kubepoddown_in_namespaceB ,and send alert message as "AppA is down"?

I just wrote the pseudocode of the expr but I want to know if this expr simply works with just AND operator..

[Christian Hoffmann]

Hi,

On 8/27/20 2:19 AM, radhamani...@gmail.com wrote:
> If pods in two different namespaces go down,then we need to send a alert
> as an appA is down..
> Can I simply write expr as Kubepoddown_in_namespaceA and
> Kubepoddown_in_namespaceB ,and send alert message as "AppA is down"?

Yes, that's what I meant with the ALERTS metric. Have you tried it? You
can check those queries using the Prometheus UI (or API) before putting
them into alert rules.

For example, check
ALERTS{alertname="Kubepoddown_in_namespaceA",alertstate="firing"}.

> I just wrote the pseudocode of the expr but I want to know if this expr
> simply works with just AND operator..

My second suggestion still applies: It might be better or more
transparent if you could directly reference the relevant metrics.
However, to provide more specific guidance, we would need to know the
details of your existing two alerts.

Kind regards,
Christian

[radhamani...@gmail.com]

Thankyou very much..I will update you soon,,