Having another severity to get proper resolved alerts with value, Is that right to do?

[Rajesh Reddy Nachireddi]

Hi,

We have created one severity level to handle resolved alert issues: Following are the details:

1. Resolved alerts are not differentiable between what is actually condition not matching with firing and metric not found or condition not met

           - Keeping the metric absence check on every rule is cumbersome

2. Resolved alerts are having last firing result , however it is known proemtheus design pattern.

           - how to deal with it, can we add expression as part of description to evaluate as part of annotations, how much load will it be on prometheus and how to write the expressions ?

So, having above 2 issues, we have added 'CLEARED' as an another severity and disabled send_revolved flag. Will it be the right way to do it  or any other suggestions to do similar

and also do you have any case studies to go over large deployments E2E life cycle with proemtheus and alertings, remote storage.

Please suggest, it will help us and the community to go stronger with better understanding.

Thanks in advance.

Regards,

Rajesh