Prometheus Alert-Generator Compliance testing for Cortex

[Ganesh Vernekar]

Hello Cortex team,

I hope this email finds you well.

As a part of Prometheus conformance program, we had floated a doc on "Prometheus Alert-Generator Compliance" many months ago and had finalized the specification for that (see here).

The test suite to test the specification is now ready and instructions on how to run the test suite are present here.

If you wish Cortex to be compliant with Prometheus Alert-Generator, please test your software with the above test suite and report back the results (results being the entire log output of the test run).

If you do need any additional help from the test suite (for example setting custom headers to some requests), please let me know and I will add those abilities to the test suite.

If you face issues in running the test suite or understanding any error messages, I am happy to answer your queries.

We plan to publish the results during the third week of May 2022 (during KubeCon EU).

PS: I already tested Cortex locally since I was familiar with it and it almost passed. It needs a Prometheus upgrade so that it can support a couple of new template functions that were introduced after the Prometheus version that is vendored in Cortex (Prometheus v2.33 is the latest that the compliance is looking at).

Thanks,
Ganesh (codesome)
Prometheus team

[Jérôme Decq]

Thanks, created https://github.com/cortexproject/cortex/issues/4656 to track the work

[Jean-Philippe Quéméner]

Update on the alert generator compliance:

Some vendors cannot expose some of the APIs required by the Prometheus alert generator compliance test suite.

We modified the test suite so it can be run by vendors themselves and submit results to Prometheus/CNCF. CNCF will contractually protect against submitting wrong results on purpose.

If there are any questions, please feel free to ask in this thread.

Thanks,

Jean-Philippe Quéméner (johnnyqqqq)