Prometheus Alert-Generator Compliance testing for Google Managed Prometheus

[Ganesh Vernekar]

Hello Fabian,

I hope this email finds you well.

As a part of Prometheus conformance program, we had floated a doc on "Prometheus Alert-Generator Compliance" many months ago and had finalized the specification for that (see here).

The test suite to test the specification is now ready and instructions on how to run the test suite are present here.

If you wish Google Managed Prometheus to be compliant with Prometheus Alert-Generator, please test your software with the above test suite and report back the results by replying to this thread (results being the entire log output of the test run).

If you do need any additional help from the test suite (for example setting custom headers to some requests), please let me know and I will add those abilities to the test suite.

Going forward, we would like to automate this process. We ask you to add your test-suite config template by opening a PR against prometheus/compliance and create test-gmp.yaml in the alert_generator directory. See test-.*.yaml files here for example. Please also add instructions on how to set up as comments in the same file.

If you face issues in running the test suite or understanding any error messages, I am happy to answer your queries.

We plan to publish the results during the third week of May 2022 (during KubeCon EU).

Thanks,
Ganesh (codesome)
Prometheus team

[Jean-Philippe Quéméner]

Update on the alert generator compliance:

Some vendors cannot expose some of the APIs required by the Prometheus alert generator compliance test suite.

We modified the test suite so it can be run by vendors themselves and submit results to Prometheus/CNCF. CNCF will contractually protect against submitting wrong results on purpose.

If there are any questions, please feel free to ask in this thread.

Thanks,

Jean-Philippe Quéméner (johnnyqqqq)