dynamically creating an alert in prometheus

[Ashish Jain]

Hi,

I have a scenario where I have my prometheus server running on a remote host say HOSTA, I have a client which is running on remote HOSTB. Using client on HOSTB I want to dynamically create rules in prometheus. I see following challenges in doing so and would like prometheus developers to provide there inputs

1) Adding a alert file in HOSTA
2) Dynamically updating the prometheus.yml on HOSTA to include new alert rule

The approach which I currently see to achieve this is to
1) Write a alert file on HOSTB
2) Create an updated prometheus.yml file on HOSTB
3) SCP both the files from HOSTB to HOSTA at appropriate location
4) Reload prometheus configuration

Is there an alternative which is available in form of an API endpoint exposed by prometheus to achieve this? If not is there a better approach which can be used to achieve this up?

Regards
Ashish

[Ben Kochie]

No, this type of thing is best done with some kind configuration management system.  For example with something like ansible or chef.  We solve this problem for our developers by allowing them to check in rules files into a source control repository (git, for example) and then automatically upload these into chef which deploys and reloads the prometheus server if necessary.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ashish Jain]

Ben thanks, this is helpful.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

[Johannes Ziemke]

If you don't have/want full blown config management, you can also use consul + consul-template. You would simply write the alert definitions to the consul k/v store and consul-template takes care of updating the alerting rules and reloading prometheus.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

[Ashish Jain]

Hi Johannes,

I have never used consul but will love to explore the option suggested by you. Is there any example or reference you can provide which talks about a similar scenario?

Regards
Ashish

[xgt...@gmail.com]

This becomes even more interesting when you are generating alerts on the fly through a service discovery mechanism in Prometehus say like EC2 especially if you have a mix of on-demand and spot instances.

Right now I am following an approach of

a) Have a Python script on host A - my prometheus host to populate my rules file for newly registered containers. This can be done by scraping the targets registered through the SD.

One problem with this approach is cleaning up hosts when they die instead of alerting in Prometheus.

[Björn Rabenstein]

On 13 December 2016 at 10:49, <xgt...@gmail.com> wrote:
> This becomes even more interesting when you are generating alerts on the fly through a service discovery mechanism in Prometehus say like EC2 especially if you have a mix of on-demand and spot instances.

In usual scenarios, you should be able to create PromQL alerting
expressions that don't have to be modified if you add instances on the
fly. That's kind of the point of Prometheus style monitoring &
alerting.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B