Stop prometheus scraping metrics and sending alerts when services are upgrading

[huihui...@gmail.com]

Hi,

I am new to prometheus. I am using prometheus to scrape metrics from services and monitoring some api by altermanager,.

Here is my problem:  I want prometheus stop scraping and stop sending alerts when the services are upgrading, and restart scraping and alert after upgrade is dong.  But I don't want to kill or stop prometheus/altermanager manually, is there another way to do this?

[Conor Broderick]

Why do you want to stop monitoring during an upgrade? 

When you say you want Prometheus to stop sending out alerts, is it the Alertmanager that you mean? 

Prometheus generates the alerts and sends them to the Alertmanager which then manages those alerts in various ways such as sending out notifications via methods such as email, and messaging services.

If this is the case I believe one would typically silence their alerts during the upgrade period to achieve this. You can silence alerts from the alert manager itself. 

More info on the silencing of alerts can be found here and here.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5fd141e5-9049-476f-8823-ad9430441fd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

[huihui...@gmail.com]

Thank you for your answer! 

I just neglected the silences on altermanager UI......

I still have two questions:

First: When I add new silence, I must select the start time and end time, what if I don't know the end time? Because the time spent on service upgrades is uncertain.

First if I want to match all alters, should the name and value all be *? Just as below 

Second 

On Tuesday, May 23, 2017 at 6:46:53 PM UTC+8, Conor Broderick wrote:

Why do you want to stop monitoring during an upgrade? 

When you say you want Prometheus to stop sending out alerts, is it the Alertmanager that you mean? 

Prometheus generates the alerts and sends them to the Alertmanager which then manages those alerts in various ways such as sending out notifications via methods such as email, and messaging services.

If this is the case I believe one would typically silence their alerts during the upgrade period to achieve this. You can silence alerts from the alert manager itself. 

More info on the silencing of alerts can be found here and here.

On 23 May 2017 at 11:28, <huihui...@gmail.com> wrote:

Hi,

I am new to prometheus. I am using prometheus to scrape metrics from services and monitoring some api by altermanager,.

Here is my problem:  I want prometheus stop scraping and stop sending alerts when the services are upgrading, and restart scraping and alert after upgrade is dong.  But I don't want to kill or stop prometheus/altermanager manually, is there another way to do this?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5fd141e5-9049-476f-8823-ad9430441fd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

[Julius Volz]

On Tue, May 23, 2017 at 3:50 PM, <huihui...@gmail.com> wrote:

Thank you for your answer! 

I just neglected the silences on altermanager UI......

I still have two questions:

First: When I add new silence, I must select the start time and end time, what if I don't know the end time? Because the time spent on service upgrades is uncertain.

You can set some duration that should be longer than you need and then delete the silence once you finish your maintenance / upgrade.

 

First if I want to match all alters, should the name and value all be *? Just as below 

The first field is the label name on which to match, the second one is the label value. If you check the "Regex" checkbox, you an also supply a regex (but only in the value field). "*" is not a valid regex though, you'd have to use ".*". However, you probably want to match something more specific, depending on the labels of your alerts.

 

Second 

On Tuesday, May 23, 2017 at 6:46:53 PM UTC+8, Conor Broderick wrote:

Why do you want to stop monitoring during an upgrade? 

When you say you want Prometheus to stop sending out alerts, is it the Alertmanager that you mean? 

Prometheus generates the alerts and sends them to the Alertmanager which then manages those alerts in various ways such as sending out notifications via methods such as email, and messaging services.

If this is the case I believe one would typically silence their alerts during the upgrade period to achieve this. You can silence alerts from the alert manager itself. 

More info on the silencing of alerts can be found here and here.

On 23 May 2017 at 11:28, <huihui...@gmail.com> wrote:

Hi,

I am new to prometheus. I am using prometheus to scrape metrics from services and monitoring some api by altermanager,.

Here is my problem:  I want prometheus stop scraping and stop sending alerts when the services are upgrading, and restart scraping and alert after upgrade is dong.  But I don't want to kill or stop prometheus/altermanager manually, is there another way to do this?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5fd141e5-9049-476f-8823-ad9430441fd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5fe51674-2955-41a2-9a3b-9dd31a0eaa25%40googlegroups.com.

[huihui...@gmail.com]

I wonder if there is any alert API exposed to silence the alerts