Does AlertManager repeatedly try POST request on web-hook url if web-hook returns 500?
1,568 views
Skip to first unread message
anu....@gmail.com
May 2, 2019, 9:25:55 AM5/2/19
to Prometheus Users
Prometheus alert rule file has one of the rules to check whether service is down or not. And my customized receiver does not handle it, so it is generating some nullPointerException. and alert manager shows 500 Internal Server error for POST request. Though repeat_interval for this group is 1 hour, but still it continuously calls customized receiver request.How does alertManager work ? What kind of response does it expect from POST request ?
Does alert manager process old alerts if alertmanage is reloaded / restarted ?
Lars Schotte
May 2, 2019, 9:30:12 AM5/2/19
to anu....@gmail.com, promethe...@googlegroups.com
Prometheus alert rule file has one of the rules to check whether service is down or not. And my customized receiver does not handle it, so it is generating some nullPointerException. and alert manager shows 500 Internal Server error for POST request. Though repeat_interval for this group is 1 hour, but still it continuously calls customized receiver request.How does alertManager work ? What kind of response does it expect from POST request ?
Does alert manager process old alerts if alertmanage is reloaded / restarted ?
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cdd54c91-f886-4dac-80af-d8dcc8fe8d1e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Simon Pasquier
May 2, 2019, 9:39:15 AM5/2/19
to anu....@gmail.com, Prometheus Users
On Thu, May 2, 2019 at 3:26 PM <anu....@gmail.com> wrote:
>
> Prometheus alert rule file has one of the rules to check whether service is down or not. And my customized receiver does not handle it, so it is generating some nullPointerException. and alert manager shows 500 Internal Server error for POST request. Though repeat_interval for this group is 1 hour, but still it continuously calls customized receiver request.How does alertManager work ? What kind of response does it expect from POST request ?
AlertManager will retry sending the notification in case of a 5xx
>
> Prometheus alert rule file has one of the rules to check whether service is down or not. And my customized receiver does not handle it, so it is generating some nullPointerException. and alert manager shows 500 Internal Server error for POST request. Though repeat_interval for this group is 1 hour, but still it continuously calls customized receiver request.How does alertManager work ? What kind of response does it expect from POST request ?
error as it expects the error to be transient.
On the opposite, a notification getting a 4xx error won't be retried.
>
> Does alert manager process old alerts if alertmanage is reloaded / restarted ?
Simon Pasquier
May 2, 2019, 9:41:58 AM5/2/19
to Lars Schotte, anu....@gmail.com, promethe...@googlegroups.com
On Thu, May 2, 2019 at 3:30 PM 'Lars Schotte' via Prometheus Users
<promethe...@googlegroups.com> wrote:
>
> Yes, I am also onto this.
>
> 1.) What do you mean by prometheus alert rule file? Isn't that
> configured in prometheus.yml where he should fire the alert?
> And do you have some working example with a working alert?
> Because I seem to run into unmarshalling errors on prometheus.yml.
>
> 2.) For for that we would need to understand how alertmanager
> communicates with prometheus. Anyone here has a clue? I havent looked
> into it yet.
Prometheus sends alerts to AlertManager using its alerts API:
https://prometheus.io/docs/alerting/clients/
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f9bfaf485822224fcec603f4da57785c62190673.camel%40hpc.at.
<promethe...@googlegroups.com> wrote:
>
> Yes, I am also onto this.
>
> 1.) What do you mean by prometheus alert rule file? Isn't that
> configured in prometheus.yml where he should fire the alert?
> And do you have some working example with a working alert?
> Because I seem to run into unmarshalling errors on prometheus.yml.
>
> 2.) For for that we would need to understand how alertmanager
> communicates with prometheus. Anyone here has a clue? I havent looked
> into it yet.
Prometheus sends alerts to AlertManager using its alerts API:
https://prometheus.io/docs/alerting/clients/
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
Lars Schotte
May 2, 2019, 9:47:00 AM5/2/19
to spas...@redhat.com, anu....@gmail.com, promethe...@googlegroups.com
OK, so basically he is going over the front gate using port 9090 (or
whichever is configured), sends up a POST JSON request and that's how
he finds out if there are alerts in the pipe.
Right?
Then the question original question is obviously no, like you already
have stated for restarts, because he has no way to know if there were
alerts fired up in the past which were not consumed, as prometheus does
not keep track of transmitting those alerts.
whichever is configured), sends up a POST JSON request and that's how
he finds out if there are alerts in the pipe.
Right?
Then the question original question is obviously no, like you already
have stated for restarts, because he has no way to know if there were
alerts fired up in the past which were not consumed, as prometheus does
not keep track of transmitting those alerts.
Simon Pasquier
May 2, 2019, 10:07:20 AM5/2/19
to Lars Schotte, anu....@gmail.com, promethe...@googlegroups.com
On Thu, May 2, 2019 at 3:47 PM 'Lars Schotte' via Prometheus Users
<promethe...@googlegroups.com> wrote:
>
> OK, so basically he is going over the front gate using port 9090 (or
> whichever is configured), sends up a POST JSON request and that's how
> he finds out if there are alerts in the pipe.
>
> Right?
Yes. Prometheus evaluates alerts periodically and will post them to
<promethe...@googlegroups.com> wrote:
>
> OK, so basically he is going over the front gate using port 9090 (or
> whichever is configured), sends up a POST JSON request and that's how
> he finds out if there are alerts in the pipe.
>
> Right?
<alertmanager address>:9093/api/v1/alerts at regular intervals
(new/resolved alerts will be sent immediately while already firing
alerts will be sent at lower frequency).
>
> Then the question original question is obviously no, like you already
> have stated for restarts, because he has no way to know if there were
> alerts fired up in the past which were not consumed, as prometheus does
> not keep track of transmitting those alerts.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
Lars Schotte
May 2, 2019, 10:16:55 AM5/2/19
to spas...@redhat.com, anu....@gmail.com, promethe...@googlegroups.com
LOL, this is nice.
You seem to have just solved one of my misunderstandings.
I was always under the impression that alertmanager gets those alerts
from prometheus, but now you are saying that it is the other way round.
So prometheus is giving away those alerts to alertmanager and has to be
configured at prometheus side.
My problem now is that I could not find any place where to configure
the port of prometheus but instead I have to configure the port of
alertmanager in order to connect them.
You seem to have just solved one of my misunderstandings.
I was always under the impression that alertmanager gets those alerts
from prometheus, but now you are saying that it is the other way round.
So prometheus is giving away those alerts to alertmanager and has to be
configured at prometheus side.
My problem now is that I could not find any place where to configure
the port of prometheus but instead I have to configure the port of
alertmanager in order to connect them.
Simon Pasquier
May 2, 2019, 10:39:15 AM5/2/19
to Lars Schotte, anu....@gmail.com, promethe...@googlegroups.com
Prometheus sends alerts:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config
anu....@gmail.com
May 3, 2019, 3:22:23 AM5/3/19
to Prometheus Users
Why do AlertManager retry sending the notification in case of a 5xx
error ? It would make continuous POST call on web-hook url though web-hooked service is not responding. If required , then also there should be some delay (like repeat_interval) for retry.
Lars Schotte
May 3, 2019, 4:38:07 AM5/3/19
to spas...@redhat.com, anu....@gmail.com, promethe...@googlegroups.com
Well, first thing that a beginner has to understand is that those
sections are themselves only references on something else.
For example this <alertmanager_config> is good for nothing.
What I need is inside this <alertmanager_config> a section of
<static_config> where I can set the target.
So all this configuration is all but easy and what I am missing there
is an explanation of those. Maybe even a link to YAML as such.
A list of examples would help much.
sections are themselves only references on something else.
For example this <alertmanager_config> is good for nothing.
What I need is inside this <alertmanager_config> a section of
<static_config> where I can set the target.
So all this configuration is all but easy and what I am missing there
is an explanation of those. Maybe even a link to YAML as such.
A list of examples would help much.
Simon Pasquier
May 3, 2019, 6:31:29 AM5/3/19
to anu....@gmail.com, Prometheus Users
On Fri, May 3, 2019 at 9:22 AM <anu....@gmail.com> wrote:
>
> Why do AlertManager retry sending the notification in case of a 5xx error ? It would make continuous POST call on web-hook url though web-hooked service is not responding. If required , then also there should be some delay (like repeat_interval) for retry.
Because 5xx errors are supposed to be transient. If the webhook
>
> Why do AlertManager retry sending the notification in case of a 5xx error ? It would make continuous POST call on web-hook url though web-hooked service is not responding. If required , then also there should be some delay (like repeat_interval) for retry.
receiver can't process the request because it considers it as invalid,
it should respond with a 4xx status code (400 bad request for
instance).
AlertManager retries with exponential back-off to avoid over-loading
the receiver.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
Simon Pasquier
May 3, 2019, 6:35:33 AM5/3/19
to Lars Schotte, anu....@gmail.com, promethe...@googlegroups.com
In [1], there's a link to an example configuration file [2].
[1] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file
[2] https://github.com/prometheus/prometheus/blob/release-2.9/config/testdata/conf.good.yml
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAM6RFu4NFaQT03aWnurZR1uwNxdRHtVtw%2BChEJRs3rBVRvrMEg%40mail.gmail.com.
[1] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file
[2] https://github.com/prometheus/prometheus/blob/release-2.9/config/testdata/conf.good.yml
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a8a370a251436a615327d21bff297e0cb03be1f6.camel%40hpc.at.
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
0 new messages