alert generation to Alert Manager from microservice

[Nitin Gupta]

Hello,

I would like to explore options, if someone has tried converting the
Alerts into snmp traps on the far end points.
As we understand the Alert rules configured on prometheus, get
executed on metric data based on their frequency of arrival at
prometheus. How to report application based alerts directly to Alert
Manager (AM) ? anybody tried this option to raise the Alerts directly
to AM and also clearing !!

We would like to seek help on approach, generating Alerts from
microservice to AM !!
Thanks
N.g

[Benjamin Ritcey]

You _can_ send alerts directly to Alertmanager:

https://prometheus.io/docs/alerting/clients/

In terms of clearing, Alertmanager alerts "resolve" when the alerts stop firing - i.e., if you stop sending the alert to Alertmanager, it will eventually be considered resolved.

However, if at all possible, you're far better off getting metrics from the microservice into Prometheus and generating alerts from there.

You're also talking about SNMP traps from Alertmanager, which someone has addressed:  https://github.com/maxwo/snmp_notifier

[Nitin Gupta]

Thanks benjamin. 

Limitations through prometheus is data conveyed via metric v/s sending this data directly by mservice.. . to alert manager. 

Is AM can be indicated for the alert raised by mservice to be hold as is, till it (mservice send) clea to alert manager(AM) ? 

Is there a way to achieve!! 

As the mservice doesn't have book-keeping of alert once it's reported out.. . Therefore it can't send same again to AM if it expect to be so.. 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/25f2ae7c-3f8e-4268-a872-9f4ebc2917d2%40googlegroups.com.

[Benjamin Ritcey]

"Is AM can be indicated for the alert raised by mservice to be hold as is, till it (mservice send) clea to alert manager(AM) ?"

No, Alertmanager will consider the alert cleared after resolve_timeout (5m, by default) if the alert has not been updated - that's part of its basic design. 

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Nitin Gupta]

Thanks Benjamin for clarification on handling part.

I have followup question for Alert clearing path, could we expect AM
to emit out single clear Alert in all below scenarios !!
Assuming Alert is raised
1# mservice sends an clear Alert to AM, and AM has cleared the Alert
due to no update from mservice ?
2# mservice sends an clear Alert to AM, and AM has not cleared (e.g.
5m not elapsed), !

resolve_timeout seems to give AM capability to auto-clear the Alerts,
and more aligned with prometheus metric based path, but missing some
basic required pieces for Alert clients !

Still if we could like to use Alert client, How can one disable
[resolve_timeout=FALSE] for set of Alerts, but have global
resolve_timeout set to TRUE ? or something on similar lines !
could we turn-off resolve_timeout for specific set of Alert by
defining some Scoping ! such as type of Alert, service .. could label
be of any use here for scoping ?

so are the suggestions to avoid using AM with Alert clients ?

Thanks

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/31b5b7d3-1c41-49c8-883f-c0a174090ef7%40googlegroups.com.

[Benjamin Ritcey]

Thank you, Chris - I was unaware of the details around endsAt, since I've only used Alertmanager from Prometheus. 

That's timely info, as a co-worker is experimenting with sending Splunk alerts via Alertmanager (since I've already got it wired up to our ticketing system, etc.)

On Wed, Nov 6, 2019 at 4:29 PM Chris Siebenmann <cks.prom...@cs.toronto.edu> wrote:

 My understanding is that this depends on what information you send to
Alertmanager in your alert. If you send an alert with an 'endsAt' and
then do nothing more, Alertmanager will only consider the alert resolved
when it reaches that endsAt time. If you send an alert with no 'endsAt',
Alertmanager sets its endsAt time to now plus your resolve_timeout value
(five minutes by default), and so it will resolve in five minutes from
your notification.

(If you re-send your alert with a different endsAt, or without one,
the new endsAt value (either yours or the computed one) overwrites
the current endsAt. Prometheus normally sends alerts to Alertmanager
with an endsAt a few minutes in the future and then keeps re-sending
them to keep Alertmanager from resolving the alert.)

 If you want to mark an alert as explicitly resolved, I believe that you
re-send it with an endsAt date that is now or in the immediate past.

 If I'm understanding this correctly, you could implement one-shot
alert notification by sending the initial alert with an endsAt date
well into the future. However this isn't necessarily safe; if the
service is unable to send a notification to Alertmanager when the
alert is resolved, the alert will linger on as active. Constantly
renewing the alert serves as a useful signal that the alert really
still is there.

(Of course this has downsides. If you have a bunch of alerts and then
Prometheus dies and stays down for a few minutes, I believe that
Alertmanager will declare all of those alerts resolved even though they
may not be at all. Then when Prometheus comes back with the alerts still
active and re-sends them to Alertmanager, Alertmanager will consider them
to be new alerts and send out new notices for them. In some situations
latching alerts as on when you lose contact with the source triggering
the alert is a feature.)

        - cks

> "Is AM can be indicated for the alert raised by mservice to be hold as
> is, till it (mservice send) clea to alert manager(AM) ?"
>
> No, Alertmanager will consider the alert cleared after               
> resolve_timeout (5m, by default) if the alert has not been updated - 
> that's part of its basic design.                                     
>
>
> On Wednesday, November 6, 2019 at 1:37:19 PM UTC-5, Nitin Gupta wrote:
> >
> > Thanks benjamin.
> >
> > Limitations through prometheus is data conveyed via metric v/s sending
> > this data directly by mservice.. . to alert manager.
> >
> > Is AM can be indicated for the alert raised by mservice to be hold as is,
> > till it (mservice send) clea to alert manager(AM) ?
> > Is there a way to achieve!!
> >
> > As the mservice doesn't have book-keeping of alert once it's reported
> > out.. . Therefore it can't send same again to AM if it expect to be so..
> >
> >
> >
> >

> > On Wed, 6 Nov 2019, 22:57 Benjamin Ritcey, <b...@ritcey.com <javascript:>>

[Nitin Gupta]

Thanks Chris, Benjamin

We are also learning the Alert manager as we go along.
Thanks for sharing the endsAt/ startsAt details and how Alert manager uses them.

How a Alert Client talking to AM through AM-Service method
(NodeIP/NodePort) where AM deployed in Stateful set of 2 instances.
We see that Raise and Clear are sent to different Alert Manager
instances and thus doesn't get cleared.
How a Client should be communicating with AM if not through Service ?

Will share more queries as we encountered in our journey.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAJ1qobSGoopjQZ6O-xYfe%2B4z39Nzg%2BVFAySRRWLfxCqe6aERGg%40mail.gmail.com.

[Adarsh Kumar Pandey]

hie it was a great reading the discussion ..If its not to much to ask ,I would  like to ask for your code snippet of the client ..Can you tell me at what api endpoint and in what format you sent the alerts .. I'm also very new to prometheus and so I'm also trying to understand this ..

Thanks and Cheers! :)