Get charts by API

186 views
Skip to first unread message

Dmitriy Nova

unread,
Nov 10, 2017, 11:52:55 AM11/10/17
to Prometheus Users

Is there a way to get charts using API?

We have notifications using messenger with small specialized service. We wish to have charts for fired alerts in notifications like Grafana does. Is there API to get chart as an image that we will be able to get from Prometheus and attach it to the notification?

Brian Brazil

unread,
Nov 10, 2017, 11:55:51 AM11/10/17
to Dmitriy Nova, Prometheus Users
On 10 November 2017 at 16:52, Dmitriy Nova <stalk...@gmail.com> wrote:

Is there a way to get charts using API?

We have notifications using messenger with small specialized service. We wish to have charts for fired alerts in notifications like Grafana does. Is there API to get chart as an image that we will be able to get from Prometheus and attach it to the notification?

There isn't, but you could build something on top of the query APIs we do offer.

In general I'd suggest including a link to a dashboard with your notifications rather than a graph.

--

Dmitriy Nova

unread,
Nov 11, 2017, 6:04:34 AM11/11/17
to Prometheus Users
Got it.

But it could be a useful feature for Prometheus. Prometheus can draw graphs but to notify someone with graph you need also Grafana =( In this case do I really need Alertmanger for alerts or simply use Grafana to send alerts with graphs? =)

PS: my Grafana is behind VPN but I can send graphs to messenger with graphs.

пятница, 10 ноября 2017 г., 19:55:51 UTC+3 пользователь Brian Brazil написал:

Ben Kochie

unread,
Nov 11, 2017, 7:04:06 AM11/11/17
to Dmitriy Nova, Prometheus Users
I think what we would want here is a way to have the Alertmanager GET a URL in an alert annotation and include that output as an inline, or an attachment, to the notification.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e1abe89c-807d-451e-a4be-4bf8ea236137%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brian Brazil

unread,
Nov 11, 2017, 12:54:39 PM11/11/17
to Ben Kochie, Dmitriy Nova, Prometheus Users
On 11 November 2017 at 12:04, Ben Kochie <sup...@gmail.com> wrote:
I think what we would want here is a way to have the Alertmanager GET a URL in an alert annotation and include that output as an inline, or an attachment, to the notification.

That's been suggested before, it's not viable on bandwidth grounds. A link to a dashboard is nice and cheap, and doesn't go out of date.

Brian
 

For more options, visit https://groups.google.com/d/optout.



--

Ben Kochie

unread,
Nov 11, 2017, 12:56:33 PM11/11/17
to Brian Brazil, Dmitriy Nova, Prometheus Users
Not at the prometheus side, but as part of the alertmanager side.  The theory is the alertmanager is inside a backend network and can pull data, where as say an email client or slack wouldn't.  Then it would only be one query at time of alert template execution.

Nicholas Capo

unread,
Nov 11, 2017, 1:16:40 PM11/11/17
to Ben Kochie, Brian Brazil, Dmitriy Nova, Prometheus Users

I much prefer the dashboard link method.

But, for what it's worth, I run my Alertmanagers's as a cross-datacenter mesh (one or more AM in each DC), meaning that it is possible that the AM sending an alert (e.g. to pagerduty) is actually quite far away from the datacenter that originated the alert.

While I think it's reasonable to expect Prometheus servers to be proximate to their scrape targets, that may or may not be true of the Alertmanager when sending an alert.

Nicholas


On Sat, Nov 11, 2017, 11:56 Ben Kochie <sup...@gmail.com> wrote:
Not at the prometheus side, but as part of the alertmanager side.  The theory is the alertmanager is inside a backend network and can pull data, where as say an email client or slack wouldn't.  Then it would only be one query at time of alert template execution.
On Sat, Nov 11, 2017 at 6:54 PM, Brian Brazil <brian....@robustperception.io> wrote:
On 11 November 2017 at 12:04, Ben Kochie <sup...@gmail.com> wrote:
I think what we would want here is a way to have the Alertmanager GET a URL in an alert annotation and include that output as an inline, or an attachment, to the notification.

That's been suggested before, it's not viable on bandwidth grounds. A link to a dashboard is nice and cheap, and doesn't go out of date.

Brian
 
On Sat, Nov 11, 2017 at 12:04 PM, Dmitriy Nova <stalk...@gmail.com> wrote:
Got it.

But it could be a useful feature for Prometheus. Prometheus can draw graphs but to notify someone with graph you need also Grafana =( In this case do I really need Alertmanger for alerts or simply use Grafana to send alerts with graphs? =)

PS: my Grafana is behind VPN but I can send graphs to messenger with graphs.

пятница, 10 ноября 2017 г., 19:55:51 UTC+3 пользователь Brian Brazil написал:
On 10 November 2017 at 16:52, Dmitriy Nova <stalk...@gmail.com> wrote:

Is there a way to get charts using API?

We have notifications using messenger with small specialized service. We wish to have charts for fired alerts in notifications like Grafana does. Is there API to get chart as an image that we will be able to get from Prometheus and attach it to the notification?

There isn't, but you could build something on top of the query APIs we do offer.

In general I'd suggest including a link to a dashboard with your notifications rather than a graph.

--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpOv5L538fv8M71EXyLOeMebEuh6q9TgdRFkby%2B1TMRcA%40mail.gmail.com.

Brian Brazil

unread,
Nov 11, 2017, 1:18:41 PM11/11/17
to Ben Kochie, Dmitriy Nova, Prometheus Users
On 11 November 2017 at 17:56, Ben Kochie <sup...@gmail.com> wrote:
Not at the prometheus side, but as part of the alertmanager side.  The theory is the alertmanager is inside a backend network and can pull data, where as say an email client or slack wouldn't.  Then it would only be one query at time of alert template execution.

That 's still performed at every alert template execution, which could easily be occurring many times per second across an entire AM. For a 1MB image that could eat a 100Mbit network link.



--

Dmitriy Nova

unread,
Nov 15, 2017, 11:13:00 AM11/15/17
to Prometheus Users
Made screenshot from grafana - 36kb. Hard to understand when you need 1MB picture for graph...

суббота, 11 ноября 2017 г., 21:18:41 UTC+3 пользователь Brian Brazil написал:
On 11 November 2017 at 17:56, Ben Kochie <sup...@gmail.com> wrote:
Not at the prometheus side, but as part of the alertmanager side.  The theory is the alertmanager is inside a backend network and can pull data, where as say an email client or slack wouldn't.  Then it would only be one query at time of alert template execution.

That 's still performed at every alert template execution, which could easily be occurring many times per second across an entire AM. For a 1MB image that could eat a 100Mbit network link.

Brian
 
On Sat, Nov 11, 2017 at 6:54 PM, Brian Brazil <brian....@robustperception.io> wrote:
On 11 November 2017 at 12:04, Ben Kochie <sup...@gmail.com> wrote:
I think what we would want here is a way to have the Alertmanager GET a URL in an alert annotation and include that output as an inline, or an attachment, to the notification.

That's been suggested before, it's not viable on bandwidth grounds. A link to a dashboard is nice and cheap, and doesn't go out of date.

Brian
 
On Sat, Nov 11, 2017 at 12:04 PM, Dmitriy Nova <stalk...@gmail.com> wrote:
Got it.

But it could be a useful feature for Prometheus. Prometheus can draw graphs but to notify someone with graph you need also Grafana =( In this case do I really need Alertmanger for alerts or simply use Grafana to send alerts with graphs? =)

PS: my Grafana is behind VPN but I can send graphs to messenger with graphs.

пятница, 10 ноября 2017 г., 19:55:51 UTC+3 пользователь Brian Brazil написал:
On 10 November 2017 at 16:52, Dmitriy Nova <stalk...@gmail.com> wrote:

Is there a way to get charts using API?

We have notifications using messenger with small specialized service. We wish to have charts for fired alerts in notifications like Grafana does. Is there API to get chart as an image that we will be able to get from Prometheus and attach it to the notification?

There isn't, but you could build something on top of the query APIs we do offer.

In general I'd suggest including a link to a dashboard with your notifications rather than a graph.

--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--




--

fi...@freigeist.org

unread,
Jul 31, 2018, 6:44:42 AM7/31/18
to Prometheus Users
Hi everyone,

so I've just got the same requirement. We want a graph of what caused a alert to fire in our Slack messages, similar to what others like Grafana and Datadog offers.
This is considered an important UX aspect and the main reason why people here consider using grafana to alert instead of the alertmanager pipeline.

Correct me if I'm wrong, but currently it doesn't seem to be possible to build this without changing the alertmanager since it doesn't know anything about what expression caused an alert to fire.

But it shouldn't be too hard to pass this to the alertmanager's template struct, correct? With that in place, one could build a simple graph renderer that takes a prometheus expression and returns a png. That can be used in slack's attachment as image url.

That would be enough to make the team happy but even better would be a graph that shows the threshold and where it's been reached, again similar to grafana/datadog. But since Prometheus alerts have no notion of threshold, I'm not sure that's doable in a sane way without changing the whole alert semantics. Maybe though we could split the expression on the binary operator (if any) and return each sub expression. Graphing them should give you a graph of your observed metric and it's threshold as separate timeseries.

Curious if people have though about this too. It's one of the thing I imaging will come up more and more in the future, given that other monitoring systems provide this and people seem to like/want/need that.

I also considered including a dashboard link in the notifications, which certainly is an alternative but requires significant more ongoing work to keep alerts and dashboards in sync.
That being said, maybe Grafana could drive the alertmanager configuration via it's alerts system. But this again requires significant changes, so also curious how people feel about this option.

Brian Brazil

unread,
Jul 31, 2018, 6:54:56 AM7/31/18
to Johannes Ziemke, Prometheus Users
On 31 July 2018 at 11:44, <fi...@freigeist.org> wrote:
Hi everyone,

so I've just got the same requirement. We want a graph of what caused a alert to fire in our Slack messages, similar to what others like Grafana and Datadog offers.
This is considered an important UX aspect and the main reason why people here consider using grafana to alert instead of the alertmanager pipeline.

Correct me if I'm wrong, but currently it doesn't seem to be possible to build this without changing the alertmanager since it doesn't know anything about what expression caused an alert to fire.

But it shouldn't be too hard to pass this to the alertmanager's template struct, correct? With that in place, one could build a simple graph renderer that takes a prometheus expression and returns a png. That can be used in slack's attachment as image url.

You can pass whatever you like in an annotation.
 

That would be enough to make the team happy but even better would be a graph that shows the threshold and where it's been reached, again similar to grafana/datadog. But since Prometheus alerts have no notion of threshold, I'm not sure that's doable in a sane way without changing the whole alert semantics. Maybe though we could split the expression on the binary operator (if any) and return each sub expression. Graphing them should give you a graph of your observed metric and it's threshold as separate timeseries.

Curious if people have though about this too. It's one of the thing I imaging will come up more and more in the future, given that other monitoring systems provide this and people seem to like/want/need that.

This issue is that Prometheus alerts aren't simple thresholds on single variables as is common in other monitoring systems, so this isn't possible in general.
 

I also considered including a dashboard link in the notifications, which certainly is an alternative but requires significant more ongoing work to keep alerts and dashboards in sync.

That's what I'd recommend, and I'd expect it to be less effort to maintain as you don't need to copy expressions and thresholds around as they change.

Brian
 
That being said, maybe Grafana could drive the alertmanager configuration via it's alerts system. But this again requires significant changes, so also curious how people feel about this option.



--

Johannes Ziemke

unread,
Jul 31, 2018, 8:52:36 AM7/31/18
to Brian Brazil, Prometheus Users
On Tue, Jul 31, 2018 at 12:54 PM Brian Brazil <brian....@robustperception.io> wrote:
On 31 July 2018 at 11:44, <fi...@freigeist.org> wrote:
But it shouldn't be too hard to pass this to the alertmanager's template struct, correct? With that in place, one could build a simple graph renderer that takes a prometheus expression and returns a png. That can be used in slack's attachment as image url.

You can pass whatever you like in an annotation.

But I don't want to duplicate the alerting expression into the annotation. Can't we make the expression part of the alert message sent to the alertmanager and have it expose that to the config templates like GeneratorURL? 

Brian Brazil

unread,
Jul 31, 2018, 9:04:22 AM7/31/18
to Johannes Ziemke, Prometheus Users
On 31 July 2018 at 13:52, Johannes Ziemke <fi...@freigeist.org> wrote:

You could use templating from configuration management to do so, if you want to avoid duplication. Such information is going to be little use to you on its own, as you'd need at least a PromQL parser to do something with it.

--

Johannes Ziemke

unread,
Jul 31, 2018, 9:10:19 AM7/31/18
to Brian Brazil, Prometheus Users
I don't need a promql parser to just send the expression to query_range and render an image from that. Also I don't want to introduce configuration management just for this. I'm doing very well without since several years now.

On Tue, Jul 31, 2018 at 3:04 PM Brian Brazil <brian....@robustperception.io> wrote:
On 31 July 2018 at 13:52, Johannes Ziemke <fi...@freigeist.org> wrote:
On Tue, Jul 31, 2018 at 12:54 PM Brian Brazil <brian....@robustperception.io> wrote:
On 31 July 2018 at 11:44, <fi...@freigeist.org> wrote:
But it shouldn't be too hard to pass this to the alertmanager's template struct, correct? With that in place, one could build a simple graph renderer that takes a prometheus expression and returns a png. That can be used in slack's attachment as image url.

You can pass whatever you like in an annotation.

But I don't want to duplicate the alerting expression into the annotation. Can't we make the expression part of the alert message sent to the alertmanager and have it expose that to the config templates like GeneratorURL? 
Reply all
Reply to author
Forward
0 new messages