Prometheus remote read and alerting rules

[peter...@metaswitch.com]

I have a stream of metrics that I would like to display in Prometheus. These arrive as timestamped metrics over an API and I'd like the option of storing them in a TSDB such as InfluxDB which gives me long term storage.

The obvious solution is to run Prometheus using the remote read API for these metrics. However, I am concerned about two possible issues.

• I need to define alerts based on the metrics that are stored in the TSDB and retrieved by remote read. Is this possible? (I saw a presentation on youtube that said not, but not sure if that is up to date.)
• I am concerned that Prometheus might be very inefficient when generating (for example) grafana graphs that span (say) total numbers of requests over the past 48 hours or something similar that requires a lot of raw input data. Is there any guidance on what performance should be here? Obviously, if recording rules can be configured on remotely read data then this worry is much less.

For both of the above, I can prototype, but it would be interesting to know both the current situation and whether it is likely to change in the future.

Thanks, Peter White

[Brian Brazil]

On 20 February 2018 at 12:15, peter.white via Prometheus Users <promethe...@googlegroups.com> wrote:

I have a stream of metrics that I would like to display in Prometheus. These arrive as timestamped metrics over an API and I'd like the option of storing them in a TSDB such as InfluxDB which gives me long term storage.

The obvious solution is to run Prometheus using the remote read API for these metrics. However, I am concerned about two possible issues.

• I need to define alerts based on the metrics that are stored in the TSDB and retrieved by remote read. Is this possible? (I saw a presentation on youtube that said not, but not sure if that is up to date.)

This is possible, but not recommended generally. 

• I am concerned that Prometheus might be very inefficient when generating (for example) grafana graphs that span (say) total numbers of requests over the past 48 hours or something similar that requires a lot of raw input data. Is there any guidance on what performance should be here? Obviously, if recording rules can be configured on remotely read data then this worry is much less.

It'd be best to have Prometheus scrape the targets directly, rather than coming from some other push-style monitoring system.

Brian

 

For both of the above, I can prototype, but it would be interesting to know both the current situation and whether it is likely to change in the future.

Thanks, Peter White

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/25bc46f8-5519-4552-8578-3ac3dd394fd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io