Millisecond Scrape Interval Configuration Gives Errors

[Hardik Surana]

Hey!

I am currently using Prometheus - 1.7.0 for profiling docker containers in Kubernetes - 1.10.0. I wish to collect network usage data for each container in 1-millisecond intervals. The Prometheus documentation page (https://prometheus.io/docs/prometheus/latest/configuration/configuration/) mentions that the scrape_interval can be set to milliseconds according to the regex for the <duration> placeholder. But, when I created the below-mentioned configuration, I got a context deadline exceeded error and a dial tcp 10.244.2.151:8080: i/o timeout error in the Status -> Targets page in the Prometheus dashboard.

Part of the Configuration File:

apiVersion: v1

data:

prometheus.yaml: |

global:

scrape_interval: 1ms

scrape_timeout: 1ms

evaluation_interval: 1ms

I used this file for configuring Prometheus and grafana with kubernetes: https://github.com/giantswarm/kubernetes-prometheus/blob/master/manifests-all.yaml

Can someone please guide as to what the right configuration should be for getting the data with 1-millisecond granularity?

Or, if there is any other way to collect data at this interval scale?

[Jeffrey Ollie]

On Fri, Jun 8, 2018 at 5:45 PM, Hardik Surana <hardiks...@gmail.com> wrote:

Can someone please guide as to what the right configuration should be for getting the data with 1-millisecond granularity?

Or, if there is any other way to collect data at this interval scale?

I'd say that Prometheus is not going to be the right tool to use. Prometheus appears to be re-using TCP connections so you don't have to worry about the three-way handshake but the HTTP request/reply transaction between Prometheus and node_exporter still takes tens of milliseconds on the hosts I looked at.

If I was building this I'd probably use InfluxDB as the backend and write a custom tool to batch write the data.

--

Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.

[Ben Kochie]

Yes, batching is the only sane way to collect this amount of data.

Just doing some rough numbers, if you wanted to stream this data for 10 containers, with in/out bytes/packets, we're talking over 600kB per second in just raw sample data (16 bytes per sample). That's not including any metric metadata.

Prometheus / TSDB only store 1ms accurate timestamps, so it's probably not a good fit there either. InfluxDB supports 1ns timestamps, IIRC.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD-DQxT1A48e7Lwjkv1BSW4%3D3b8KkZ%3DVYQ0LWbjhe_x63uuh4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Richard Hartmann]

As an aside, Prometheus 2.x (current release) should be quicker than
1.x; you're still easily missing an order of magnitude in network
speeed.

You could also try and batch process this into local files and then
expose the data with timestamps later.


Richard

[Brian Brazil]

On 9 June 2018 at 06:53, Ben Kochie <sup...@gmail.com> wrote:

Yes, batching is the only sane way to collect this amount of data.

Just doing some rough numbers, if you wanted to stream this data for 10 containers, with in/out bytes/packets, we're talking over 600kB per second in just raw sample data (16 bytes per sample). That's not including any metric metadata.

Prometheus / TSDB only store 1ms accurate timestamps, so it's probably not a good fit there either.

On top of that, you can't do much with data with a resolution more than 1s in PromQL. At most you can get the raw data back out again. Prometheus is not suitable for this use case.

 

InfluxDB supports 1ns timestamps, IIRC.

Yip. Influx would be a better choice for this.

Brian

 

On Sat, Jun 9, 2018 at 5:02 AM Jeffrey Ollie <je...@ocjtech.us> wrote:

On Fri, Jun 8, 2018 at 5:45 PM, Hardik Surana <hardiks...@gmail.com> wrote:

Can someone please guide as to what the right configuration should be for getting the data with 1-millisecond granularity?

Or, if there is any other way to collect data at this interval scale?

I'd say that Prometheus is not going to be the right tool to use. Prometheus appears to be re-using TCP connections so you don't have to worry about the three-way handshake but the HTTP request/reply transaction between Prometheus and node_exporter still takes tens of milliseconds on the hosts I looked at.

If I was building this I'd probably use InfluxDB as the backend and write a custom tool to batch write the data.

--

Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAD-DQxT1A48e7Lwjkv1BSW4%3D3b8KkZ%3DVYQ0LWbjhe_x63uuh4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABbyFmrkCYS6VKeRmWhhwxhA2R1ybeA33_q_uG2G%2BqZJP7QKqw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io