prometheus snmp pre-collector

[Виталий Ковалев]

Hello. I use prometheus + snmp_exporter to monitor network swithes(D-Link and Huawei) with snmp.

I have an issue with huawei switches - some of them can't give ports utilization and i have to calculate it manualy.

As i know prometheus can't setup per job data retention(Is it rigth?), and i decided to start another one prometheus server which will collect IfInOctets/IfOutOctets every 5 seconds, and calculate utilization, when first server will scrape second server.

I dont need to keep IfInOctets/IfOutOctets too long,and i decided setup storage.tsdb.min-block-duration to 30 m and storage.tsdb.retention.time to 1d.

So, the questions are:

Is it right way?

What difference between storage.tsdb.min-block-duration and storage.tsdb.max-block-duration?

Will prometheus calculate utilization for metrics which still in memory?

[Brian Candler]

IMO, you shouldn't be doing this at all.  For all devices, simply store the raw ifHCInOctets / if HCOutOctets (etc) in prometheus.  Then when you want to graph it or alert on it, use a PromQL query using rate() or irate().

Storing the raw data keeps you closer to the source of truth, and makes it easier to debug, be able to observe counter resets, etc.  You can also answer questions like "what was the average rate over 5 seconds? Over 30 seconds? Over 5 minutes?" using the same source data.

If for some reason you end up with some specific queries that are too expensive to run, then you can use recording rules to generate the results or partial results.  But remember there are other ways to improve query performance for dashboards, e.g. https://github.com/Comcast/trickster

[Ben Kochie]

This is a classic use case for Federation.

You have a high frequency scrape server that keeps the raw data for some amount of time. I would probably keep 1 month around, say 35 days of raw data. It depends a bit on how much you're scraping.

Then for long-term history, you have a set of recording rules like this:

groups:
- name: network bandwidth
  interval: 1m
  rules:
  - record: instance:IfInOctets:rate1m
    expr: rate(IfInOctets[1m])
  - record: instance:IfOutOctets:rate1m
    expr: rate(IfOutOctets[1m])

This will give you 1 minute averaged data. The Federation server can then scrape only these rules and store them for much longer periods of time.

There's no need to adjust --storage.tsdb.min-block-duration, the latest Prometheus release (2.19.0) now writes out head chunk data via mmap, eliminating the memory penalty for higher frequency scrapes.

All of that said, you might consider looking at Thanos. It provides tiered downsampling and resolution retention. This allows you to keep raw data for up to some amount of time, but it will transparently provide downsample data for much longer time ranges. Providing the best of both worlds, fast raw scrapes, and forever retention of raw counter data.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/08d2ca92-8f2a-4bcc-a98a-3e6d4530e3d4o%40googlegroups.com.

[kvp9...@gmail.com]

But would is there memory  and disk usage overhead?

четверг, 18 июня 2020 г. в 23:25:51 UTC+12, b.ca...@pobox.com:

[Ben Kochie]

Disk yes, memory no. Prometheus uses mmap to read the on-disk data.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c4b5f9fc-a40a-4198-88e0-2d3ca3ce1e33n%40googlegroups.com.

[kvp9...@gmail.com]

Thanks!

пятница, 19 июня 2020 г. в 18:46:00 UTC+12, sup...@gmail.com: