Prometheus exposition format and HTTP response compression problem

40 views
Skip to first unread message

Isabel Noronha

unread,
May 27, 2020, 8:13:05 AM5/27/20
to Prometheus Users
Hello,

I see very high CPU consumption (beyond 700%) by cAdvisor.
Attached is a png file that clearly shows that it's a problem with compressing and exposing metrics in Prometheus format.

Prometheus v2.18
cAdvisor v0.35
No.of containers 2K.

Can I store scraped metrics in an external DB like VictoriaMetrics and query using promql and grafana to view dashboards?

Scenario:
target1,target 2 
Both targets have cAdvisor running.
I want to scrape all the cAdvisor metrics and send them to TSDB (say Victoriametrics).
Then use this DB to run promql queries from http://localhost:9090

Is this possible?

Thanks,
Regards,
Isabel





profile.png

Ben Kochie

unread,
May 27, 2020, 11:23:05 AM5/27/20
to Isabel Noronha, Prometheus Users
You don't need an external TSDB for a small setup like this. Prometheus has one built in that works very well.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b64ecc3c-ae9b-491b-8726-6ea8372d50ec%40googlegroups.com.

Isabel Noronha

unread,
May 28, 2020, 1:38:13 AM5/28/20
to Prometheus Users
The infra is going to be with 20 servers with each server running >2K containers 
So  I see CPU surge in case od cAdvisor when it exposes metrics in Prometheus format and compresses it.
I cannot afford to crash my infra nor lose scraped metrics.

Ben Kochie

unread,
May 28, 2020, 5:06:30 AM5/28/20
to Isabel Noronha, Prometheus Users
This sounds like an issue with cAdvisor, I recommend opening an issue there.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a8bdf34a-cd0c-449b-8207-8063d1b72a80%40googlegroups.com.

Isabel Noronha

unread,
May 28, 2020, 6:40:33 AM5/28/20
to Prometheus Users

Yes already done that!
Seems like the no.of containers is also quite large.
It is recommended from their end that either disable Prometheus compression or use another TSDB like VictoriaMetrics.
User storage flags in cAdvisor and store metrics in VictoriaMetrics and then use Prometheus remote_read.

On Wednesday, May 27, 2020 at 5:43:05 PM UTC+5:30, Isabel Noronha wrote:
Reply all
Reply to author
Forward
0 new messages