metrics scrap time is increasing over the period of time

[sunils...@gmail.com]

Hi All, 

I am using prometheus to collect metrics from kafka exporter . 

Noticed that when restart kafka , scrap interval is in milli-second .  

But over the period scrap interval increases and then starts seeing error that context deadline exceeded . 

On server I see for exporter port netstat status as :  CLOSE WAIT . 

Need suggestion on this . 

Thanks 

[Julius Volz]

Hi,

I know nothing about the Kafka exporter itself, but a gradually increasing scrape duration could mean that it's exposing more and more time series over time (possibly due to tracking short-lived values in some label dimension, which then blows up over time). Does the number of series from the exporter grow over time? You can get that from the "scrape_samples_scraped" metric that Prometheus records for each target scrape.

If it's not that, then maybe there is some other kind of "leak" in the Kafka exporter.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a8c94608-d1d1-4f43-93ca-acb7412a9e54n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Sagar]

Thanks Julius, 

This is an interesting metric , I can see it increased over the period to 15,000 . It seems to be big . I will check if something can be removed . 

Thanks,