Rate query showing some gaps in data

[Anoop]

Hi,

I have configured a job in Prometheus with 30s as scrape interval.

When I am checking the query with rate function, I can see some gaps in the query. The rate interval for the query is [1m].

Please find the below screenshots to understand the issue. In this case, I executed the query for the time range 1m.

Query without rate function:

Query with rate function:

Can someone please explain why the rate query is behaving like this?

Thanks,

[Ben Kochie]

You are zoomed in to where the resolution of each point on the graph is 1 second. When you are graphing data, Prometheus has to look back the time range of your range vector [1m]. The data points needed for where your graph show a gap are actually outside the window of what you are showing for the raw data.

Basically, you're zoomed in too far with the resolution too high for any useful comparison.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/655a3227-c544-4625-a508-ab419cef927fo%40googlegroups.com.

[Anoop Mohan]

Hi Ben,

Many thanks for your prompt reply.

I think I had chosen 1m as the time range in the test, not 1s. 

Maybe I can share another couple of screenshots, which have the time range of 30m. Please have a look.

Query without rate function:

Query with rate function:

Thanks & Regards,

Anoop Mohan

Mob# +91-7293009486

E-mail : anoopmo...@gmail.com

[Ben Kochie]

I'm guessing you have gaps in your scrapes. I would graph `up` for your target.

It would also be useful to grab the raw samples (as text, not screenshot) from the Console view.

If you query `metric{filter}[30m]` from the Console view, without rate(), it will return the raw data.

[Anoop Mohan]

Hi Ben,

Thanks for checking this.

I do not see any gaps in the scrapes. You can see the images in my first email, which is showing the zoomed data only for the time range 1 minute. You can see the difference in the data for both queries (with and without rate function.) 

Also, I checked the  graph `up` for my target and I do not find any gaps in that. Please find the image below.

Here the console data for the same query for 30 minutes range.

Thanks & Regards,

Anoop Mohan

Mob# +91-7293009486

E-mail : anoopmo...@gmail.com

[Ben Kochie]

It looks like your Prometheus server is extremely unhealthy. If you look closely at the timestamps of your data, you can see they are not spaced evenly. There are sometimes gaps of 45+ seconds between your samples.

Usually this is an indication that Prometheus is overloaded and can't scrape correctly. Possibly resource starved for CPU. Do you apply CPU resources limits to Prometheus?

[Anoop Mohan]

Hi Ben,

Yes I have applied CPU resource limits for Prometheus and the limits for Prometheus is 4 cores. But the average CPU usage is less than 1 core only. Hence, CPU shouldn't be a problem. I can see the memory usage is almost reaching the limit. Let me check if that could be the issue. 

Thank you Ben for providing your suggestion.

Thanks & Regards,

Anoop Mohan

Mob# +91-7293009486

E-mail : anoopmo...@gmail.com

[Ben Kochie]

Prometheus CPU use is bursty, and limits cause problems. It's recommended to avoid CPU limits in Kubernetes in general.

https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes-c5b20bd8a718