How to get the cumulative/total resource usage of resources say RAM or CPU over a selected duration of time?

[touseef yousuf]

Hi, I want to calculate the total consumption of memory/cpu usage (with underlying kubernetes infrastructure) monotonously over a period of time(lets say 3 months).

I tried query like
sum(container_memory_usage_bytes{namespace=~"$namespace",pod=~"$pod"})/1024/1024/1024

I also tried to use the increase() and sum_over_time() functions to get the cumulative sum of the total resource consumed.
However it doesn’t give me the cumulatively total usage over a selected duration time.It just gives me sum at that time instant.So the graph obtained goes up and down based on usage at that time instant.

Any suggestion for the alternative function/formulla to achieve a sum total usage of resource over a selected time would be appreciated.

Thanks!

[Brian Candler]

I'm not sure what you're trying to achieve.  Memory usage is a gauge.  Are you trying to get a usage figure in "megabyte-hours" or somesuch?

In that case, you can calculate the average memory usage over the period of interest, and then multiply it by the duration of the period.

sum(foo) or average(foo) will give you calculations at a single instant in time, across multiple timeseries.

average_over_time(foo[range]) will give you calcuations separately across each timeseries, over a given range.

sum_over_time() is almost certainly not what you're looking for, as it will get messed up if any samples in the period are missed.

[Nicolae Rosu]

@Brian Candler.,

What Touseef is trying to do is sum all the container CPU and Memory and get a counter over a period of time.

For example,  yesterday 5 Pods with 7 containers were spawned. In total, the cumulative CPU request and Memory request was 300CPU and 500GB of RAM over the last 24h.

This would be useful when you need to calculate the cost for a namespace, as you would have the total CPU/uptime usage and add it to a counter that you can use for more calculations.

[Brian Candler]

I think you have a problem with your units.

"Summing" a gauge doesn't make much sense, because the sum will vary depending on how many data points you take.  What you want is to integrate it over time.  In that case, the problem is with your terminology and your units.  "500GB of RAM over the last 24h" doesn't make much sense, unless you're talking about an average. If you want integrated usage, then your units are in the wrong format: do you mean 500GB-days? 500GB-seconds?

It's like measuring power from your home.  If you put a kettle on, the instantaneous power usage might be 2kW.  But the electricity company doesn't charge you in kW, it charges in kWh (energy consumed).  Leave the kettle on for 30 minutes and your meter will click up to 1kWh.  Leave it on for a whole hour and you'll be charged for 2kWh.

So if you are using 16GB of RAM, and you used it for 1 hour, then your usage is 16GB-hours, or 57600GB-seconds,  or 0.667GB-days.

Anyway: you should find there are already CPU-usage metrics which are counters (in CPU-seconds), so those aren't a problem.

For memory usage, which is just a gauge, you can integrate the memory usage over a time period as follows:

* take the *average* memory usage over the period of interest (say 24 hours).  If this is across multiple machines, then sum the averages.

* convert to GB (divide by 1024^3)

* if you want the answer in GB-days, then you have the answer

* if you want the answer in GB-seconds, then multiply it by 86400 (in this example).

If you want to accumulate this into a counter, then it's harder.  I believe you can use a recording rule to accumulate, by adding to its own previous value (ugh).  Or you could make a custom exporter which handles this logic.