Different Prometheus metrics for different trials

[intelops appstek]

Prometheus metrics are different in different iterations even though application, load testing, hardware configurations are same in all cases. Kindly, anyone let me know the reason for this.

I have created a container with simple "Hello World" web application and load testing it with 1000 users. Container is deployed in Digital Ocean cloud environment and container metrics are gathered using Prometheus. I'm capturing the maximum CPU and maximum memory utilization.

For every iteration(where I load test with 1000 users) I run, I'm getting different maximum CPU and Memory utilization metrics.

My assumption is, if the test parameters and hardware configurations remains same, the maximum utilization metrics also should remain same.

Question - why I'm getting different metrics?

[Stuart Clark]

I'd not expect identical results, but I'd expect things to be pretty
similar. The real world intrinsically contains an amount of randomness,
so things should be slightly different - scrapes happen at slight
different points, data takes slightly longer or shorter to read from the
disk or network, background processes kick in at slightly different
points, etc.

Are you seeing wildly different results between your runs?

--
Stuart Clark

[intelops appstek]

Please find the attached excel which consists of training data of Kubernetes i.e. Hardware Configuration and Prometheus Metrics. 

In excel, let's take an example of [11m, 105MiB] hardware configuration in which Prometheus metrics are different in different iterations even though load testing is same in all cases.

Metrics for [11m, 105MiB] are  [9.9, 0.36], [8.48, 0.26], [6.13, 0.33], [9.69. 0.48], [17.9, 0.43].

Please let me know, if I get Prometheus metrics in this way only.

Expressions to capture metrics from Prometheus in browser:

max (rate(container_cpu_usage_seconds_total{pod=~"hello-world*-.*"}[5m]))  --- For CPU

avg(container_memory_usage_bytes{pod_name=~"hello-world-68f66b886-xz2pm"})  --- For Memory