Re: [prometheus-users] Seeking advice on `instance` label values for Prometheus metrics in Kubernetes

[Ben Kochie]

There is nothing to handle, the instance/pod IP is required for uniqueness tracking. Different instances of the same pod need to be tracked individually. In addition, most Deployment pods are going to get new generated pod names every time anyway.

Prometheus compacts memory every 2 hours, so old data is flushed out of memory.

I would recommend upgrading to the latest release, there have been several memory use improvements since v2.40.7.

On Wed, Aug 16, 2023 at 9:30 AM Peter Nguyễn <win...@gmail.com> wrote:

Hi all,

I have a Prometheus Pod (v2.40.7) running on our Kubernetes (k8s) cluster for metric scraping from multiple k8s targets.

Recently, I have observed that whenever I restart a target (a k8s Pod) or perform a Helm upgrade, the memory consumption of Prometheus keeps increasing. After diving, I discovered that the cause lies in the dynamic nature of the values assigned to the instance and pod_name labels for each time series.

The instance label currently uses the __address__ format, which is <pod_IP>:port. Consequently, whenever a Pod gets restarted, it receives a new allocated IP address, resulting in a new value for the instance label. Besides, we have pod_name label set to Kubernetes pod name so when a Pod does not belong to a StatefulSet controller, it will receive a new pod name upon restart.

Given these circumstances, I am writing to seek advice on the best practices for handling these label values, particularly for the instance label.

Thank you.

Regards, Vu

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d9a8b926-70f7-4865-bf2b-b8e61336aaa1n%40googlegroups.com.