Removing label id from kubelet job

[Thomas Bianchi]

Hi, in a big kubernetes cluster I have a very huge prometheus. Trying to contain his memory requests and OOM, I noticed that in the kubelet metrics there a very high cardinality label: "id". So added this configuration:

metric_relabel_configs:
  - action: labeldrop   
  regex: "id"         
                        

I maneged like this to extremely reduce memory consuption, apparently at no costs, all my dashboard continues to work.
However sending metrics via remote_write to Cortex lead to this:
Error on senting writerequest to Cortex: code 400 Body user=mia-platform: series={__name__=\"container_cpu_system_seconds_total\", [...] }, timestamp=2021-05-12T09:42:06.89Z: out of order sample\n"

For this metric, totally removing id label lead to be not unique:
container_fs_writes_total{container="",device="/dev/sda",id="/system.slice/docker.service",image="",name="",namespace="",pod=""} 3.794105e+06 1620812171482 container_fs_writes_total{container="",device="/dev/sda",id="/system.slice/kubelet.service",image="",name="",namespace="",pod=""} 33558 1620812167445

Is there a method to contains this high cardinality lead Prometheus to use an accettable amount of RAM?

Thanks

Thomas