Huge amount of metrics metadata slows down Grafana Dashboard editing

[Florian Schoetz]

Hi all,

currently running Prometheus v2.23.0 in Docker.

Facing the issue that there were multiple thousand metrics created for each pod running in a kubernetes cluster (for each pod a series was created with the pod id).

When trying to edit now a dashboard in grafana it loads the metrics metadata for each configured query, which results in downloading 3MB of metadata for each query right now. This makes editing really slow and also the metadata handling in Grafana seems not to be optimal.

Is there any possibility to get rid of the metadata? We already deleted the series via the admin api and cleaned tombstones, but metadata is still there and we are aware that its not supposed to be fixed soon (https://github.com/prometheus/prometheus/issues/7932) - but anyway maybe someone had a similar problem and can provide a workaround.