Dropped sample for series that was not explicitly dropped via relabelling

[Julio Leal]

Hi folks.

We have configured a remote write to Victoria Metrics cluster and for some reason that we do not know we started get this error message: Dropped sample for series that was not explicitly dropped via relabelling

Our Prometheus instances are running in the kubernetes cluster

Before we guessed was because cpu throttling and adjusted the GOMAXPROCS to the same value that cpu limit (in our configuration cpu limit and request have the same value)

The CPU throttling decrease and so, we enabled auto-gomaxprocs feature, but in both cases  did not resolve.

The next step was delete wal files from prometheus instances and some curious happened.

When I delete the wal file and delete pod in sequence the drop samples disappear, but if I delete the wal file and run a rollout processs (delete the first pod, wait this pod get ready) and so the another pod is deleted, the last pod deleted (that was working until the pod 1 to be deleted) do not disappear the drop sample message. I've tried delete the wal again and delete manually in sequence, the drop message do not appear after start process.

Looking for the source code https://github.com/prometheus/prometheus/blob/main/storage/remote/queue_manager.go#L579

this message can happen in two situations:

1. When it is not possible recover the labels and

2. When it is not found the reference id to sample.

My question is:

what causes these two situations?

How to avoid this?

How to resolve this?

[hartfordfive]

I'm also observing the same situation and was wondering if any additional clarity could be given on this subject by some of the core prometheus developers? 

I've also come across Github issues #11589 which seems to be related to this same issue.