job label missing from discoveredLabels (prometheus v2.42.0)

[Vu Nguyen]

Hi all,

We have a test code that reads target metadata info and job label name from `discoveredLabels` list. That list is included in the response we get from '/api/v1/targets' endpoint.

During the test, we noticed that the response from target endpoint is inconsistent: the job label sometimes is missing from `discoveredLabels` for a few discovered targets.

Below output is what I extracted from our deployment: the first target have the job label in its discovered labels but missing in the second target, on the same config job.

{
  "status" : "success",
  "data" : {
    "activeTargets" : [ {
      "discoveredLabels" : {
        "__meta_kubernetes_pod_phase" : "Running",
        "__meta_kubernetes_pod_ready" : "true",
        "__meta_kubernetes_pod_uid" : "a7b4cce2-1be7-4df9-a032-c7a51bb655db",
        "__metrics_path__" : "/metrics",
        "__scheme__" : "http",
        "__scrape_interval__" : "15s",
        "__scrape_timeout__" : "10s",
        "job" : "kubernetes-pods"
      },
      "labels" : {
        "app" : "testapp",
        "job" : "kubernetes-pods",
        "kubernetes_namespace" : "spider1"
      },
      "health" : "down",
      "scrapeInterval" : "15s",
      "scrapeTimeout" : "10s"
    }, {
      "discoveredLabels" : {
         "__meta_kubernetes_pod_phase" : "Running",
        "__meta_kubernetes_pod_ready" : "true",
        "__meta_kubernetes_pod_uid" : "85dfeac6-985d-479e-8459-fc20ae8dcec3",
        "__metrics_path__" : "/metrics",
        "__scheme__" : "http",
        "__scrape_interval__" : "15s",
        "__scrape_timeout__" : "10s",
        "app" : "another-testapp"
      },
      "labels" : {
        "app" : "another-testapp",
        "job" : "kubernetes-pods",
        "kubernetes_namespace" : "spider1"
      },
      "scrapePool" : "kubernetes-pods",
      "health" : "down",
      "scrapeInterval" : "15s",
      "scrapeTimeout" : "10s"
    } ]
  }
}

Could you please help us understand why we have this inconsistency? Is that correct way to get job level value from `discoveredLabels` set?

Thanks,

Vu

[Brian Candler]

I don't see this with v2.45.5, and I'm also concerned about why "app": "another-testapp" occurs in one of your discoveredLabels.

I suggest you try that, and/or the latests v2.52.1 (you can of course set up a completely separate instance but point it to the same service discovery source) and see if you can replicate the issue. Also check the changelogs and git history to see if there's anything relevant there.