Honor labels not honored when external labels set

[Jimmy the Greek]

If I am reading the documentation correctly as documented here - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config - any existing labels on a time series should not be over-written by the external label value if honor labels is set to true.

However in reality I am not seeing this as the case. Configuration snippet:

global: 

  scrape_interval: 15s 

  scrape_timeout: 10s 

  evaluation_interval: 30s 

  external_labels: 

    account: services 

    environment: infrastructure 

    prometheus: prometheus/prometheus prometheus_replica: 

    prometheus-prometheus-0 

    region: us-west-2

scrape_configs:

- job_name: kafka/node 

  honor_labels: true 

  honor_timestamps: true 

  scrape_interval: 15s 

  scrape_timeout: 10s 

  metrics_path: /metrics 

  scheme: http 

  follow_redirects: true 

  enable_http2: true 

  static_configs: 

   - targets: 

      - b-1.kafka.xxxxx.kafka.us-west-2.amazonaws.com:11002 

      - b-2.kafka.xxxxxxx.kafka.us-west-2.amazonaws.com:11002

      labels: 

        account: non-prod

When I query these metrics from the prometheus server directly the account name is non-prod. However when pulled through Thanos which aggregates across many clusters the value is set to services. 

Thanks

[Brian Candler]

> If I am reading the documentation correctly as documented here - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config - any existing labels on a time series should not be over-written by the external label value if honor labels is set to true.

I presume you are referring to this part of the documentation?

# Note that any globally configured "external_labels" are unaffected by this

# setting. In communication with external systems, they are always applied only

# when a time series does not have a given label yet and are ignored otherwise.

The first part explicitly says that honor_labels (i.e. the scrape job setting) has no effect on external labels.  honor_labels applies to a scrape job and only affects scraping, not reading.

On the other hand, the second part does say that external labels are only applied when the label does not already exist.  Is that the issue you're raising?

If so, perhaps you can describe things like the versions of prometheus and thanos you're using. I think it would also be worth trying to query the federate endpoint, to see whether external labels behave the way you expect there (i.e. is this specifically a problem with thanos?)