extrenal labels missing

[dc3o]

In first Prometheus instance:

global:
 external_labels:
   origin_prometheus: prometheus01

second instance:

global: 
 external_labels: 
   origin_prometheus: prometheus02

Is it expected for label origin_prometheus to be accessible in each timeseries? When I execute a query in prom webui I'm not seeing any of these labels appear in response. I would like to use these labels as a variables for particular Grafana dashboard.

[Julius Volz]

Hi,

External labels are "invisible" in the respective Prometheus itself and only get attached whenever Prometheus sends samples or alerts to some external system (like a remote_write destination or Alertmanager). Thanos also uses external labels for identifying Prometheus servers. So it's normal that you won't be able to see external labels in the local data of a Prometheus server itself.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e7c41d9c-272c-41fe-bc29-2abacbeb39bfn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[dc3o]

I'm trying to re-use public node exporter templates from grafana forge. Most of these are data source blind. In my setup I'm using single Grafana instance to wrap up dashboards from different Prometheus data sources(customers). So instead of creating the same dashboard for every tenant I'd like to have universal dashboard using Grafana variables. Where I can choose different data source from withing the same dashboards. Seems to me that this is not possible without customizing public dashboards. I understand that this is not Prometheus related - just if anyone can share approach on this topic?  

[Julius Volz]

If the dashboard you are importing does not yet have a template variable for the datasource, then yes, indeed you will have to modify it to add such a template variable (https://grafana.com/docs/grafana/latest/variables/variable-types/add-data-source-variable/).

Another situation is if you use a system like Thanos to get an aggregated view over multiple Prometheus servers. In that case, the returned data would contain the external labels of each server. If you then wanted to narrow down the data to only a specific server, you could add label matchers in PromQL (potentially with template variables again) for that server's external labels.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d38acdb4-9652-457c-a703-efeac4f09de6n%40googlegroups.com.