Label Multiple Instances Name to Static Name

[Julian Ade Putra]

Hi Expert,

Currently i have a hundred Weblogic JVM Instances that already configures into Prometheus and JMX Exporter. However, i need to change label name of the instance (hostname:port) into weblogic instance name. Here is the configuration of prometheus.yml :

  - job_name: 'jmx'

  #  scrape_interval: 60s

    static_configs:

    - targets: ['xptcregosb01:8080','xptcregosb02:8080','xptcregosb03:8080','xptcregosb04:8080','xptcregsoa01:8080','xptcregsoa02:8080','xptcregsoa03:8080','xptcregsoa04:8080']

      labels:

        group: 'CRS'

    - targets: ['xptesbosbphost2:8080','xptesbosbphost2:8081','xptesbosbphost3:8080','xptesbosbphost3:8081']

      labels:

        group: 'UMB'

For example below target :

['xptcregosb01:8080','xptcregosb02:8080','xptcregosb03:8080','xptcregosb04:8080','xptcregsoa01:8080','xptcregsoa02:8080','xptcregsoa03:8080','xptcregsoa04:8080']

i need to label it with the weblogic instances name into :

WLS_OSB1,WLS_OSB2,WLS_OSB3,WLS_OSB4,WLS_SOA1,WLS_SOA2,WLS_SOA3,WLS_SOA4

I found this article https://stackoverflow.com/questions/49829423/prometheus-add-target-specific-label-in-static-configs but seems i need to define target instance one by one. Is there any others way to configure it?

Thanks,
Julian

[Brian Candler]

Use file_sd.  It's the same as static_configs (i.e. list of targets+labels), but bundled into a separate file, which makes it easier to edit or generate.  Also, prometheus picks up changes automatically without a HUP.  It's true though that each group of targets will need to be configured with its weblogic instance.

If you don't want to directly label the targets at scrape time, then make separate timeseries containing the mapping from instance <=> weblogic name, and then join on these other instances at query time.  For example, using node_exporter and textfile_collector on each node, you can make each node *tell* prometheus what its weblogic instance name is.

See:

https://www.robustperception.io/how-to-have-labels-for-machine-roles

https://www.robustperception.io/exposing-the-software-version-to-prometheus

To make these joins possible, you want to keep the port number :8080 out of the instance label. See:

https://www.robustperception.io/controlling-the-instance-label

[Julian Ade Putra]

Thank you Brian,

Finally i use file_sd and its work!

Whoever have same question with me, here is the configuration that use :

prometheus.yml - snippet 

scrape_configs:

  - job_name: 'jmx'

    file_sd_configs:

      - files:

        - jmx_targets.json

jmx_targets.json - snippet 

[

  {

    "targets": ["xptcregosb01:8080"],

    "labels": {

      "instance": "WLS_OSB1",

      "group": "CRS"

    }

  },

  {

    "targets": ["xptcregosb02:8080"],

    "labels": {

      "instance": "WLS_OSB2",

      "group": "CRS"

    }

  }
]

Thanks,
Julian