hostname label in a target group?

[Harmen B]

Hi,

our prometheus config is maintained by chef, and uses the simple <target_group> config option. A snippet:

  - job_name: 'redis'

    target_groups:

      - targets:

        - 10.0.100.10:5555

        - 10.0.100.11:5555

Nothing fancy, and works fine.

However, I would like to add a label with the server ID or the hostname, to make the graphs easier to read. The more fancy service discovery methods have things such as __meta_consul_node and __meta_nerve_endpoint_name, which seems like they enable just that, but I can't see anything similar in the simple <target_group> option. Did I miss something?

For the node_exporter I could add it via a text file, but that doesn't work for all the other exporters we use.

Thoughts?

Thanks!,

Harmen

[Brian Brazil]

This is what the "instance" label is intended for. It defaults to the host:port (the "__address__" label), but you can set it separately via relabelling or if you're using file_sd by specifying it explicitly.

 

--

Brian Brazil

www.robustperception.io

[Harmen B]

Thanks for your quick reply, Brian.

Using relabel would be with a relabel entry for every host? Like this?

 - job_name: 'redis'
    target_groups:
      - targets:
        - 10.0.100.10:5555
        - 10.0.100.11:5555

    relabel:
       - source_labels: [ __address__ ]
         target_label: instance
         regex: 10.0.100.10:5555
         replacement: redis-1
       - source_labels: [ __address__ ]
         target_label: instance
         regex: 10.0.100.11:5555
         replacement: redis-2

(untested)
That would work, but seems... weird.

How would file_sd help? That doesn't seem to allow more than the target_group option.

Thanks!

Harmen

[Brian Brazil]

On 8 May 2016 at 01:06, Harmen B <har...@typetypetype.net> wrote:

Thanks for your quick reply, Brian.

Using relabel would be with a relabel entry for every host? Like this?

 - job_name: 'redis'
    target_groups:
      - targets:
        - 10.0.100.10:5555
        - 10.0.100.11:5555
    relabel:
       - source_labels: [ __address__ ]
         target_label: instance
         regex: 10.0.100.10:5555
         replacement: redis-1
       - source_labels: [ __address__ ]
         target_label: instance
         regex: 10.0.100.11:5555
         replacement: redis-2

(untested)
That would work, but seems... weird.

In this case I'd just leave it down to the job being redis, and them being arbitrary instances of it. Once you get beyond a handful of machines, individually naming them doesn't scale.

 

How would file_sd help? That doesn't seem to allow more than the target_group option.

You can specify all the labels in one go.

Brian 

Thanks!

Harmen

On Sun, May 8, 2016 at 1:04 AM, Brian Brazil <brian....@robustperception.io> wrote:

On 7 May 2016 at 23:36, Harmen B <har...@typetypetype.net> wrote:

Hi,

our prometheus config is maintained by chef, and uses the simple <target_group> config option. A snippet:

  - job_name: 'redis'

    target_groups:

      - targets:

        - 10.0.100.10:5555

        - 10.0.100.11:5555

Nothing fancy, and works fine.

However, I would like to add a label with the server ID or the hostname, to make the graphs easier to read. The more fancy service discovery methods have things such as __meta_consul_node and __meta_nerve_endpoint_name, which seems like they enable just that, but I can't see anything similar in the simple <target_group> option. Did I miss something?

For the node_exporter I could add it via a text file, but that doesn't work for all the other exporters we use.

Thoughts?

This is what the "instance" label is intended for. It defaults to the host:port (the "__address__" label), but you can set it separately via relabelling or if you're using file_sd by specifying it explicitly.

 

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[Harmen B]

In case someone else wants to do the same, this seems to work:

prometheus_config.yml:

global:

  scrape_interval:     15s

scrape_configs: 

  - job_name: "all"

    file_sd_configs:

      - names:

          - "scrape.yml"

scrape.yml:

- targets:

    - "192.168.4.100:1234"

  labels:

    "job": "something"

    "instance": "host4100"

- targets:

    - "192.168.5.100:1234"

  labels:

    "job": "something"

    "instance": "host5100"

- targets:

    - "192.168.5.100:1235"

  labels:

    "job": "else"

    "instance": "host5100"

Now you'll have a "something" and a "else" job, with whatever you want for the instance field.

[gaurav...@cenx.com]

Hi Prometheus Users,
Please help me to set hostname if i am deploying node exporter on all 4 nodes of my swarm cluster.

- job_name: 'prometheus'
dns_sd_configs:
- names:
- 'tasks.prometheus'
type: 'A'
port: 9090
static_configs:
- targets: ['localhost:9090']
- job_name: 'cadvisor'
dns_sd_configs:
- names:
- 'tasks.cadvisor'
type: 'A'
port: 8080
static_configs:
- targets: ['cadvisor:8080']
- job_name: 'alertmanager'
dns_sd_configs:
- names:
- 'tasks.alertmanager'
type: 'A'
port: 9093
- job_name: 'node-exporter'
dns_sd_configs:
- names:
- 'tasks.node-exporter'
type: 'A'
port: 9100
static_configs:
- targets: ['cadvisor:9100']

labels:
"job": "something"
"instance": "host4100"

Regards
Gaurav Goyal

[gaurav...@cenx.com]

Hi Brian,

Greetings for the day!

I want to display node hostname in my alert messages rather than internal IPs which are auto assigned to my containers.

Moreover since i am deploying node exporter on all SWARM nodes , how can i display each host separately

Here is my prometheus.yml file

15-096-Goyal:prometheus gaurav.goyal$ cat prometheus.yml
# my global config
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # By default, scrape targets every 15 seconds.
# scrape_timeout is set to the global default (10s).

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'my-project'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
- 'alert.rules'
# - "first.rules"
# - "second.rules"

# alert
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- "alertmanager:9093"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: 'grafana'
dns_sd_configs:
- names:
- 'tasks.grafana'
type: 'A'
port: 3000
# - job_name: 'node'

# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

# static_configs:
# - targets: ['localhost:9090','cadvisor:8080','node-exporter:9100']


Regards
Gaurav Goyal