Node hostname as label

[gaelp...@yahoo.fr]

Hi there,

I am a newby with Prometheus, I am not very confortable so far.

I am trying to monitor nodes of a swarm cluster. The way I configured it does not allow me to get the hostname in the instance label, but rather the ip of the node-exporter container. This value changes after each service restart.

I use a fork of the node exporter (stefanprodan/swarmprom-node-exporter) which gets the node hostname by sharing the node host file:

docker-compose.yml
...
volumes:
- /etc/hostname:/etc/nodename
...

This image exposes a metric 'node_meta" containing these labels, exemple:

node_meta{container_label_com_docker_swarm_node_id="555atqfgoxh0gskjori7s0y2l",instance="10.0.0.6:9100",job="node-exporter",node_id="555atqfgoxh0gskjori7s0y2l",node_name="dc0-srv-docker3"}

My idea was to use the relabelling functionnality to replace the 'instance' label, or to add a new label to all metrics.

I have configured the prometheus.yml like that:

relabel_configs:
- source_labels: ['node_name']
regex: '(.*)'
replacement: '$1'
target_label: 'instance'

But this does not work. 'instance' is not changed. It works fine, with other source_labels, but as soon as I use 'node_id' or 'node_name', 'instance' is not changed.

I read the documentation but cannot find a way to investigate further. How could I see the available labels, and how can I export labels dynamically?

Can anyone help me to go further?
Thanks in advance
Gael

[Ben Kochie]

This is a problem with docker swarm discovery.  Node names are supposed to be handled by the discovery method.  I'm guessing you're using DNS A record discovery, which is not going to make this easy.

I would look at https://github.com/ContainerSolutions/prometheus-swarm-discovery

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/28e1cacb-f31f-438b-b9ec-214fb5575752%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gaelp...@yahoo.fr]

Thank you Ben, I will try it.
In the same time, I would also be interested to learn how to rewrite labels properly.

[ste...@natter.at]

Am Mittwoch, 24. Januar 2018 16:28:52 UTC+1 schrieb gaelp...@yahoo.fr:
> Thank you Ben, I will try it.
> In the same time, I would also be interested to learn how to rewrite labels properly.

Hi, where you successful? I am at the exact point right now (using stefanprodan/swarmprom-node-exporter as well). Thanks for your help.

[gaelp...@yahoo.fr]

Hallo Stefan,

yes solved using Grafana variables. I use 2 variables:

Node_name: label_values(node_meta, node_name)
Node_id: label_values(node_meta{node_name="[[Node_name]]"}, node_id)

When I select eth Node_name, then the Node_id is set automatically with the proper value.