How can I get the hostname in node-exporter?

[basili...@softonic.com]

Hi,

I suppose this is a silly question with a simple answer, but I have not found the hostname in the metrics provided by the node-exporter when launched as a docker container.

My goal is to be able to launch the node exporter as a global service in a swarm cluster and to be able to filter the collected values per host. If I cannot know from which node the data comes from I cannot filter the data correctly.

In the docs I've not seen any collector that would allow me to get this data either.

I've launched the node-exporter with this values:

```

docker \
service create --mode global \
--name node-exporter \
--network monitoring \
--publish 9100 \
prom/node-exporter \
--collectors.enabled netdev,stat,vmstat,conntrack,meminfo,time,diskstats,entropy,loadavg,mdadm,sockstat,textfile,uname,filefd,filesystem,netstat,ipvs,systemd

```

[Matthias Rampke]

This is information that is known at the time of discovery – the preferred way to add this label is on the Prometheus server side. The specifics depend on the discovery mechanism used; if all else fails relabelling will help.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[basili...@softonic.com]

And how can this be done? My Prometheus configuration is as simple as this:

´´´
global:
scrape_interval: 30s
evaluation_interval: 30s

labels:
cluster: swarm
replica: "1"

scrape_configs:
- job_name: 'prometheus'
dns_sd_configs:
- names:
- 'tasks.prometheus'
type: 'A'
port: 9090

- job_name: 'cadvisor'
dns_sd_configs:
- names:
- 'tasks.cadvisor'
type: 'A'
port: 8080

- job_name: 'node-exporter'
dns_sd_configs:
- names:
- 'tasks.node-exporter'
type: 'A'
port: 9100
´´´
It means that the discovery used is based on DNS, and at this moment I don't have any relabelling rule, which do I need to make this work?

[Julius Volz]

So in general, Prometheus automatically attaches the hostname or IP address (and port) as an "instance" label to all time series collected from a given target.

If you want the hostname and not the IP address, your problem may be that the "tasks.<service>" A records in Docker Swarm resolve directly into IP addresses, rather than hostnames. That's really only a hacky way for doing service discovery in Swarm at the moment. We'll want to have native Docker Swarm service discovery in Prometheus that can give us all the nice metadata about a target, but it's not there yet. Or maybe the IP address is sufficient for you?

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
>

> To post to this group, send email to prometheus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a81c1266-1460-4005-a05d-0917a7877550%40googlegroups.com.

[Ben Kochie]

A records will always resolve to IPs.  Swarm will need to publish SRV for hostnames to work.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
>

> To post to this group, send email to prometheus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a81c1266-1460-4005-a05d-0917a7877550%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxycOAkxzd3BLOY%3DFCD9xZcizaOG6ZwHP6smzbSH4oSZA%40mail.gmail.com.

[Julius Volz]

Yup, SRV would help, if they do that (not sure).

[basili...@softonic.com]

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> To post to this group, send email to prometheus...@googlegroups.com.
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a81c1266-1460-4005-a05d-0917a7877550%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

Hi Julius,

As far as I've seen this IP is the "node-exporter" task (container) IP:port in the Prometheus network, then it's something that allows me to split the values per node-exporter, and it's the value that I'm currently using in my Grafana Dashboard (https://grafana.net/dashboards/609).

But as it's not the node IP or hostname it's something that does not allow me to identify the node automatically, I need to manually check where's running the specific node-exporter container to realise it.
And I cannot use a global filter in my dashboard to filter by node, because the "same data" from the cAdvisor exporter is provided by a swarm container label that identifies the node in the swarm (container_label_com_docker_swarm_node_id).

Do you know when it's planned to support swarm service discovery natively in Prometheus?

Ah, and thanks for your help, it is very appreciated. ;)

[basili...@softonic.com]

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> To post to this group, send email to prometheus...@googlegroups.com.
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a81c1266-1460-4005-a05d-0917a7877550%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> To post to this group, send email to prometheus...@googlegroups.com.
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxycOAkxzd3BLOY%3DFCD9xZcizaOG6ZwHP6smzbSH4oSZA%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

And I think they are not registering the SRV records:

```
docker service create --name dnstools --mode global --network monitoring tutum/dnsutils sleep 100000
docker exec dnstools.0.b1rwsg6l1ylz9u6a6ad6zqxkh dig tasks.node-exporter

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> tasks.node-exporter
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59682
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;tasks.node-exporter. IN A

;; ANSWER SECTION:
tasks.node-exporter. 600 IN A 10.0.3.7
tasks.node-exporter. 600 IN A 10.0.3.8
tasks.node-exporter. 600 IN A 10.0.3.9

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Thu Oct 13 10:20:06 UTC 2016
;; MSG SIZE rcvd: 142

docker exec dnstools.0.b1rwsg6l1ylz9u6a6ad6zqxkh dig srv tasks.node-exporter

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> srv tasks.node-exporter
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 57764
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;tasks.node-exporter. IN SRV

;; AUTHORITY SECTION:
. 60 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2016101300 1800 900 604800 86400

;; Query time: 35 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Thu Oct 13 10:20:31 UTC 2016
;; MSG SIZE rcvd: 123

```

[Ben Kochie]

I think the typical recommendation is to run the node_expoeter on host networking, so that if can correctly monitor all real network interfaces.

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

>
> To post to this group, send email to prometheus...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a81c1266-1460-4005-a05d-0917a7877550%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.

Hi Julius,

As far as I've seen this IP is the "node-exporter" task (container) IP:port in the Prometheus network, then it's something that allows me to split the values per node-exporter, and it's the value that I'm currently using in my Grafana Dashboard (https://grafana.net/dashboards/609).

But as it's not the node IP or hostname it's something that does not allow me to identify the node automatically, I need to manually check where's running the specific node-exporter container to realise it.
And I cannot use a global filter in my dashboard to filter by node, because the "same data" from the cAdvisor exporter is provided by a swarm container label that identifies the node in the swarm (container_label_com_docker_swarm_node_id).

Do you know when it's planned to support swarm service discovery natively in Prometheus?

Ah, and thanks for your help, it is very appreciated. ;)

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/49e7b5c5-1054-466a-87d5-eb9a8402751f%40googlegroups.com.

[basili...@softonic.com]

> > > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > >
>
> >
>
> > > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6c5c8747-e80b-4fc6-967a-b7ca7fb8ebae%40googlegroups.com.
>
> >
>
> > >
>
> >
>
> > > For more options, visit https://groups.google.com/d/optout.
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> > You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> >
>

> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a81c1266-1460-4005-a05d-0917a7877550%40googlegroups.com.
>
> >
>
> >
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> Hi Julius,
>
>
>
> As far as I've seen this IP is the "node-exporter" task (container) IP:port in the Prometheus network, then it's something that allows me to split the values per node-exporter, and it's the value that I'm currently using in my Grafana Dashboard (https://grafana.net/dashboards/609).
>
>
>
> But as it's not the node IP or hostname it's something that does not allow me to identify the node automatically, I need to manually check where's running the specific node-exporter container to realise it.
>
> And I cannot use a global filter in my dashboard to filter by node, because the "same data" from the cAdvisor exporter is provided by a swarm container label that identifies the node in the swarm (container_label_com_docker_swarm_node_id).
>
>
>
> Do you know when it's planned to support swarm service discovery natively in Prometheus?
>
>
>
> Ah, and thanks for your help, it is very appreciated. ;)
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> To post to this group, send email to prometheus...@googlegroups.com.
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/49e7b5c5-1054-466a-87d5-eb9a8402751f%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

I'm not sure this can be done when it's launched as a "swarm service"

[Julius Volz]

On Thu, Oct 13, 2016 at 11:18 AM, <basili...@softonic.com> wrote:

Do you know when it's planned to support swarm service discovery natively in Prometheus?

No, but at the Docker Summit last weekend, there were some people interested in working on it. No concrete plans yet though, but I think everyone agrees that it makes sense for Prometheus to support native Docker Swarm discovery somehow. It's just very new. 

[juan.esc...@gmail.com]

I was looking for this earlier today.

Here is what I found. You can get the nodename(hostname) from the uname variable.
In Prometheus query node_uname_info. You can the parse the data to give you your hostname. I did the parsing using grafana.

[ntel...@gmail.com]

Are you getting the actual host name in which node_exporter is running or the container 'hostname'?
Below my results:

node_uname_info{domainname="(none)",instance="192.168.3.27:9100",job="node_exporter",machine="x86_64",nodename="2f14fc7ec7a5",release="4.4.0-87-generic",sysname="Linux",version="#110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017"} 1
node_uname_info{domainname="(none)",instance="192.168.3.28:9100",job="node_exporter",machine="x86_64",nodename="17c427fccfdf",release="4.4.0-87-generic",sysname="Linux",version="#110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017"} 1
node_uname_info{domainname="(none)",instance="192.168.3.29:9100",job="node_exporter",machine="x86_64",nodename="145b5a3bec00",release="4.4.0-87-generic",sysname="Linux",version="#110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017"} 1
...

[ntel...@gmail.com]

Are there any news on this?
It would be great to have at least the IP (or hostname) of the 'base' host in which the containers are running.

Is there any work-around? e.g. some kind of relabeling?

Thank you!

[Johannes Ziemke]

We recommend to run the node-exporter directly on the host, not containerized or use host-networking which should make it return the right hostname. This will also cause the metric to has the host's IP in the instance label.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d24f6040-ed66-4c04-b0fc-0ae61bea29f2%40googlegroups.com.

[ilias]

Thanks Johannes for the recommendations. Really appreciate it.

We would like though to have all our services containerized.

When you are saying 'use host-networking' you mean bind node-exporter and listening port to the host network and configure the communication with prometheus through that port/network. Correct?

The prometheus endpoints in this cases will have to be static and NOT automatically set through docker service discovery. That means whenever the cluster is scaled, prometheus should be configured with the new nodes IP. Right?

Another approach we are thinking is to leverage the textfile collector in node-exporter:

a. inject the hostname (and maybe other info as well) in a file inside node-exporter image

b. have a script for getting this hostname and the node-exporter container IP,  and formulate appropriately a custom metric via the textfile collector

c. in Grafana create a drop down menu with key-value pairs (hostname-containerIP) coming from all node-exporters that are dynamically discovered

d. the user will select the hostname and the respective containerIP will be used to filter all other metrics

Do you see this working?

Not sure how to address the same issue when it come to alerting though...

*Ilias

2017-12-19 20:39 GMT+01:00 Johannes Ziemke <fi...@freigeist.org>:

We recommend to run the node-exporter directly on the host, not containerized or use host-networking which should make it return the right hostname. This will also cause the metric to has the host's IP in the instance label.

On Tue, Dec 19, 2017 at 10:58 AM <ntel...@gmail.com> wrote:

Τη Πέμπτη, 13 Οκτωβρίου 2016 - 12:33:42 μ.μ. UTC+2, ο χρήστης Julius Volz έγραψε:
> On Thu, Oct 13, 2016 at 11:18 AM,  <basili...@softonic.com> wrote:
> Do you know when it's planned to support swarm service discovery natively in Prometheus?
>
>
>
> No, but at the Docker Summit last weekend, there were some people interested in working on it. No concrete plans yet though, but I think everyone agrees that it makes sense for Prometheus to support native Docker Swarm discovery somehow. It's just very new. 

Are there any news on this?
It would be great to have at least the IP (or hostname) of the 'base' host in which the containers are running.

Is there any work-around? e.g. some kind of relabeling?

Thank you!

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

[Johannes Ziemke]

Oh much easier! Just use hostNetwork: true in you pod spec. This will make it run in the host network namespace and Prometheus will still discover them but with the host ip as instance label.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To post to this group, send email to prometheus...@googlegroups.com.

[Ben Kochie]

One thing about node_exporter and containerization.  We consider the node_exporter a base OS service, like sshd.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.

To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d24f6040-ed66-4c04-b0fc-0ae61bea29f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAKSCP52z1OdAmDsb1%3Dax_nfSrO7SKcf1O%2BumkGUXiqpfJNyTjg%40mail.gmail.com.

[ilias]

I guess the equivalent for docker-compose would be network_mode: "host" ?

This seems to be ignored, as when I do 'docker stack deploy --compose-file docker-compose.yml --with-registry-auth blabla' I  am getting the warning "Ignoring unsupported options: build, network_mode, ulimits"

I am using docker 17.09.1-ce, docker-compose 1.17.1 and 3.3 inside docker-compose file.

Prometheus (and the whole ecosystem - cadvisor, alertmanager, ...) should also configured in this way? Or just leave them in their container-level 'private' network?

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

[ilias]

hm... that's interesting.

Our general direction (will see in the future how it goes) is to have the lightest OS  (e.g. kernel + basic services) as a basis and run on top of it all the other services containerized.

[Ben Kochie]

Yes, that's a good direction.  But IMO, node_exporter is a basic node service. :-)

Running the node_exporter under systemd cgroups is fully supported, and is very lightweight.  The main problem is Docker provides the wrong level of abstraction for system services like node_exporter, sshd, etc.

Compare Docker to Rkt Fly stage1[0], which is more in line with what the node_exporter needs for operating.

There is also the problem that I see of people conflating packaging, distribution, and orchestration.

[0]: https://coreos.com/rkt/docs/latest/running-fly-stage1.html