Relabel DNS-SD based node_exporter instance to it's IP

Lukas Lösche

unread,

Sep 1, 2016, 11:55:06 AM9/1/16

to Prometheus Developers

What source_label contains the IP address of the exporter Prometheus connected to when polling it?

I'm running node_exporter on every node of a Mesos cluster and using Mesos-DNS based DNS-SD SRV records to discover the nodes in the cluster.

The config looks something like this:

scrape_configs:

- job_name: node

dns_sd_configs:

- refresh_interval: 15s

names:

- '_node-exporter.prometheus._tcp.marathon.mesos'

relabel_configs:

- source_labels: ['__address__']

target_label: 'instance'

Now because of the way those SRV records work I'm ending up with instance labels like

node-exporter.prometheus-6ms1y-s1.marathon.mesos:14181

node-exporter.prometheus-54eio-s0.marathon.mesos:12227

node-exporter.prometheus-1e1ow-s2.marathon.mesos:31798

which is rather useless for my use case.

Instead I'd like the instance to contain the IP address of the host the node_exporter was running on / the SRV record resolved to.

I tried source_labels: ['__ip__'] but that's not it.

Brian Brazil

unread,

Sep 1, 2016, 2:01:42 PM9/1/16

to Lukas Lösche, Prometheus Developers

On 1 September 2016 at 16:55, Lukas Lösche <lu...@mesosphere.io> wrote:

What source_label contains the IP address of the exporter Prometheus connected to when polling it?

This is not exposed by Prometheus, and would not be available to relabelling as scraping happens after relabelling.

You likely need file_sd with a custom script here. Have you considered using the Marathon SD?

--

Brian Brazil

www.robustperception.io

Tobias Schmidt

unread,

Sep 1, 2016, 2:11:58 PM9/1/16

to Brian Brazil, Lukas Lösche, Prometheus Developers

On Thu, Sep 1, 2016 at 2:01 PM Brian Brazil <brian....@robustperception.io> wrote:

On 1 September 2016 at 16:55, Lukas Lösche <lu...@mesosphere.io> wrote:
What source_label contains the IP address of the exporter Prometheus connected to when polling it?

This is not exposed by Prometheus, and would not be available to relabelling as scraping happens after relabelling.

We could consider moving the DNS resolution from hostname to IP into DNS-sd itself though instead of doing it during each scrape.

I'm running node_exporter on every node of a Mesos cluster and using Mesos-DNS based DNS-SD SRV records to discover the nodes in the cluster.

The config looks something like this:

scrape_configs:
- job_name: node
dns_sd_configs:
- refresh_interval: 15s
names:
- '_node-exporter.prometheus._tcp.marathon.mesos'
relabel_configs:
- source_labels: ['__address__']
target_label: 'instance'

Now because of the way those SRV records work I'm ending up with instance labels like
node-exporter.prometheus-6ms1y-s1.marathon.mesos:14181
node-exporter.prometheus-54eio-s0.marathon.mesos:12227
node-exporter.prometheus-1e1ow-s2.marathon.mesos:31798
which is rather useless for my use case.

Instead I'd like the instance to contain the IP address of the host the node_exporter was running on / the SRV record resolved to.
I tried source_labels: ['__ip__'] but that's not it.

You likely need file_sd with a custom script here. Have you considered using the Marathon SD?

--
Brian Brazil
www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Brian Brazil

unread,

Sep 1, 2016, 2:26:06 PM9/1/16

to Tobias Schmidt, Lukas Lösche, Prometheus Developers

On 1 September 2016 at 19:11, Tobias Schmidt <tob...@gmail.com> wrote:

On Thu, Sep 1, 2016 at 2:01 PM Brian Brazil <brian.brazil@robustperception.io> wrote:
On 1 September 2016 at 16:55, Lukas Lösche <lu...@mesosphere.io> wrote:
What source_label contains the IP address of the exporter Prometheus connected to when polling it?

This is not exposed by Prometheus, and would not be available to relabelling as scraping happens after relabelling.

We could consider moving the DNS resolution from hostname to IP into DNS-sd itself though instead of doing it during each scrape.

That'd be inconsistent with the other SDs, ff we have to do if for one we have to do it for them all.

Brian

I'm running node_exporter on every node of a Mesos cluster and using Mesos-DNS based DNS-SD SRV records to discover the nodes in the cluster.

The config looks something like this:

scrape_configs:
- job_name: node
dns_sd_configs:
- refresh_interval: 15s
names:
- '_node-exporter.prometheus._tcp.marathon.mesos'
relabel_configs:
- source_labels: ['__address__']
target_label: 'instance'

Now because of the way those SRV records work I'm ending up with instance labels like
node-exporter.prometheus-6ms1y-s1.marathon.mesos:14181
node-exporter.prometheus-54eio-s0.marathon.mesos:12227
node-exporter.prometheus-1e1ow-s2.marathon.mesos:31798
which is rather useless for my use case.

Instead I'd like the instance to contain the IP address of the host the node_exporter was running on / the SRV record resolved to.
I tried source_labels: ['__ip__'] but that's not it.

You likely need file_sd with a custom script here. Have you considered using the Marathon SD?

--
Brian Brazil
www.robustperception.io

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Lukas Lösche

unread,

Sep 1, 2016, 5:27:44 PM9/1/16

to Brian Brazil, Prometheus Developers

On Thu, Sep 1, 2016 at 8:01 PM, Brian Brazil <brian....@robustperception.io> wrote:

You likely need file_sd with a custom script here. Have you considered using the Marathon SD?

I'm running Marathon on DC/OS EE with service account authentication which Marathon SD doesn't support.

file_sd seems the way to go which is a shame because so far I was able to use the unmodified prom/prometheus Docker image. Was hoping me not being able to get the node IP in that label was pebkac :-)

I saw that different SD methods have different meta labels. Would it be feasible to have a dedicated meta label with the IP just for DNS-SD? Since SRV records are the only ones that produce these rather useless hostnames it seems like a problem specific to the DNS SD discovery method.

Tobias Schmidt

unread,

Sep 1, 2016, 5:34:50 PM9/1/16

to Lukas Lösche, Brian Brazil, Prometheus Developers

That's what I have suggested above by moving the hostname to IP resolution into the discovery routine. So far this resolution happens on scrape time. Feel free to create an issue for that, but as Brian noticed this might require changes to other discovery mechanisms as well.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

Lukas Lösche

unread,

Sep 1, 2016, 5:44:04 PM9/1/16

to Tobias Schmidt, Brian Brazil, Prometheus Developers

Well I was thinking more along the lines of doubling the work for this SD method specifically via a config flag. I'd just add toggleable name resolution to dns.go for the sole purpose of creating an additional meta label, not touching the existing resolution code. Though I'm not experienced enough with the other SD methods to know if it would be of value or not to have the IP address in a generally available label.

On Thu, Sep 1, 2016 at 11:34 PM, Tobias Schmidt <tob...@gmail.com> wrote:

That's what I have suggested above by moving the hostname to IP resolution into the discovery routine. So far this resolution happens on scrape time. Feel free to create an issue for that, but as Brian noticed this might require changes to other discovery mechanisms as well.

On Thu, Sep 1, 2016 at 5:27 PM Lukas Lösche <lu...@mesosphere.io> wrote:

On Thu, Sep 1, 2016 at 8:01 PM, Brian Brazil <brian.brazil@robustperception.io> wrote:
You likely need file_sd with a custom script here. Have you considered using the Marathon SD?

I'm running Marathon on DC/OS EE with service account authentication which Marathon SD doesn't support.
file_sd seems the way to go which is a shame because so far I was able to use the unmodified prom/prometheus Docker image. Was hoping me not being able to get the node IP in that label was pebkac :-)

I saw that different SD methods have different meta labels. Would it be feasible to have a dedicated meta label with the IP just for DNS-SD? Since SRV records are the only ones that produce these rather useless hostnames it seems like a problem specific to the DNS SD discovery method.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

Brian Brazil

unread,

Sep 2, 2016, 2:52:33 AM9/2/16

to Tobias Schmidt, Lukas Lösche, Prometheus Developers

On 1 September 2016 at 22:34, Tobias Schmidt <tob...@gmail.com> wrote:

That's what I have suggested above by moving the hostname to IP resolution into the discovery routine. So far this resolution happens on scrape time. Feel free to create an issue for that, but as Brian noticed this might require changes to other discovery mechanisms as well.

It'd also be a breaking change, and I imagine most users don't want this behaviour. I'm not sure there's a sane way to do this.

Brian

On Thu, Sep 1, 2016 at 5:27 PM Lukas Lösche <lu...@mesosphere.io> wrote:

On Thu, Sep 1, 2016 at 8:01 PM, Brian Brazil <brian.brazil@robustperception.io> wrote:
You likely need file_sd with a custom script here. Have you considered using the Marathon SD?

I'm running Marathon on DC/OS EE with service account authentication which Marathon SD doesn't support.
file_sd seems the way to go which is a shame because so far I was able to use the unmodified prom/prometheus Docker image. Was hoping me not being able to get the node IP in that label was pebkac :-)

I saw that different SD methods have different meta labels. Would it be feasible to have a dedicated meta label with the IP just for DNS-SD? Since SRV records are the only ones that produce these rather useless hostnames it seems like a problem specific to the DNS SD discovery method.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Reply all

Reply to author

Forward