node_exporter behind a firewall

[mkemp...@googlemail.com]

Hi,

Is it possible to use node_exporter behind a firewall? If yes, how would i implement such a thing?

Thanks,
Marcel

[Brian Brazil]

On Tue, Aug 4, 2015 at 3:42 PM, <mkemp...@googlemail.com> wrote:

Hi,

Is it possible to use node_exporter behind a firewall? If yes, how would i implement such a thing?

Can you explain your full setup?

Generally we'd advise running the prometheus server behind the firewall too, to keep thing aligned with failure domains.

Brian

 

Thanks,
Marcel

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

In cases where you do need to scrape distributed/remote endpoints, you could put a reverse proxy infront of the node exporter which does https+basic auth (see http://prometheus.io/docs/operating/configuration/#scrape-configurations-scrape_config for how to scrape such endpoints). Bearer tokens and SSL client certificates were also just added as authentication options in the server. Those will be documented in the next days. But securing the endpoint itself needs to happen outside of Prometheus.

[mkemp...@googlemail.com]

Hi Julius,

Thanks for this hint. I installed a reverse proxy in front of the node_exporter, then i changed the scrape URL to the IP:PORT/metrics of the reverse proxy, but now (as expected) promeutheus uses the IP of the reverse proxy. Is there a chance using the source IP of the node_exporter? Some kind of rewriting?

Cheers,
Marcel

[Brian Brazil]

You can use relabelling to separately set "__address__" (what's talked to over http) and the "instance" label (what the instance label ends up as). See https://github.com/prometheus/snmp_exporter#prometheus-configuration for an example.

Brian

 

Cheers,

[mkemp...@googlemail.com]

Thanks Brian, this did the job for me.

[mkemp...@googlemail.com]

Hi,

Just one more thing regarding relabeling, is there a relabeling variable that contains the http source address, so i don't need to hardcode the replacement value?

----------------------------------------
relabel_configs:
- source_labels: [__address__]
regex: (.*):80
target_label: __address__
replacement: 10.70.0.50:9100 <---
----------------------------------------

Thanks,
Marcel

[Brian Brazil]

On Thu, Aug 6, 2015 at 8:16 AM, <mkemp...@googlemail.com> wrote:

Am Mittwoch, 5. August 2015 16:17:56 UTC+2 schrieb mkemp...@googlemail.com:
> Am Mittwoch, 5. August 2015 14:07:58 UTC+2 schrieb Brian Brazil:
> > On Wed, Aug 5, 2015 at 12:58 PM,  <mkemp...@googlemail.com> wrote:
> > Am Dienstag, 4. August 2015 16:42:27 UTC+2 schrieb mkemp...@googlemail.com:
> >
> > > Hi,
> >
> > >
> >
> > > Is it possible to use node_exporter behind a firewall? If yes, how would i implement such a thing?
> >
> > >
> >
> > > Thanks,
> >
> > > Marcel
> >
> >
> >
> > Hi Julius,
> >
> >
> >
> > Thanks for this hint. I installed a reverse proxy in front of the node_exporter, then i changed the scrape URL to the IP:PORT/metrics of the reverse proxy, but now (as expected) promeutheus uses the IP of the reverse proxy. Is there a chance using the source IP of the node_exporter? Some kind of rewriting?
> >
> >
> >
> > You can use relabelling to separately set "__address__" (what's talked to over http) and the "instance" label (what the instance label ends up as). See https://github.com/prometheus/snmp_exporter#prometheus-configuration for an example.
> >
> >
> > Brian
> >  
> >
> >
> > Cheers,
> >
> >
> >
> > Marcel
> >
> >
> >
> > --
> >
> > You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
> >
> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
> >
> > For more options, visit https://groups.google.com/d/optout.
>
> Thanks Brian, this did the job for me.

Hi,

Just one more thing regarding relabeling, is there a relabeling variable that contains the http source address, so i don't need to hardcode the replacement value?

In this setup you currently have to hardcode the destination address, but you can use DNS.

Brian

 

----------------------------------------
relabel_configs:
        - source_labels: [__address__]
          regex: (.*):80
          target_label: __address__
          replacement: 10.70.0.50:9100 <---
----------------------------------------

Thanks,

Marcel

[mkemp...@googlemail.com]

OK and what if there was a complete cluster behind that reverse proxy? How would you relabel the target IP address then?

Thanks,
Marcel

[Brian Brazil]

Yes, though it's recommended that you put the Prometheus/Prober in a place where it doesn't need to go via a reverse proxy as this is simpler and more reliable.

Brian

 

Thanks,
Marcel

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[mkemp...@googlemail.com]

Is it possible to forward all known metrics from one prometheus to another?

Thanks,
Marcel

[Julius Volz]

You could use federation (http://prometheus.io/docs/operating/federation/), though forwarding *all* time series between servers is not a typical use case.

Is this to run a Prometheus on each side of the firewall? If so, wouldn't it be better to just run a Prometheus where your targets are and then only use that Prometheus directly (via reverse proxy or similar)? Or do you have some requirement to also persistently collect monitoring data outside of your cluster where the targets are?

[mkemp...@googlemail.com]

Thanks for your answer. We have different use cases, some of the clusters are directly reachable through VPN tunnels and some clusters are behind firewalls but reachable through staging areas...if i understand you correctly there are two options. First use a reverse proxy on that staging area, disadvantage relabeling multiple nodes..., second put another prometheus on that staging area and use federation. Still trying to figure out whats the best way to achieve this. I'll try the federation mode.

Regards,
Marcel

[mkemp...@googlemail.com]

Hi Julius,

I've setup federation between to two Prometheus instances.
-----------------------------------------
Scrape configuration:
-----------------------------------------
- job_name: 'Federation'
scrape_interval: 5s
honor_labels: true
metrics_path: '/federate'
params:
'match[]': ['{__name__=~"^job:.*"}']
target_groups:
- targets:
- '10.70.0.50:9090'
-----------------------------------------
Target configuration:
-----------------------------------------
- job_name: 'node'
scrape_interval: 5s

# Override the global default and scrape targets from this job every 5 seconds.
scrape_timeout: 10s

target_groups:
- targets: ['10.70.0.51:9100','10.70.0.52:9100','10.70.0.53:9100','10.70.0.54:9100','10.70.0.55:9100']
labels:
group: 'clusterb50'

Unfortunately this doesn't seem to work, when accessing http://10.70.0.50:9090/federate?match[]={__name__%3D~%22^job%3A.*%22} an empty page is returned, even though Prometheus reports the scrape target as up.

Thanks,
Marcel

[Brian Brazil]

Do you have rules producing metrics with a job: prefix?

Brian

 

    target_groups:
      - targets:
        - '10.70.0.50:9090'
-----------------------------------------
Target configuration:
-----------------------------------------
  - job_name: 'node'
    scrape_interval: 5s

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_timeout: 10s

    target_groups:
      - targets: ['10.70.0.51:9100','10.70.0.52:9100','10.70.0.53:9100','10.70.0.54:9100','10.70.0.55:9100']
        labels:
          group: 'clusterb50'

Unfortunately this doesn't seem to work, when accessing http://10.70.0.50:9090/federate?match[]={__name__%3D~%22^job%3A.*%22} an empty page is returned, even though Prometheus reports the scrape target as up.

Thanks,
Marcel

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[mkemp...@googlemail.com]

Replaced above example with - 'job="prometheus"' now i get this error "Parse error at char 4: could not parse remaining input "=\"prometheus\""..."

- job_name: 'Federation'
scrape_interval: 5s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':

- 'job="prometheus"'

target_groups:
- targets:
- '10.70.0.50:9090'

Any ideas?

Thanks,
Marcel