Prometheus exporter security

[Dominic Cioccarelli]

[cross post from the user forum as I figured it might be better answered here]...

Hi all,

we are currently evaluating Prometheus as part of a bigger monitoring solution. As part of the investigation, the push vs. pull discussion came up and as part of that I started investigating the security aspect.

Obviously, in a push based approach security is less of an issue as the metric exporter is connecting from the monitored node to the monitoring system. In a pull based approach, where every exporter listens on a TCP port, we are effectively opening potential attack vectors (depending purely on how resilient the exporters are). One typical attack would be a buffer overflow.

Has anyone investigated how much of an issue this may be? Has much effort been put into the Go and Python Prometheus HTTP server modules to ensure that they are resilient to malicious external access?

Cheers,

Dominic.

[Richard Hartmann]

The first and most obvious line of defense would be to use common
tools like reverse proxies, iptables, and network ACLs as part of
protecting internal interfaces from external access; even if only to
not expose internal metrics to outsiders.

Pull does not imply, and should not include, allowing malicious
external access. And, for the record, neither should push.


Richard

[Dominic Cioccarelli]

Hi Richard,

many thanks for the reply. Obviously, the recommended solutions would work if we were trying to protect the exporters from external threats (e.g. from an Internet zone). Unfortunately, current best practices mandate that even local networks (e.g. intranets) should also be treated as inherently hostile. As a result, the scraped nodes will potentially reside directly within a potentially hostile network zone and can't be necessarily partitioned off or protected via an external network appliance.

Note that these attack vectors are only pertinent to a pull (or agent) based monitoring solution as they will only impact anything that opens a listening TCP port.

Typically, all services that listen on TCP ports and, hence, are externally accessible need to provide some protection against malicious attacks. This is why there are so many CVEs for components that operate in this mode (e.g. Tomcat, Apache HTTPD, etc).

How well do the current Prometheus exporters hold up well against a full pen test using a product like Rapid7?

Cheers,

Dominic.

[Julius Volz]

On Fri, May 5, 2017 at 10:42 PM, Dominic Cioccarelli <dominic.c...@gmail.com> wrote:

Hi Richard,

many thanks for the reply. Obviously, the recommended solutions would work if we were trying to protect the exporters from external threats (e.g. from an Internet zone). Unfortunately, current best practices mandate that even local networks (e.g. intranets) should also be treated as inherently hostile. As a result, the scraped nodes will potentially reside directly within a potentially hostile network zone and can't be necessarily partitioned off or protected via an external network appliance.

Putting a reverse proxy (like nginx) configured with TLS and auth (client certs, bearer token, or basic auth) in front of each exporter would allow you to restrict the pulling to only Prometheus.

 

Note that these attack vectors are only pertinent to a pull (or agent) based monitoring solution as they will only impact anything that opens a listening TCP port.

Typically, all services that listen on TCP ports and, hence, are externally accessible need to provide some protection against malicious attacks. This is why there are so many CVEs for components that operate in this mode (e.g. Tomcat, Apache HTTPD, etc).

How well do the current Prometheus exporters hold up well against a full pen test using a product like Rapid7?

We have not explicitly pen-tested our exporters, but most of the official ones are written in Go - a memory-safe language that does not allow for buffer overflow attacks or the like (unless there is a bug in the Go runtime). We also abstain from implementing security-critical functionality in the exporters themselves (like TLS + auth, for which we still require people to use e.g. a reverse proxy in addition to the exporter), so there's not much attack surface of that kind in typical exporters. Further, we don't have functionality in them that executes other processes or accesses the filesystem arbitrarily according to user-provided inputs. So use your own judgement, but personally I would not have major concerns on that front. Do check the specific exporters that you end up using though.

 

Cheers,

Dominic.

On Friday, 5 May 2017 16:36:50 UTC+2, Richard Hartmann wrote:

The first and most obvious line of defense would be to use common
tools like reverse proxies, iptables, and network ACLs as part of
protecting internal interfaces from external access; even if only to
not expose internal metrics to outsiders.

Pull does not imply, and should not include, allowing malicious
external access. And, for the record, neither should push.


Richard

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d4e2df12-99c2-4f92-85e9-c30cc55bdbad%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dominic Cioccarelli]

Hi Julius,

ok, many thanks. That makes complete sense. Thanks for the comprehensive reply.

Cheers,

Dominic.

[alban....@gmail.com]

Hello,

Sorry to come back on this topic even if the answers are already pretty clear. :)
I am still asking myself a question:

We are willing to execute actions based on prometheus monitoring (automatic recovery). Lets take as an exemple to cut traffic on a VM if CPU ~100% for 10m.

My question is, if node exporter is not secure, it is possible (and easy?) for an attacker to do a man-in-the-middle attack between prometheus and node-exporter (or even between nginx and node-exporter) to fake a metrics endpoints with 100% CPU.

In such way, this attacker can modify our traffic routing by faking the node-exporter plain-text page...

What do you think? Is securing the http servers in the exporters still out of the scope?

Best Regards,
Alban

[Ben Kochie]

We're currently working with a security consulting company to come up with a security guideline. I don't have any real ETA on if/when the current policy will change, but we're talking about officially adding some kind of transport security to our implementation guidelines.

It's still completely possible to implement this yourself with a reverse proxy, where you only run the exporter on localhost.  If your localhost is compromised, you probably have other problems ;-)

[Matt Doughty]

Or don't use the signal from prometheus to drive automated remediation. Nothing to stop you from monitoring the CPU locally and have your remediation triggered from there.  I would be very careful in general with driving remediation from the external monitoring platform.

Not that a transport security layer wouldn't valuable for a variety of reasons.

--Matt

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CABbyFmo79O0DoYrY_VGK%2BrwvLRKBWqohUL%3D6Ne4e5MxQJbeqkg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

--Matt