Scraping Prometheus Node Exporter data over HTTPS

[Patrik Iselind]

Hi,

I'm trying to get my prometheus server to scrape my targets over HTTPS. I have the following job config:

  - job_name: 'prometheus-https'

    file_sd_configs:
        - files: ['bonjour_https/*.json']

    scheme: https

    tls_config:
        insecure_skip_verify: false

In the Prometheus target overview page i see the error message "http: server gave HTTP response to HTTPS client". I cannot find any way to configure the node exporter running on my target to handle HTTPS. I have no idea what i'm missing.

At the moment i don't care about verifying the TLS certificate for my node exporter, but i'll want to do that later so i might as well as right away. Where and how do i install a certificate for the node exporter to use?

Thanks a lot in advance,
Patrik

[Conor Broderick]

Hi Patrik, 

Unfortunately what you're trying to do here is unsupported.  

In this case you should use a reverse proxy between the node exporter and Prometheus to scrape your targets over HTTPS.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d22d3f1-0098-438b-b3aa-c7c1a297ab3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

[mrx]

Hi Conor,

Thanks a lot for your direct answer, much appreciated.

Why is this unsupported. Is it due to some limitation in the node exporter specifically, or this is not supported by any exporter at this point?

Patrik Iselind

[Ben Kochie]

Correct, the node_exporter does not currently support HTTPS.  Most Prometheus exporters currently don't support https.  This is a current implementation choice due to the limited size of our developer community.

https://prometheus.io/docs/introduction/faq/#why-don-t-the-prometheus-server-components-support-tls-or-authentication-can-i-add-those

On Tue, Apr 18, 2017 at 12:41 PM, mrx <patri...@gmail.com> wrote:

Hi Conor,

Thanks a lot for your direct answer, much appreciated.

Why is this unsupported. Is it due to some limitation in the node exporter specifically, or this is not supported by any exporter at this point?

Patrik Iselind

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CANzOjBhwWuH04hF_uDAfTWcMMNiy%3Dof6ZwuBgbLNfjZ%3Dzqd5%2Bg%40mail.gmail.com.

[mrx]

Hi Ben and Conor,

But still, using the suggested reverse proxy wouldn't protect the data from end to end, just from the reverse proxy to the server. The data would still be unencrypted from target to the reverse proxy. Correct? If i want a TLS tunnel from end to end i guess i'll have to construct a tunnel on my own.

Is there something in the roadmap to include TLS encryption?

Is it correct then to say that today, Prometheus can only use TLS when communicating with other servers like Consule for example?

Patrik Iselind

[Ben Kochie]

You would run the exporter on localhost, and then the reverse proxy would only use localhost to connect.  This keeps the un-encrypted connection secured.

There is nothing on the roadmap for TLS in exporters at this time.

[mrx]

That would only work as long as I am only interested in monitoring one machine. In my case I have several raspberry pi units I would like to monitor.

Patrik

[Ben Kochie]

The node_exporter MUST run locally on each node (Raspberry Pi in this case), it does not work any other way.

[mrx]

I know it has to run locally...

I just cannot see how your last reply solves anything unless I'm only interested in monitoring localhost. Running the exporter on localhost is mostly secure since it doesn't go through any networks.

The connection would still be insecure between the reverse proxy and the targets though, unless the target is localhost.

Patrik

[Ben Kochie]

Sorry, I should have been more clear.

You also need to run a reverse proxy on every node to provide the secure link.

This is of course not optimal, and super annoying.  The original Prometheus design was for use on a trusted backend network, where you have a firewall between your hosts and the internet.

I personally would like to see us add some minimal TLS configuration flags to exporters, but some of the Prometheus team is convinced we don't have enough people to support this right now.

[Jarod Watkins]

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoH2rBhzM%2BumgLgcbb08%2B-Pw5E2a96zW%3D56GUnPAH2XEg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Jarod Watkins
Operations Engineer
42 Lines, Inc.

[Brian Brazil]

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAAECcmpUARgqt3v_zmOc_G8ZF%3D07SQ5MKG21GTAiWzBeXcpyrw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Patrik Iselind]

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue. When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language. It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it. For some stages in a project it's more important to get the actual encryption rather than verifying the other end or getting authentication right.

I'm not in any way saying that your worries are wrong in any way, they are most likely more than valid. The best way in my opinion to get around it would be communication.

// Patrik

[Brian Brazil]

On 18 April 2017 at 19:19, Patrik Iselind <patri...@gmail.com> wrote:

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue.

None of the issues I listed are provided for us by Go. Those are all things we'd get to directly worry about as implementers. 

Look at the "Perfect SSL Labs Score with Go" example on that page, that's the sort of thing we'd need to be able to support our users requesting.

 

When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

That's effectively saying that there's no security.

 

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language.

Being ready to handle security flaws is a fundamental requirement when it comes to security. This isn't something you can throw over the wall and forget about.

 

It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it.

We've had requests for just about every form of HTTP authentication that exists.

 

For some stages in a project it's more important to get the actual encryption rather than verifying the other end or getting authentication right.

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

Brian

 

I'm not in any way saying that your worries are wrong in any way, they are most likely more than valid. The best way in my opinion to get around it would be communication.

// Patrik

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

On Apr 18, 2017 21:01, "Brian Brazil" <brian....@robustperception.io> wrote:

On 18 April 2017 at 19:19, Patrik Iselind <patri...@gmail.com> wrote:

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue.

None of the issues I listed are provided for us by Go. Those are all things we'd get to directly worry about as implementers. 

Look at the "Perfect SSL Labs Score with Go" example on that page, that's the sort of thing we'd need to be able to support our users requesting.

 

When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

That's effectively saying that there's no security.

 

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language.

Being ready to handle security flaws is a fundamental requirement when it comes to security. This isn't something you can throw over the wall and forget about.

 

It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it.

We've had requests for just about every form of HTTP authentication that exists.

This is a "slippery slope" argument, and is invalid.

[mrx]

On Tue, Apr 18, 2017 at 9:01 PM, Brian Brazil <brian....@robustperception.io> wrote:

On 18 April 2017 at 19:19, Patrik Iselind <patri...@gmail.com> wrote:

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue.

None of the issues I listed are provided for us by Go. Those are all things we'd get to directly worry about as implementers. 

Could you be a bit more specific here?

Look at the "Perfect SSL Labs Score with Go" example on that page, that's the sort of thing we'd need to be able to support our users requesting.

Sure, and most of that seems to be provided by Go packages as i see it. 

 

When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

That's effectively saying that there's no security.

That's one way of putting it, sure. But getting the boiler plate into the code base how ever uncomplete it might be would simplify related contributions many times over.

 

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language.

Being ready to handle security flaws is a fundamental requirement when it comes to security. This isn't something you can throw over the wall and forget about.

I'm not saying Prometheus should throw anything over the wall. In the end it's up to the user to choose, is this enough for me or do i need to look elsewhere for a solution.

 

It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it.

We've had requests for just about every form of HTTP authentication that exists.

 

For some stages in a project it's more important to get the actual encryption rather than verifying the other end or getting authentication right.

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use. Having even basic TLS support is an awesome checkbox feature to have.

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

[Ben Kochie]

On Wed, Apr 19, 2017 at 8:22 AM, mrx <patri...@gmail.com> wrote:

It's not just a checkbox feature anymore.  Everything is going encrypted transport by default.  I know several companies where this is true, and I'm not just talking Google sized companies.

Kuberentes is also going this way.

[Brian Brazil]

I meant that users are treating it as a checkbox feature, for example by requesting it be added and ignoring the security implications - which kinda defeats the purpose of TLS.

Brian

 

Kuberentes is also going this way.

 

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

Yea, that's definitely a thing.

On the other side of the coin, there are users where this "checkbox" is taken seriously, handled properly, and is an actual operational need.  We currently make their work a lot more difficult than it needs to be.

 

Brian

 

Kuberentes is also going this way.

 

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

--

Brian Brazil

www.robustperception.io

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLqQRx6S28c1RuhK_VQ%3De8KRfRmmKQMezzor5w7CQpJXKg%40mail.gmail.com.

[Brian Brazil]

On 19 April 2017 at 07:22, mrx <patri...@gmail.com> wrote:

On Tue, Apr 18, 2017 at 9:01 PM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 18 April 2017 at 19:19, Patrik Iselind <patri...@gmail.com> wrote:

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue.

None of the issues I listed are provided for us by Go. Those are all things we'd get to directly worry about as implementers. 

Could you be a bit more specific here?

Look at the "Perfect SSL Labs Score with Go" example on that page, that's the sort of thing we'd need to be able to support our users requesting.

Sure, and most of that seems to be provided by Go packages as i see it. 

I'm not proposing to write our own crypto, the question is how do we allow users to configure all this? Interoperability has always been fun for SSL.

 

 

When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

That's effectively saying that there's no security.

That's one way of putting it, sure. But getting the boiler plate into the code base how ever uncomplete it might be would simplify related contributions many times over.

 

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language.

Being ready to handle security flaws is a fundamental requirement when it comes to security. This isn't something you can throw over the wall and forget about.

I'm not saying Prometheus should throw anything over the wall. In the end it's up to the user to choose, is this enough for me or do i need to look elsewhere for a solution.

Developers are generally unknowledgeable when it comes to security in general, and anything touching crypto in particular. Very few would be able to correctly asses our level of security, and everyone else would just blindly use a half-baked insecure implementation.

 

 

It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it.

We've had requests for just about every form of HTTP authentication that exists.

 

For some stages in a project it's more important to get the actual encryption rather than verifying the other end or getting authentication right.

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use.

No, the boilerplate is easy. The security and ongoing maintenance is the hard bit.

 

Having even basic TLS support is an awesome checkbox feature to have.

I interpret that as saying you want TLS, but don't care in the slightest about how much security it brings.

TLS is a security feature. To get it added it needs to be given the respect that requires.

Brian

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 19 April 2017 at 08:05, Ben Kochie <sup...@gmail.com> wrote:

They're two very different sets of users, and that's where a big part the challenge comes.

We've users who "just want TLS" who presume it's easy, and are happy with whatever the defaults are as they don't know any better (at least until the next vulnerability comes along).

On the other hand we've users who know what they're doing and/or have very specific requirements for TLS. They need to be able to configure everything. If we don't offer that, we're not really helping them avoid running a reverse proxy.

As implementers, we need to care largely about the latter case as that's what the former case turns into without warning at the next vulnerability/angry HN post about how project X doesn't do TLS properly. We also need to worry about things like if we need to re-release older versions with updated TLS libs and associated configuration. How does this interact with 1.0 guarantees, are we allowed break things for security? Do we have the manpower to realistically handle this? Can we realistically have a common server TLS library for all the configuration, given we can't even agree on having one for the client?

Those are the discussions we're not having, and that we need to have before we can add anything in this area.

Brian

 

 

Brian

 

Kuberentes is also going this way.

 

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

--

Brian Brazil

www.robustperception.io

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLqQRx6S28c1RuhK_VQ%3De8KRfRmmKQMezzor5w7CQpJXKg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

On Wed, Apr 19, 2017 at 9:22 AM, Brian Brazil <brian....@robustperception.io> wrote:

On 19 April 2017 at 08:05, Ben Kochie <sup...@gmail.com> wrote:

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use. Having even basic TLS support is an awesome checkbox feature to have.

It's not just a checkbox feature anymore.  Everything is going encrypted transport by default.  I know several companies where this is true, and I'm not just talking Google sized companies.

I meant that users are treating it as a checkbox feature, for example by requesting it be added and ignoring the security implications - which kinda defeats the purpose of TLS.

Yea, that's definitely a thing.

On the other side of the coin, there are users where this "checkbox" is taken seriously, handled properly, and is an actual operational need.  We currently make their work a lot more difficult than it needs to be.

They're two very different sets of users, and that's where a big part the challenge comes.

We've users who "just want TLS" who presume it's easy, and are happy with whatever the defaults are as they don't know any better (at least until the next vulnerability comes along).

On the other hand we've users who know what they're doing and/or have very specific requirements for TLS. They need to be able to configure everything. If we don't offer that, we're not really helping them avoid running a reverse proxy.

As implementers, we need to care largely about the latter case as that's what the former case turns into without warning at the next vulnerability/angry HN post about how project X doesn't do TLS properly. We also need to worry about things like if we need to re-release older versions with updated TLS libs and associated configuration. How does this interact with 1.0 guarantees, are we allowed break things for security? Do we have the manpower to realistically handle this? Can we realistically have a common server TLS library for all the configuration, given we can't even agree on having one for the client?

We already support TLS on the Prometheus server, none of the "1.0" stuff is relevant here.

This is about exporters, none of which have any 1.0 guarantees.

I know several security engineers that would probably be willing to give guidance here.

 

Those are the discussions we're not having, and that we need to have before we can add anything in this area.

When the start of the discussion is "No" and is the rest is FUD, I wonder why we're not having it.

 

Brian

 

 

[Brian Brazil]

On 19 April 2017 at 08:34, Ben Kochie <sup...@gmail.com> wrote:

On Wed, Apr 19, 2017 at 9:22 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 19 April 2017 at 08:05, Ben Kochie <sup...@gmail.com> wrote:

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use. Having even basic TLS support is an awesome checkbox feature to have.

It's not just a checkbox feature anymore.  Everything is going encrypted transport by default.  I know several companies where this is true, and I'm not just talking Google sized companies.

I meant that users are treating it as a checkbox feature, for example by requesting it be added and ignoring the security implications - which kinda defeats the purpose of TLS.

Yea, that's definitely a thing.

On the other side of the coin, there are users where this "checkbox" is taken seriously, handled properly, and is an actual operational need.  We currently make their work a lot more difficult than it needs to be.

They're two very different sets of users, and that's where a big part the challenge comes.

We've users who "just want TLS" who presume it's easy, and are happy with whatever the defaults are as they don't know any better (at least until the next vulnerability comes along).

On the other hand we've users who know what they're doing and/or have very specific requirements for TLS. They need to be able to configure everything. If we don't offer that, we're not really helping them avoid running a reverse proxy.

As implementers, we need to care largely about the latter case as that's what the former case turns into without warning at the next vulnerability/angry HN post about how project X doesn't do TLS properly. We also need to worry about things like if we need to re-release older versions with updated TLS libs and associated configuration. How does this interact with 1.0 guarantees, are we allowed break things for security? Do we have the manpower to realistically handle this? Can we realistically have a common server TLS library for all the configuration, given we can't even agree on having one for the client?

We already support TLS on the Prometheus server, none of the "1.0" stuff is relevant here.

As a client, not as a server.

 

This is about exporters, none of which have any 1.0 guarantees.

I'd presume it'd apply to everything uniformly that we have in Go, including Prometheus. If this ends up as a config file thing, then 1.0 guarantees apply, and if as flags things get potentially messy given how 1.0 guarantees are written (we can't hard-break a flag, which may be the right thing to do security-wise if someone has specified it but we now ignore it).

I know several security engineers that would probably be willing to give guidance here.

That'd be useful.

 

 

Those are the discussions we're not having, and that we need to have before we can add anything in this area.

When the start of the discussion is "No" and is the rest is FUD, I wonder why we're not having it.

Realistic security and maintenance concerns based on many years of watching TLS protocols and best practices evolve are not FUD.

--

Brian Brazil

www.robustperception.io

[Ben Kochie]

"Unknown unknowns" boils down to FUD IMO.

To move this in a positive direction, I suggest we take this discussion to a design doc and properly spec out what it will really take to support transport security for metrics.  Long-term we're going to want to support this as a project.

@mrx, @Jarrod, would you be interested in writing up a design doc?  I can recruit additional security engineers to participate, review, etc.

 

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 19 April 2017 at 08:55, Ben Kochie <sup...@gmail.com> wrote:

On Wed, Apr 19, 2017 at 9:46 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 19 April 2017 at 08:34, Ben Kochie <sup...@gmail.com> wrote:

On Wed, Apr 19, 2017 at 9:22 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 19 April 2017 at 08:05, Ben Kochie <sup...@gmail.com> wrote:

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use. Having even basic TLS support is an awesome checkbox feature to have.

It's not just a checkbox feature anymore.  Everything is going encrypted transport by default.  I know several companies where this is true, and I'm not just talking Google sized companies.

I meant that users are treating it as a checkbox feature, for example by requesting it be added and ignoring the security implications - which kinda defeats the purpose of TLS.

Yea, that's definitely a thing.

On the other side of the coin, there are users where this "checkbox" is taken seriously, handled properly, and is an actual operational need.  We currently make their work a lot more difficult than it needs to be.

They're two very different sets of users, and that's where a big part the challenge comes.

We've users who "just want TLS" who presume it's easy, and are happy with whatever the defaults are as they don't know any better (at least until the next vulnerability comes along).

On the other hand we've users who know what they're doing and/or have very specific requirements for TLS. They need to be able to configure everything. If we don't offer that, we're not really helping them avoid running a reverse proxy.

As implementers, we need to care largely about the latter case as that's what the former case turns into without warning at the next vulnerability/angry HN post about how project X doesn't do TLS properly. We also need to worry about things like if we need to re-release older versions with updated TLS libs and associated configuration. How does this interact with 1.0 guarantees, are we allowed break things for security? Do we have the manpower to realistically handle this? Can we realistically have a common server TLS library for all the configuration, given we can't even agree on having one for the client?

We already support TLS on the Prometheus server, none of the "1.0" stuff is relevant here.

As a client, not as a server.

 

This is about exporters, none of which have any 1.0 guarantees.

I'd presume it'd apply to everything uniformly that we have in Go, including Prometheus. If this ends up as a config file thing, then 1.0 guarantees apply, and if as flags things get potentially messy given how 1.0 guarantees are written (we can't hard-break a flag, which may be the right thing to do security-wise if someone has specified it but we now ignore it).

I know several security engineers that would probably be willing to give guidance here.

That'd be useful.

 

 

Those are the discussions we're not having, and that we need to have before we can add anything in this area.

When the start of the discussion is "No" and is the rest is FUD, I wonder why we're not having it.

Realistic security and maintenance concerns based on many years of watching TLS protocols and best practices evolve are not FUD.

"Unknown unknowns" boils down to FUD IMO.

By my count I've asked 10 concrete questions:

- How does the user configure this?

- What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)?

- When TLS 1.4 comes out, what does that mean for configuration?

- When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation?

- How quickly does that have to happen?

- What do we do about older vulnerable releases?

- Do we need to support letsencrypt?

- How does this interact with 1.0 guarantees, are we allowed break things for security?

- Can we realistically have a common server TLS library for all the configuration, given we can't even agree on having one for the client?

- Do we have the manpower to realistically handle all this?

These are not FUD, and none of them have been properly addressed.

Unknown unknowns are also one of the things we need to plan for too, as that seems to be about the only certainty when it comes to security (for example, who expected the best practice to go to TLS 1.3 and then back to TLS 1.2?). The above questions should cover this.

 

To move this in a positive direction, I suggest we take this discussion to a design doc and properly spec out what it will really take to support transport security for metrics.  Long-term we're going to want to support this as a project.

Long-term yes, I believe we currently lack both the attitude and manpower to do it properly. I'm strongly against doing it improperly.

I have been hoping some other project will produce&maintain a library that'll do all the tricky bits for us (particularly for authentication&authorisation, which hasn't been touched here), which'd cover 6 and 1/2 of the above questions.

--

Brian Brazil

www.robustperception.io

[mrx]

On Wed, Apr 19, 2017 at 9:06 AM, Brian Brazil <brian....@robustperception.io> wrote:

On 19 April 2017 at 07:22, mrx <patri...@gmail.com> wrote:

On Tue, Apr 18, 2017 at 9:01 PM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 18 April 2017 at 19:19, Patrik Iselind <patri...@gmail.com> wrote:

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue.

None of the issues I listed are provided for us by Go. Those are all things we'd get to directly worry about as implementers. 

Could you be a bit more specific here?

Look at the "Perfect SSL Labs Score with Go" example on that page, that's the sort of thing we'd need to be able to support our users requesting.

Sure, and most of that seems to be provided by Go packages as i see it. 

I'm not proposing to write our own crypto, the question is how do we allow users to configure all this? Interoperability has always been fun for SSL.

Well, Prometheus seems to get by just fine with tls_config sections. Why can we not just continue with the same?

 

 

When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

That's effectively saying that there's no security.

That's one way of putting it, sure. But getting the boiler plate into the code base how ever uncomplete it might be would simplify related contributions many times over.

 

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language.

Being ready to handle security flaws is a fundamental requirement when it comes to security. This isn't something you can throw over the wall and forget about.

I'm not saying Prometheus should throw anything over the wall. In the end it's up to the user to choose, is this enough for me or do i need to look elsewhere for a solution.

Developers are generally unknowledgeable when it comes to security in general, and anything touching crypto in particular. Very few would be able to correctly asses our level of security, and everyone else would just blindly use a half-baked insecure implementation.

But in the end there is always a user/customer that has to make the choice if this is secure "enough" or not.

 

 

It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it.

We've had requests for just about every form of HTTP authentication that exists.

 

For some stages in a project it's more important to get the actual encryption rather than verifying the other end or getting authentication right.

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use.

No, the boilerplate is easy. The security and ongoing maintenance is the hard bit.

How can you be sure? Open source software can be seen as best effort, if some customer need something faster they'll have to either put in the time to fix it them selves or limit their exposure.

 

Having even basic TLS support is an awesome checkbox feature to have.

I interpret that as saying you want TLS, but don't care in the slightest about how much security it brings.

Oh, i do care! It's just that before the security implications can even be discussed there has to be some form of functioning foundation from which we can see which parts are actually used.

TLS is a security feature. To get it added it needs to be given the respect that requires.

Couldn't agree more, but as i've said before it's up to the customer in the end to make the choice if the security is enough or not.

Brian

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

--

Brian Brazil

www.robustperception.io

Sorry for the delay in my answer, i've been home sick for a few days.

[mrx]

On Wed, Apr 19, 2017 at 9:55 AM, Ben Kochie <sup...@gmail.com> wrote:

Given my familly situation it's not easy for be to get large amounts of free time for this but i can be in on a corner. Perhaps i can get some of my work time to be a part of this but i'll have to clear that with the boss.

[Brian Brazil]

On 21 April 2017 at 09:06, mrx <patri...@gmail.com> wrote:

On Wed, Apr 19, 2017 at 9:06 AM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 19 April 2017 at 07:22, mrx <patri...@gmail.com> wrote:

On Tue, Apr 18, 2017 at 9:01 PM, Brian Brazil <brian.brazil@robustperception.io> wrote:

On 18 April 2017 at 19:19, Patrik Iselind <patri...@gmail.com> wrote:

Den 2017-04-18 kl. 19:50, skrev Brian Brazil:

On 18 April 2017 at 18:13, Jarod Watkins <ja...@42lines.net> wrote:

I would love to see some minimal TLS support in at least the node_exporter. To expect any segment of the network to be trusted in this day in age is dangerous. Having to run a reverse proxy just to secure monitoring traffic is adding a lot of overhead. How much overhead would it be to maintain TLS in Prometheus and the node exporter? I would love to volunteer for this effort if the Prometheus devs have an idea how they would like for this to be handled.

It'd be nice to have this feature. The challenge is that this is not a one-off thing that someone can implement some minimal form of and that's it. As a both a cryptography and security feature, there's a much higher bar than that which requires ongoing commitment.

For example, how would ciphers be selected and negotiated? What happens when there's a new type of cipher with new parameters (e.g. curve selection for ECC)? When TLS 1.4 comes out, what does that mean for configuration? When the next TLS vulnerability comes, who's going to rebuild and release the 15 Go projects that would be affected in the Prometheus github organisation? How quickly does that have to happen? What do we do about older vulnerable releases? Do we need to support letsencrypt?

This isn't even touching authentication/authorisation/configuration questions if client auth is desired.

In my opinion it's better to make it very clear that security is the end user's responsibility and let them use standard well-maintained implementations such as apache/nginx, rather than leading users into a false of security by a half-baked TLS implementation on our part.

This isn't something that one person can do, it requires a team of several people who can jump in at a moment's notice and handle these issues when they come up. We're not a big enough team to do this currently, we're just about managing to keep on top of our existing features. Adding any security related features would bring us a new level of support and maintenance workloads.

I think the best route to this happening would be to help stabilise and complete the myriad of features that we already have, and generally expanding the active contributors that could help respond to a security incident.

Brian

Based on https://github.com/denji/golang-tls it seems that most of the cryptographic stuff is already included in Go so that part is probably not an issue.

None of the issues I listed are provided for us by Go. Those are all things we'd get to directly worry about as implementers. 

Could you be a bit more specific here?

Look at the "Perfect SSL Labs Score with Go" example on that page, that's the sort of thing we'd need to be able to support our users requesting.

Sure, and most of that seems to be provided by Go packages as i see it. 

I'm not proposing to write our own crypto, the question is how do we allow users to configure all this? Interoperability has always been fun for SSL.

Well, Prometheus seems to get by just fine with tls_config sections. Why can we not just continue with the same?

Firstly, that's been incompletely designed and we should be offering more than tls_config currently does. Vulnerabilities requiring client side action are rarer, but they do happen - typically downgrade attacks.

Secondly the server side is where almost all the configuration is in practice, as you're largely trying to prevent data getting out. You need to deal with multiple client implementations plus whatever ancient clients, proxies or firewalls might be in use.

 

 

When it comes to all the other worries you list, I'd say it's all about communication. Make the project's stand perfectly clear, there is TLS support but it's up to contributions to keep it up to date with ciphers and everything else.

That's effectively saying that there's no security.

That's one way of putting it, sure. But getting the boiler plate into the code base how ever uncomplete it might be would simplify related contributions many times over.

 

Also there is no reason to expect a complete 100% coverage immediately. No one expects that we should accomplish world peace in once stroke either. Skip authentication and other 'more complicated parts' to start with, let them be added later. Again, it's all about communication. As long as the project is clear on what is there and what is not it's up to the users to decide if they want to accept the limitations, or not.

Getting the basic TLS support in place is probably by far a harder task than once it's included amending security flaws and so on, which to some extent would probably be distributed by the Go language.

Being ready to handle security flaws is a fundamental requirement when it comes to security. This isn't something you can throw over the wall and forget about.

I'm not saying Prometheus should throw anything over the wall. In the end it's up to the user to choose, is this enough for me or do i need to look elsewhere for a solution.

Developers are generally unknowledgeable when it comes to security in general, and anything touching crypto in particular. Very few would be able to correctly asses our level of security, and everyone else would just blindly use a half-baked insecure implementation.

But in the end there is always a user/customer that has to make the choice if this is secure "enough" or not.

In the same way we don't allow users to decrement counters, we can choose to not let the user shoot themselves in the foot with security.

 

 

It seems that many users want TLS support. It might even be that once TLS support is there you would find that very few users is interested in the authentication part of it.

We've had requests for just about every form of HTTP authentication that exists.

 

For some stages in a project it's more important to get the actual encryption rather than verifying the other end or getting authentication right.

Encryption is pointless if you're not verifying the other end, as you're leaving yourself open to MITM.

Agree, but my point is as previously mentioned that the hardest part is getting the basic boiler plate into the code base. From that point it's up to the user to choose which features to use.

No, the boilerplate is easy. The security and ongoing maintenance is the hard bit.

How can you be sure?

The generally quoted number is that maintenance takes 80% of the effort of a software project. I see nothing here that could make that better in this case.

 

Open source software can be seen as best effort, if some customer need something faster they'll have to either put in the time to fix it them selves or limit their exposure.

I'm interpreting your argument as saying that you need security, but whenever the next vulnerability comes you no longer require security.

Prometheus is production software already used in hundreds of companies across the world, we don't have the luxury of being a weekend project that can treat security as something that we can half-ass.

When there's a vulnerability being actively exploited, that's not the time to start getting into Prometheus and Go development, learning how TLS works, have debates around how we should structure new configuration options or discovering that there's no capability to release updated binaries in the needed timeframe.

To be clear, the timeframes we're talking about here are often measured in hours.

 

 

Having even basic TLS support is an awesome checkbox feature to have.

I interpret that as saying you want TLS, but don't care in the slightest about how much security it brings.

Oh, i do care! It's just that before the security implications can even be discussed there has to be some form of functioning foundation from which we can see which parts are actually used.

That works for most features, not however for anything correctness, privacy or security related. It's exactly the other way around for those, and up front design and planning is required (in some cases, by law).

TLS is not some innovative UX feature that needs to be MVPed. The space is quite well understood already.

TLS is a security feature. To get it added it needs to be given the respect that requires.

Couldn't agree more, but as i've said before it's up to the customer in the end to make the choice if the security is enough or not.

The Prometheus project is not a commercial enterprise, we have users rather than customers.

You're presuming that all users are informed enough to make that decision and understand the full implications. That's not a good assumption to be making, particularly about security.

For example it's not at all likely that the same type of person that right now might argue that the choice is up to them, and yet when there's a vulnerability vilify us for not acting instantly and further claiming that we're incompetent for not planning for this inevitability.

I'd suggest you take up Ben's offer to work on a design doc that'll address all this, as discussion along the current lines isn't going anywhere.

Brian

 

Brian

This is a complex feature with no easy solutions. Just because the defaults appear to work for you today, doesn't mean that they work for everyone or will be considered secure in a month's time. Security and crypto are a rapidly changing landscape, and that's the space we'd be getting into as soon as even "minimal" TLS support is added.

I'm not saying that any one solution works for everyone. I cannot imagine that has ever happened in the history of mankind...

// Patrik

--

Brian Brazil

www.robustperception.io

Sorry for the delay in my answer, i've been home sick for a few days.

--

Brian Brazil

www.robustperception.io