Any work done to secure prometheus query API and dashboard endpoints with https/TLS

[vvuth...@ebay.com]

The timeseries data in prometheus is accessible via plain http. Same for the UI Dashboard. Is there a way to secure them with a signed certificate? Seems like Prometheus provides TLS_config to scrape secure targets and also perform client cert auth with those targets if needed. But the access to the timeseries data stored in prometheus itself is left unsecured. We can have workarounds like use a reverse proxy, load balancer etc in front of the prometheus pod/instance and terminate the TLS handling there, but would be nice to secure the pod/instance endpoint itself by configuring a certificate. If there is a way to do this that I am missing from the docs, please let me know.

[Ben Kochie]

This is a discussion topic for our next developer summit.

Currently, the policy is that the server itself does not implement any security.

I'm working to change this.

On Tue, Jul 31, 2018, 21:01 <vvuth...@ebay.com> wrote:

The timeseries data in prometheus is accessible via plain http. Same for the UI Dashboard. Is there a way to secure them with a signed certificate? Seems like Prometheus provides TLS_config to scrape secure targets and also perform client cert auth with those targets if needed. But the access to the timeseries data stored in prometheus itself is left unsecured. We can have workarounds like use a reverse proxy, load balancer etc in front of the prometheus pod/instance and terminate the TLS handling there, but would be nice to secure the pod/instance endpoint itself by configuring a certificate. If there is a way to do this that I am missing from the docs, please let me know.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d74da52e-6aae-4031-88f5-98afeb1247f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luc Perkins]

FYI, I put together a guide in the official docs for enforcing TLS via an nginx reverse proxy: https://prometheus.io/docs/guides/tls-encryption. It's a very simple setup but it illustrates the important moving parts.

I also put together a few Docker-Compose-based reverse proxy scenarios, both of which enforce both TLS and basic auth:

• HAProxy
• nginx

Let me know if that helps. If not, I'd love to know how I can improve those guides/examples.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmop3T6%3Db%3DcPOo9UOFK_8D91D3Z5QAnNqm%3DR91xVePi_xQ%40mail.gmail.com.

[vuth...@gmail.com]

Luc,

Thanks for the pointers. 

Would appreciate what you would suggest in a kubernetes setup where the prometheus instances are pods and are fronted by k8s lb service.  Should we have an nginx reverse proxy as a sidecar in every prometheus pod ? What domain would we use in this case for creating the cert?

[vuth...@gmail.com]

Luc

As shown in attached image, when using the reverse proxy and browsing thru chrome, the UI appears all stripped out

I cannot make any of the buttons or UI work properly. Am I missing something

On Tuesday, July 31, 2018 at 1:02:03 PM UTC-7, Luc Perkins wrote:

[vuth...@gmail.com]

I am using the haproxy docker-compose from https://github.com/lucperkins/prometheus-playground/tree/master/haproxy

[david...@digital.cabinet-office.gov.uk]

I think that is caused by the prometheus path. I.e you have it pointing to localhost/prometheus as the root. I think the asset are located at localhost/asset, so it might not be able to load the CSS and other things. 

Please check your developer console window to review errors. There might something you can with HA-proxy to resolve this.

[david...@digital.cabinet-office.gov.uk]

--web.route-prefix=<path> this flag should do it for you. 

Start prometheus with it and set the value as /prometheus then it should pick up the static folder.

[vuth...@gmail.com]

Yup. Thanks David.