understanding Prometheus TLS configuration

2,526 views
Skip to first unread message

akshay sharma

unread,
Jul 19, 2021, 8:26:21 AM7/19/21
to Prometheus Users
Hi, 

I was going through the Prometheus tls configuration. I've come across two different tls configurations in Prometheus. 
One is tls_config and the other one is tls_server_config. 

I want to understand the difference between these two. 


I want to achieve the following:
1) I've a service running with MTLS, and I want  Prometheus to scrape metrics from that service. 
2) Service needs client cert and client key and cacert as well. 

For ex: 
     1) curl -v --cert ./client.crt --key ./client.key GET -H "Content-type: application/json" https://xxx.xxx.xxx.xx:xxxx/metrics --cacert server.crt
     2) What if I don't want to pass cacert, I want to use -k(insecure in configuration) . How can I do that?

So, what configuration can I use in prometheus to achieve this?


Thanks,
Akshay

Julius Volz

unread,
Jul 20, 2021, 8:41:19 AM7/20/21
to akshay sharma, Prometheus Users
Hi Akshay,

Though it's not really clear from https://prometheus.io/docs/prometheus/latest/configuration/https/, the "tls_server_config" is about the serving endpoints in Prometheus itself. So this won't be the interesting part for your scenario, you'll want to set the "tls_config" in your "scrape_config" in your main config file instead: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config

If you want to disable verification of the endpoint that Prometheus is scraping from, you can set "insecure_skip_verify: true" in your "tls_config" block.

Btw., I'm not sure why you are sending a "Content-type: application/json" header as part of the request in your curl example? Probably that's just a copy-and-paste bug?

Regards,
Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOrgXNJ7e1hGc59YBjGmFfLVud-0%2BfDZ0vEFiAPhh4cuTqo_LQ%40mail.gmail.com.


--
Julius Volz
PromLabs - promlabs.com

akshay sharma

unread,
Jul 20, 2021, 8:50:49 AM7/20/21
to Julius Volz, Prometheus Users
Hi , 

Thanks for your reply. I've seen tls_config in the prometheus doc. 
One query here: does it support mtls? Because there is no such information related to this. If yes, can you please give me a dummy configuration for mtls.

and yes, that's a typo. 

thanks,
Akshay

Julius Volz

unread,
Jul 20, 2021, 10:50:48 AM7/20/21
to akshay sharma, Prometheus Users
Hi Akshay,

Yes. If my understanding is correct, mTLS just means that the server (in this case, your application) also validates the client (Prometheus), in addition to the client validating the server (which you can turn off via "insecure_skip_verify: true"). That means configuring a client cert & key via the "cert_file" and "key_file" fields of the "tls_config". See also this article about Prometheus + mTLS: https://smallstep.com/hello-mtls/doc/client/prometheus

Regards,
Julius
Reply all
Reply to author
Forward
0 new messages