Https issue when using prometheus federation

[Shi Yan]

I am trying to configure the prometheus federation, but the target is not up and the only error I can see is `read: connection reset by peer`

The scrape_config I've added is as follows:

- job_name: federate
  scrape_interval: 30s
  scrape_timeout: 15s
  scheme: https
  honor_labels: true
  metrics_path: "/federate"
  params:
    match[]:
    - '{job="jobname"}'
  static_configs:
  - targets:
    - another_prom_server

But if I use `curl` command from this central prometheus server, it works and can return the metrics correctly. 

another_prom_server is the one deployed by kube-prometheus-stack helm chart. Not sure what is the issue here? Could anyone help advise, thanks! 

[Brian Candler]

Can you show the exact curl command line, with just the hostname replaced with "example.com" ?

Try:

  - targets:
    - another_prom_server:9090

or

  - targets:
    - another_prom_server:443

or whatever is appropriate.  (I note you set "scheme: https" - is that correct? Is this prometheus running behind a reverse proxy or ingress proxy, or configured with web.config to serve TLS?)

[Shi Yan]

Thanks, Brian for helping look into it.

Yes, in our setup, `another_prom_server` is deployed on the k8s cluster and it is behind an F5 ingress proxy, which terminates the TLS protocol. So we use HTTPS here.

And I've tried to add port 443 explicitly in the targets config, but the error is still the same.

msg="Scrape failed" err="Get \"https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\": read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"

While I can manually curl it with either

 > curl https://example.com
 <a href="/graph">Found</a>

or the one with the exact URL parameters from the error msg.

 > curl 'https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
 .....# can get all the metrics correctly

Cheers

[Julien Pivotto]

Could you try adding http2_enable: false and see if there is an improvement?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9304c742-68a9-427d-8449-ec6cf83e0f9an%40googlegroups.com.

[Stuart Clark]

On 19/07/2022 14:51, Shi Yan wrote:
> Thanks, Brian for helping look into it.
>
> Yes, in our setup, `another_prom_server` is deployed on the k8s
> cluster and it is behind an F5 ingress proxy, which terminates the TLS
> protocol. So we use HTTPS here.
> And I've tried to add port 443 explicitly in the targets config, but
> the error is still the same.
>
> msg="Scrape failed" err="Get
> \"https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D\":
> read tcp x.x.x.x:58342->y.y.y.y:443: read: connection reset by peer"
>
> While I can manually curl it with either
>  > curl https://example.com
>  <a href="/graph">Found</a>
>
> or the one with the exact URL parameters from the error msg.
>  > curl
> 'https://example.com:443/federate?match%5B%5D=%7Bjob%3D%22jobname%22%7D'
>  .....# can get all the metrics correctly
>

How long does it take curl to respond with all the metrics?

Could it be that it takes a while and your load balancer is configured
with a shorter timeout?

--
Stuart Clark

[Shi Yan]

Thanks Julien, but adding enable_http2: false doesn't help. Still same error.

[Shi Yan]

Hi Stuart

The curl actually gets a response immediately,  as we only need to get a small set of metrics.

The curl command time is as follows

real    0m0.077s
user    0m0.016s
sys    0m0.018s

[Shi Yan]

oh, sorry, you mean all the metrics. It's about several seconds

real    0m7.361s
user    0m0.130s
sys    0m0.999s

On Wednesday, July 20, 2022 at 1:03:36 AM UTC+10 Stuart Clark wrote: