Proxy authentication credentials using proxy_connect_header not working

[Kesavanand Chavali]

Hello,

we have a proxy that requires authentication using username and password. we upgraded to 2.42 version of prometheus and tried to use proxy_connect_header. 

Basically we need to send Authorization header as "Basic base64_encoded(user:password)"

as given in below links we tried to pass header without "Basic" word.

https://github.com/prometheus/prometheus/issues/11626

we tried passing as plain string, base64 and secret (bcrypt encrypted string) and nothing works. we also tried with and without Basic keyword. But no luck

Any help is highly appreciated

Thanks in advance

Here is our prometheus.yml, running on windows 2016 server as a windows service

global:
  scrape_interval: 4m
  scrape_timeout: 30s
  external_labels:    
    region: eu-west
    market: APAC
    country: IN
    city: Bangalore  
    environment: Test    
scrape_configs:
- job_name: prometheus_self_monitoring_job
  honor_timestamps: true
  honor_labels: true
  static_configs:
  - targets:
    - hostname1:9090
  tls_config:
    insecure_skip_verify: true
  scheme: http  
remote_write:
- url: https://test.remote-write.com/api/v1/receive
  follow_redirects: true
  proxy_url: http://1.2.3.4:8080
  proxy_connect_header:
    #user1:password1
    #first with bcrypt and then with base64
    #Authorization: [$2a$10$6wrfos7.qvCq5qT7Kqn00udJnjwHqILVGZe8A56UabnySwrRHIXd2]
    Authorization: [dXNlcjE6cGFzc3dvcmQx]
  headers:
    THANOS-TENANT: ThanosTenantId
  tls_config:
    insecure_skip_verify: false
    cert_file: ..\Certs\hostcert.crt
    key_file: ..\Certs\key.unencrypted.pem

Regards,

Kesav

[Brian Candler]

The base64 one you've shown looks correct (user1:password1).  You definitely *don't* send a hash of the password; the plaintext is sent to the proxy, and the proxy hashes that value and compares it with its stored hash.

If you use tcpdump or wireshark, you can look at the raw request going to 1.2.3.4 port 8080 and confirm that it includes

Authorization: Basic dXNlcjE6cGFzc3dvcmQx

If it does, then the problem is with your proxy (e.g. it doesn't accept basic auth, or it doesn't like the username/password you have provided) and you'll need to look at the logs there to see why it was rejected.  Also, check the text of the HTTP response which may give you more clues.

If it doesn't, then please show what you see instead.

[Kesavanand Chavali]

we did a wireshark capture and found that below works. This we figured out from other already working clients with other software; e.g. some clients working with same proxy.

Could you please update the Prometheus documentation?

proxy_connect_header:
    Proxy-Authorization: [Basic dXNlcjE6cGFzc3dvcmQx]

[Brian Candler]

On Thursday, 2 March 2023 at 06:32:07 UTC Kesavanand Chavali wrote:

we did a wireshark capture and found that below works. This we figured out from other already working clients with other software; e.g. some clients working with same proxy.

Could you please update the Prometheus documentation?

Can you identify the part of the documentation which needs updating, and propose a change?

There is https://prometheus.io/docs/prometheus/latest/configuration/configuration/

but it is generic. It just shows

# Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ <string>: [<secret>, ...] ] ]

i.e. "send whatever headers you need to".

Ticket 11626 indeed seems to have a bad example, but you can comment on the ticket.

That specific comment was "generated with the help of chatgpt" which probably explains why it's wrong.  ChatGPT generating authoritative-sounding but wrong information is the new plague on our world :-(

[Julien Pivotto]

On 02 Mar 02:28, Brian Candler wrote:
> On Thursday, 2 March 2023 at 06:32:07 UTC Kesavanand Chavali wrote:
>
> we did a wireshark capture and found that below works. This we figured out
> from other already working clients with other software; e.g. some clients
> working with same proxy.
> Could you please update the Prometheus documentation?
>
>
> Can you identify the part of the documentation which needs updating, and
> propose a change?
>
> There is
> https://prometheus.io/docs/prometheus/latest/configuration/configuration/
> but it is generic. It just shows
>
> # Specifies headers to send to proxies during CONNECT requests. [
> proxy_connect_header: [ <string>

> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#string>:
> [<secret>
> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#secret>,

> ...] ] ]
>
> i.e. "send whatever headers you need to".
>

> Ticket 11626 <https://github.com/prometheus/prometheus/issues/11626> indeed

> seems to have a bad example, but you can comment on the ticket.
>

> That specific comment was "*generated with the help of chatgpt*" which

> probably explains why it's wrong. ChatGPT generating
> authoritative-sounding but wrong information is the new plague on our world

ChatGPT only reviewed the grammar here (the feature is too young, it
does not know it). The mistake was purely my human mistake, I have fixed
the issue now :)


> :-(
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e3bd9351-ce45-4b9f-a5ae-c5ef1cc660b3n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie