Prometheus transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x02\""

[ashma suv]

Hello 

I'm trying to configure Prometheus Blackbox_Exporter for the ips with specific ports where there are certs installed and i notice the value of 'probe_http_ssl' to be 0. When running http://Prom_ip:9115/probe?target=ip:port&module=http_2xx&debug=true i see the error "transport connection broken: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x02\""

Can someone please help me with this.

Thanks!

[ashma suv]

To add to, this is how my blackbox.yml file look like:

modules:

  http_2xx:

    prober: http

    timeout: 5s

    http:

      valid_status_codes: []

      method: GET

      preferred_ip_protocol: ip4

      tls_config:

        insecure_skip_verify: true

And this is the blackbox config in prometheus.yml 

- job_name: 'blackbox'

    metrics_path: /probe

    params:

      module: [http_2xx]

    static_configs:

     - targets:

        - list of ips

[mohd zakir]

I am running my stuff on docker 17.05.0-ce on ubuntu 16.04

I am trying to monitor web application 

Not exact to your point 

But this is not correct, it’s minoring by itself I mean host not application.

docker-compose.yml

  black-exporter:

    image: prom/blackbox-exporter

    hostname: black-exporter

    volumes:

      - ./blackbox/blackbox.yml:/config/blackbox.yml

    command:

      - '--config.file=/config/blackbox.yml'

    ports:

      - '9115:9115'

    networks:

      - back-tier

    links:

      - prometheus:prometheus

    restart: always

blackbox.yml

modules:

  http_check:

    prober: http

    timeout: 10s

    http:

      valid_http_versions: ["HTTP/1.1", "HTTP/2"]

      valid_status_codes: []

      method: GET

      no_follow_redirects: false

      tls_config:

        insecure_skip_verify: false

      preferred_ip_protocol: "ip4"

 #     fail_if_matches_regexp:

#       - "Could not connect to Database"

prometheus.yml

- job_name: 'blackbox'

    metrics_path: /probe

    params:

      module: [http_check]

    static_configs:

      - targets:

        - http://www.google.com

    relabel_configs:

      - source_labels: [__address__]

        target_label: __param_target

      - source_labels: [__param__target]

        target_label: instance

      - target_label: __address__

        replacement: 10.10.10.X:9115  #Blackbox exporter

It monitoring by itself(host=10.10.10.X) not for application. Please look at the attached screenshot of Targets

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b33cfa16-9bf1-4355-802d-f49043d049b8%40googlegroups.com.

[Simon Pasquier]

It looks like the blackbox exporter is trying to probe an HTTPS endpoint.
Try:
http://blacbox_exporter/probe?target=https://ip:port&module=http_2xx&debug=true

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f68c30f0-cc93-4b1c-83c4-72db76c1fb0b%40googlegroups.com.

[ashma suv]

Hi Simon 

I tried with https as you mentioned and i get the below:

https://ip:port net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x00\\x00\\x00\\x04\\x00\\x00\\x00\\x00\\x00\""

Thanks

[Simon Pasquier]

Have you checked the probed endpoint with curl or equivalent?

[ashma suv]

Running a curl on https with the port number from Prometheus server returns this which is what expected:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option. 

Curl with http returns nothing

Thanks

[Simon Pasquier]

Sorry but I'm not certain to understand if your endpoint is HTTP or HTTPS.
Can you share the output of "curl -i ..." for both http and https?

[Simon Pasquier]

Adding back prometheus-users.

Try again with "curl -v ..." please.



On Mon, Jul 29, 2019 at 11:40 AM ashma suv <ashm...@gmail.com> wrote:
>
> curl -i https://ip:port returns this:

>
> curl: (60) SSL certificate problem: unable to get local issuer certificate
> More details here: https://curl.haxx.se/docs/sslcerts.html
>
> curl performs SSL certificate verification by default, using a "bundle"
> of Certificate Authority (CA) public keys (CA certs). If the default
> bundle file isn't adequate, you can specify an alternate file
> using the --cacert option.
> If this HTTPS server uses a certificate signed by a CA represented in
> the bundle, the certificate verification probably failed due to a
> problem with the certificate (it might be expired, or the name might
> not match the domain name in the URL).
> If you'd like to turn off curl's verification of the certificate, use
> the -k (or --insecure) option.

> You have new mail in /var/mail/root
>
> Whereas curl -i http://ip:port returns nothing
>
> Thanks

[ashma suv]

Hi Simon

This is what i get with https and http:

================ 

WITH HTTPS:

curl -v  https://IP:PORT
* Rebuilt URL to: https://IP:PORT/
*   Trying IP...
* TCP_NODELAY set
* Connected to IP (IP) port IP (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

WITH HTTP: 

 curl -v  http://IP:port
* Rebuilt URL to: http://IP:PORT/
*   Trying IP...
* TCP_NODELAY set
* Connected to IP (IP) port PORT (#0)
> GET / HTTP/1.1
> Host: IP:PORT
> User-Agent: curl/7.52.1
> Accept: */*
>

* Curl_http_done: called premature == 0
* Connection #0 to host IP left intact

[ashma suv]

Hello Simon

Can you please update me on this. 

Thanks and Regards!

[ashma suv]

Hello

Can I please get an answer on this.

Thanks

[Simon Pasquier]

It seems that the endpoint speaks HTTPS but the HTTP isn't valid.
Try "curl -vik https://..."

[ashma suv]

This is the response

* Rebuilt URL to: https://IP:PORT/
*   Trying IP...
* TCP_NODELAY set

* Connected to IP (IP) port PORT (#0)

* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:

*   CAfile: <Cert path>
  CApath: <Cert path>

* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=XXX; ST=XXX; L=XXXX; O=XXXX; OU=TEST; CN=abc..
*  start date: Nov 17 00:45:25 2018 GMT
*  expire date: Nov 17 00:45:25 2019 GMT
*  issuer: C=US; ST=XXX; L=XXXX; O=XXXX; OU=TEST; CN=ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56171089be80)

> GET / HTTP/1.1
> Host: IP:PORT
> User-Agent: curl/7.52.1
> Accept: */*
>

* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< content-type: application/grpc
content-type: application/grpc
< grpc-status: 8
grpc-status: 8
< grpc-message: malformed method name: "/"
grpc-message: malformed method name: "/"

<
* Curl_http_done: called premature == 0
* Connection #0 to host IP left intact

Thanks

[Simon Pasquier]

It looks like this is a gRPC endpoint. Prometheus only speaks HTTP.

[ashma suv]

Ok, Thanks for the update Simon. These endpoints have certificates installed. Do we have any other option to have these certs monitored for expiration?

Thanks

[Simon Pasquier]

You can add a prober module like this to your blackbox configuration
and curl 'http://.../probe?target=<endpoint:port>&module=tls_connect'.

tls_connect:
prober: tcp
tcp:
tls: true
tls_config:
# only if you don't want to/can't verify the validity of the certificate
insecure_skip_verify: true