EOF error with blackbox exporter

Eva Sheeva

unread,

Sep 19, 2019, 1:13:06 AM9/19/19

to Prometheus Users

Hi

I have a server with multiple services and there is an internal (non public) http endpoint which has certificate on port 50051. I have used blackbox exporter to notify us on cert expiration, which works fine with all other ports. But with the port 50051 i see an EOF error and i'm unable to find an answer for it. Can anyone please help. Below is what my blakcbox_exporter config look like :

======================

modules:

http_2xx:

prober: http

timeout: 5s

http:

valid_status_codes: [200,404]

method: GET

preferred_ip_protocol: ip4

tls_config:

insecure_skip_verify: true

tls_connect:

prober: tcp

tcp:

tls: true

tls_config:

# only if you don't want to/can't verify the validity of the certificate

insecure_skip_verify: true

======================

Logs showing EOF error:

Logs for the probe:

ts=2019-09-19T04:56:38.553501915Z caller=main.go:118 module=http_2xx target=PUBLICIP:50051 level=info msg="Beginning probe" probe=http timeout_seconds=4.5

ts=2019-09-19T04:56:38.553571141Z caller=utils.go:42 module=http_2xx target=PUBLICIP:50051 level=info msg="Resolving target address" ip_protocol=ip4

ts=2019-09-19T04:56:38.55360649Z caller=utils.go:75 module=http_2xx target=PUBLICIP:50051 level=info msg="Resolved target address" ip=PUBLICIP

ts=2019-09-19T04:56:38.553635643Z caller=http.go:349 module=http_2xx target=PUBLICIP:50051 level=info msg="Making HTTP request" url=http://PUBLICIP:50051 host=PUBLICIP:50051

ts=2019-09-19T04:56:38.554566404Z caller=http.go:364 module=http_2xx target=PUBLICIP:50051 level=error msg="Error for HTTP request" err="Get http://PUBLICIP:50051: EOF"

ts=2019-09-19T04:56:38.554596553Z caller=http.go:450 module=http_2xx target=PUBLICIP:50051 level=info msg="Response timings for roundtrip" roundtrip=0 start=2019-09-19T04:56:38.553697373Z dnsDone=2019-09-19T04:56:38.553697373Z connectDone=2019-09-19T04:56:38.554065361Z gotConn=2019-09-19T04:56:38.554072677Z responseStart=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z

ts=2019-09-19T04:56:38.554624319Z caller=main.go:131 module=http_2xx target=PUBLICIP:50051 level=error msg="Probe failed" duration_seconds=0.001098531

Metrics that would have been returned:

# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds

# TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 2.1521e-05

# HELP probe_duration_seconds Returns how long the probe took to complete in seconds

# TYPE probe_duration_seconds gauge

probe_duration_seconds 0.001098531

# HELP probe_failed_due_to_regex Indicates if probe failed due to regex

# TYPE probe_failed_due_to_regex gauge

probe_failed_due_to_regex 0

# HELP probe_http_content_length Length of http content response

# TYPE probe_http_content_length gauge

probe_http_content_length 0

# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects

# TYPE probe_http_duration_seconds gauge

probe_http_duration_seconds{phase="connect"} 0.000375244

probe_http_duration_seconds{phase="processing"} 0

probe_http_duration_seconds{phase="resolve"} 2.1521e-05

probe_http_duration_seconds{phase="tls"} 0

probe_http_duration_seconds{phase="transfer"} 0

# HELP probe_http_redirects The number of redirects

# TYPE probe_http_redirects gauge

probe_http_redirects 0

# HELP probe_http_ssl Indicates if SSL was used for the final redirect

# TYPE probe_http_ssl gauge

probe_http_ssl 0

# HELP probe_http_status_code Response HTTP status code

# TYPE probe_http_status_code gauge

probe_http_status_code 0

# HELP probe_http_version Returns the version of HTTP of the probe response

# TYPE probe_http_version gauge

probe_http_version 0

# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6

# TYPE probe_ip_protocol gauge

probe_ip_protocol 4

# HELP probe_success Displays whether or not the probe was a success

# TYPE probe_success gauge

probe_success 0

=========================

Prometheus configuration:

- job_name: 'test_50051'

metrics_path: /probe

params:

module: [http_2xx]

consul_sd_configs:

- server: localhost:8500

services:

- anx

relabel_configs:

- source_labels: [__meta_consul_address]

replacement: 'https://${1}:50051'

target_label: __param_target

- source_labels: [__param_target]

target_label: ip

- source_labels: [__meta_consul_node]

target_label: instance

- target_label: __address__

replacement: 127.0.0.1:9115 # The blackbox exporter

Thanks!

Eva Sheeva

unread,

Sep 20, 2019, 3:28:34 AM9/20/19

to Prometheus Users

Can I get some help on this please.

Eva Sheeva

unread,

Sep 23, 2019, 5:20:04 AM9/23/19

to Prometheus Users

Any help on this please?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c169fba9-2f20-4ef3-834e-7c8c2e2c747b%40googlegroups.com.

Eva Sheeva

unread,

Sep 24, 2019, 2:48:28 AM9/24/19

to Prometheus Users

Anything on this please?

Christian Hoffmann

unread,

Sep 25, 2019, 3:14:44 PM9/25/19

to Eva Sheeva, Prometheus Users

Hi Eva,

On 9/24/19 8:48 AM, Eva Sheeva wrote:
> ith multiple services and there is an internal (non public) http
> endpoint which has certificate on port 50051. I have used blackbox
> exporter to notify us on cert expiration, which works fine with all
> other ports. But with the port 50051 i see an EOF error and i'm
> unable to find an answer for it. Can anyone please help. Below is
> what my blakcbox_exporter config look like :

[...]

> ts=2019-09-19T04:56:38.554566404Z caller=http.go:364 module=http_2xx
> target=PUBLICIP:50051 level=error msg="Error for HTTP request"
> err="Get http://PUBLICIP:50051: EOF"

Have you tried the additional debugging strategies outlined here?
https://www.robustperception.io/debugging-blackbox-exporter-failures

Does using curl against that URL work?
Are you share that http:// is the right scheme? You've got a tls_config,
you are talking about certificates -- so I guess it might have to be
https://?

Kind regards,
Christian

ashma suv

unread,

Oct 7, 2019, 6:34:21 AM10/7/19

to Christian Hoffmann, Eva Sheeva, Prometheus Users

Hi Christian

The tls_config is for grpc endpoint.

When I run curl from prometheus to the server ,this is what i see

curl https://IP:50051
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f7194efd-08f2-9447-737f-eabab14e046f%40hoffmann-christian.info.

Eva Sheeva

unread,

Oct 7, 2019, 9:35:02 AM10/7/19

to Prometheus Users

Forgot to mention that i did try the debugging steps mentioned here https://www.robustperception.io/debugging-blackbox-exporter-failures and i see no reason for failures.

I had sent you the response of first method while third method also show same result. With second method I see "Probe id not found" for all the success and failed logs as well

On Thursday, September 19, 2019 at 10:43:06 AM UTC+5:30, Eva Sheeva wrote:

Christian Hoffmann

unread,

Oct 7, 2019, 1:20:40 PM10/7/19

to ashma suv, Eva Sheeva, Prometheus Users

On 10/7/19 12:34 PM, ashma suv wrote:
> curl https://IP:50051
> curl: (60) SSL certificate problem: self signed certificate in
> certificate chain

You are using TLS here (https://) which seems to work.

In your Prometheus/blackbox_exporter example, you attempt to use plain
HTTP (http://) against an SSL port.

In a short test, this mistake triggers EOF for me as well.

Try again with https://PUBLICIP:50051 :)

Kind regards,
Christian

Eva Sheeva

unread,

Oct 9, 2019, 4:47:49 AM10/9/19

to Christian Hoffmann, ashma suv, Prometheus Users

Hi Christian

I did try this : http://B_E_IP:9115/probe?target=https://targetIP:50051&module=http_2xx&debug=true and i still see EOF error.

Should I make any changes in the config?

Eva Sheeva

unread,

Oct 11, 2019, 2:05:20 PM10/11/19

to Christian Hoffmann, ashma suv, Prometheus Users

Any further tests or suggestion that you can think of to find out what the issue is?

Thanks

Christian Hoffmann

unread,

Oct 26, 2019, 8:27:00 AM10/26/19

to Eva Sheeva, ashma suv, Prometheus Users

Hi,

On 10/11/19 8:05 PM, Eva Sheeva wrote:
> Any further tests or suggestion that you can think of to find out what
> the issue is?

Sorry, don't have any other suggestions.

Is this plain HTTP? Which version? Does it support GET?
Is this maybe GRPC?

The latter will not work, apparently:
https://groups.google.com/forum/#!searchin/prometheus-users/grpc|sort:date/prometheus-users/WRkQ1MZSdS0/IhCCdoX_EAAJ

Maybe you can only use a TCP probe in this case.

Kind regards,
Christian

> On Wed, Oct 9, 2019 at 2:17 PM Eva Sheeva <sheev...@gmail.com
> <mailto:sheev...@gmail.com>> wrote:
>
> Hi Christian
>
> I did try this
> : http://B_E_IP:9115/probe?target=https://targetIP:50051&module=http_2xx&debug=true

> <http://35.154.215.68:9115/probe?target=https://13.233.100.246:50051&module=http_2xx&debug=true> and

> i still see EOF error.
>
> Should I make any changes in the config?
>
> On Mon, Oct 7, 2019 at 10:50 PM Christian Hoffmann

> <ma...@hoffmann-christian.info <mailto:ma...@hoffmann-christian.info>>

> wrote:
>
>
> On 10/7/19 12:34 PM, ashma suv wrote:
> > curl https://IP:50051
> > curl: (60) SSL certificate problem: self signed certificate in
> > certificate chain
> You are using TLS here (https://) which seems to work.
>
> In your Prometheus/blackbox_exporter example, you attempt to use
> plain
> HTTP (http://) against an SSL port.
>
> In a short test, this mistake triggers EOF for me as well.
>
> Try again with https://PUBLICIP:50051 :)
>
> Kind regards,
> Christian
>
> >
> > On Thu, Sep 26, 2019 at 12:44 AM Christian Hoffmann
> > <ma...@hoffmann-christian.info
> <mailto:ma...@hoffmann-christian.info>

> <mailto:ma...@hoffmann-christian.info

Eva Sheeva

unread,

Nov 7, 2019, 3:48:52 AM11/7/19

to Prometheus Users

Hi Christian

I'm told that the port that I'm scraping is a gRPC port and not an HTTP port. I then made the change accordingly as below but I now get 'no suitable address' found error. Below is the detail :

Blackbox config file:

modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [200,404]
method: GET
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: true

tls_connect:
prober: tcp
tcp:
tls: true
tls_config:
# only if you don't want to/can't verify the validity of the certificate
insecure_skip_verify: true

Prometheus.yml

- job_name: 'BE_P50051'
metrics_path: /probe
params:
module: [tls_connect]

consul_sd_configs:
- server: localhost:8500
services:

- prodB

relabel_configs:
- source_labels: [__meta_consul_address]
replacement: 'https://${1}:50051'
target_label: __param_target
- source_labels: [__param_target]
target_label: ip
- source_labels: [__meta_consul_node]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115 # The blackbox exporter

When accessing http://BE_IP:9115/probe?target=ip:50051&module=tls_connect&debug=true I get below response:

Logs for the probe:
ts=2019-11-07T08:26:22.170241302Z caller=main.go:118 module=tls_connect target=IP:50051 level=info msg="Beginning probe" probe=tcp timeout_seconds=9.5
ts=2019-11-07T08:26:22.17028689Z caller=utils.go:42 module=tls_connect target=IP:50051 level=info msg="Resolving target address" ip_protocol=ip6
ts=2019-11-07T08:26:22.170334528Z caller=utils.go:55 module=tls_connect target=IP:50051 level=warn msg="Resolution with IP protocol failed, attempting fallback protocol" fallback_protocol=ip4 err="address IP: no suitable address found"
ts=2019-11-07T08:26:22.170349653Z caller=utils.go:75 module=tls_connect target=IP:50051 level=info msg="Resolved target address" ip=IP
ts=2019-11-07T08:26:22.170363309Z caller=tcp.go:88 module=tls_connect target=IP:50051 level=info msg="Dialing TCP with TLS"
ts=2019-11-07T08:26:22.182315594Z caller=tcp.go:106 module=tls_connect target=IP:50051 level=error msg="Error dialing TCP" err=EOF
ts=2019-11-07T08:26:22.182344905Z caller=main.go:131 module=tls_connect target=IP:50051 level=error msg="Probe failed" duration_seconds=0.012078407

Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 3.0527e-05

# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge

probe_duration_seconds 0.012078407

# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0

# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0

Can you please help.

Thanks

On Saturday, October 26, 2019 at 5:57:00 PM UTC+5:30, Christian Hoffmann wrote:

Hi,

On 10/11/19 8:05 PM, Eva Sheeva wrote:
> Any further tests or suggestion that you can think of to find out what
> the issue is?

Sorry, don't have any other suggestions.

Is this plain HTTP? Which version? Does it support GET?
Is this maybe GRPC?

The latter will not work, apparently:
https://groups.google.com/forum/#!searchin/prometheus-users/grpc|sort:date/prometheus-users/WRkQ1MZSdS0/IhCCdoX_EAAJ

Maybe you can only use a TCP probe in this case.

Kind regards,
Christian

> On Wed, Oct 9, 2019 at 2:17 PM Eva Sheeva <sheev...@gmail.com
> <mailto:sheev...@gmail.com>> wrote:
>
> Hi Christian
>
> I did try this
> : http://B_E_IP:9115/probe?target=https://targetIP:50051&module=http_2xx&debug=true

Reply all

Reply to author

Forward