Getting EIF Error

[ritesh patel]

Hello Team, 

I have promethus installed on kubernet and add scraping targets from cmdb using http_sd_config. 

I am getting EOF error while i add targets. And it’s show down. 

Can someone please help me what is the issue. 

For your reference attached screenshot of error. 

Thanks and regards

Ritesh patel 

[Brian Candler]

The straightforward way to debug this by doing a scrape by hand: it's just a HTTP request.

curl -g 'https://blah.local:9126/metrics'

Add flag '-v' for more debugging if required (e.g. response headers may give you an extra clue).  Once you're able to scrape the exporter with curl, then prometheus should be able to talk to it too.

Given that it's https, if you get a certificate error then you can add flag '-k' to skip certificate verification.  If that turns out to be the problem, then there are extra flags you can pass to curl, e.g. to pass the CA root certificate (if the target's cert was signed by a CA that's not in the system trust store).  Once you've got all that working, you can make the corresponding changes to prometheus' tls_config.

[Markus Glück]

Hi guys,

I currently trying to debug a similar issue. I am also getting EOF only as error in debug level from scrape manager. I had the same issue locally with curl and LibreSSL on macOS. Updating the curl version to use openSSL fixed it. So my assumption it's something related to SSL/TLS. My working curl verbose output gives me this in regards to TLS:

* ALPN: offers h2

* ALPN: offers http/1.1

} [5 bytes data]

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

} [512 bytes data]

* TLSv1.3 (IN), TLS handshake, Server hello (2):

{ [193 bytes data]

* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

} [1 bytes data]

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

} [512 bytes data]

* TLSv1.3 (IN), TLS handshake, Server hello (2):

{ [155 bytes data]

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

{ [51 bytes data]

* TLSv1.3 (IN), TLS handshake, Certificate (11):

{ [4943 bytes data]

* TLSv1.3 (IN), TLS handshake, CERT verify (15):

{ [520 bytes data]

* TLSv1.3 (IN), TLS handshake, Finished (20):

{ [52 bytes data]

* TLSv1.3 (OUT), TLS handshake, Finished (20):

} [52 bytes data]

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

* ALPN: server accepted h2

but HOW do I get more info from Prometheus?

Thx,

  Markus

[Markus Glück]

promtool debug metrics https://myserver:8181

collecting: https://myserver:8181/metrics

2022/09/09 11:15:05 http2: Transport failed to get client conn for myserver:8181: http2: no cached connection was available

error completing debug command: error executing HTTP request: Get "https://myserver:8181/metrics": EOF

If I open the url in a browser or with curl I get proper response :( Any hint what I might be missing? testssl.sh works fine too

[Brian Candler]

What prometheus version are you using?  I'd suggest v2.37.0 (2.37 is an LTS release branch)

"promtool debug metrics" doesn't seem to have many options, but you could try in your prometheus scrape config:

enable_http2: false

and/or

tls_config:

  insecure_skip_verify: true

to try and narrow down the problem.

[Brian Candler]

Note: the ability to disable http2 via scrape config was only added in v2.35.0.

For an older version, you could try environment variables

DISABLE_HTTP2=1

and/or

GODEBUG=http2client=0

[Markus Glück]

Hi Brian,

thanks for these tips. I already tried those before my post, but they didn't help.

In the meantime I did some more investigation. The servers creating the metrics are Payara Micro 5.2021.10

The Prometheus (actually Thanos) is v2.38.0. Strangly it works with 2.12.0 (old on prem installation that got replaced), and is broken with every version starting at 2.13.0.

Should be quite obvious to figure out, but I don't see any changes in 2.13.0 that I would pinpoint this too.

I asked the guys running the servers if they could look into their config. e.g. disabling h2 didn't help either.

Very strange issue :(

Cu

 Markus

[Brian Candler]

One problem that affected me was that go version 1.15+ removed the ability to use a certificate which has only "commonName".  It now requires "subjectAltName" to be present. That, I believe, would have affected prometheus 2.21.0 onwards, so it doesn't explain your change at 2.13.0.

Prometheus 2.12.0 was built with go1.12.8, and 2.13.0 was built with go1.13.1, so it could be a similar issue.

Now, looking at the go1.13 release notes, I see that TLS 1.3 was enabled by default then - there's more information about go's TLS 1.3 support in the go1.12 release notes when it was made available but opt-in only.

Perhaps that's the issue - your target server has a broken TLS 1.3 implementation?  If so, forcing TLS 1.2 might be a workaround.  Unfortunately, it looks like prometheus' tls_config in scrape jobs allows setting a minimum TLS version, but not a  maximum :-(

That's weird, because when configuring prometheus' own https endpoint, you can select both min and max versions.

Maybe disabling TLS 1.3 in the exporter itself is an option?

[Brian Candler]

You could also prove this with prometheus 2.13.0, by setting environment variable GODEBUG=tls13=0 and see if it starts to work.

(That workaround was removed in later versions of go, so it's not usable in production - but at least it will demonstrate whether this is the problem or not)

[Markus Glück]

You're da man! :) GODEBUG=tls13=0 in 2.13.0 works. so its an incompatibility between newer GO versions and Payara.

Now I have to push the server owners to disable the TLS1.3 for the metrics endpoints, since I need to use a more recent payara (not able to rollback the data and also High CVE in GoVersion of 2.35.0 or so).

But thx very much to give the definite hint that proved the causing issue!

CU

  Markus

[Markus Glück]

ok, we found a solution. One of the servers worked all the time. Turns out it had a different JDK installed (OpenJDK Zulu vs normal OpenJDK, both 11). Using the Zulu JDK on the failing ones fixes the issue.