Prometheus v2.6.0 - unexpected end of data, got \"EOF\""

Paul Seymour

unread,

Jan 4, 2019, 12:30:17 PM1/4/19

to Prometheus Users

Hello,

Got another issue with our custom exporter in Prometheus 2.6 (worked in 2.5 and below).

We are now getting-

msg="append failed" err="unexpected end of data, got \"EOF\""

Errors - it's not a timeout as it's well below the duration/timeout values.

Curl'ing the data looks good and piping through "promtool check metrics" complains about the lack of help text but that is all.

The only thing I can possible see it that some parts have spaces in (some don't) for example:-

msg_queues_is_durable{server="myserver",vpn="#config-sync",queue="#CFGSYNC/OWNER/ONE/VPN/TWO VPNNAME/CFG"} 1.0

But this has never been an issue before, this from the openmetricsparse_test.go:-

func TestOpenMetricsParseErrors(t *testing.T) {
	cases := []struct {
	input string
	err string
	}{
	{
	input: "",
	err: "unexpected end of data, got \"EOF\"",
	},

Is the only reference I can see to it but no idea why it's getting caught ?

Any help much appreciated.

Thanks.

Paul Seymour

unread,

Jan 4, 2019, 1:01:10 PM1/4/19

to Prometheus Users

For the record running the exact same through v2.5.x works just fine.

Brian Brazil

unread,

Jan 4, 2019, 2:21:06 PM1/4/19

to Paul Seymour, Prometheus Users

On Fri, 4 Jan 2019 at 17:30, 'Paul Seymour' via Prometheus Users <promethe...@googlegroups.com> wrote:

Hello,

Got another issue with our custom exporter in Prometheus 2.6 (worked in 2.5 and below).

We are now getting-
msg="append failed" err="unexpected end of data, got \"EOF\""

What version of the Python client are you using?

Brian

Errors - it's not a timeout as it's well below the duration/timeout values.

Curl'ing the data looks good and piping through "promtool check metrics" complains about the lack of help text but that is all.

The only thing I can possible see it that some parts have spaces in (some don't) for example:-

msg_queues_is_durable{server="myserver",vpn="#config-sync",queue="#CFGSYNC/OWNER/ONE/VPN/TWO VPNNAME/CFG"} 1.0

But this has never been an issue before, this from the openmetricsparse_test.go:-
func TestOpenMetricsParseErrors(t *testing.T) {
cases := []struct {
input string
err string
}{
{
input: "",
err: "unexpected end of data, got \"EOF\"",
},

Is the only reference I can see to it but no idea why it's getting caught ?

Any help much appreciated.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/37b023e3-2141-406c-b094-9cec596f256b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 5, 2019, 2:05:35 AM1/5/19

to Prometheus Users

On Friday, 4 January 2019 19:21:06 UTC, Brian Brazil wrote:

On Fri, 4 Jan 2019 at 17:30, 'Paul Seymour' via Prometheus Users <promethe...@googlegroups.com> wrote:
Hello,

Got another issue with our custom exporter in Prometheus 2.6 (worked in 2.5 and below).

We are now getting-
msg="append failed" err="unexpected end of data, got \"EOF\""

What version of the Python client are you using?

Brian

Hi,

It's not a Python client but a Java one.

Cheers.

Brian Brazil

unread,

Jan 5, 2019, 6:45:16 AM1/5/19

to Paul Seymour, Prometheus Users

It sounds like something is killing the connection then, I'd suggest debugging with tcpdump.

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 4:34:43 AM1/7/19

to Prometheus Users

Hi,

That doesn't explain why it would work with 2.5.0 and not 2.6.0 I cannot see any resets in the stream.

Thanks

Paul

Brian Brazil

unread,

Jan 7, 2019, 4:37:35 AM1/7/19

to Paul Seymour, Prometheus Users

What's the last few lines of the output, including newlines?

--

Brian Brazil

www.robustperception.io

Ben Kochie

unread,

Jan 7, 2019, 5:16:28 AM1/7/19

to Brian Brazil, Paul Seymour, Prometheus Users, Matthias Rampke

We should not be breaking scrapes, even if they're not technically correct, without a major version bump.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLoSKSSMuK5nRS_-ABLj5iXe%3DjSAF88FhWm-1Gw4MHjEZg%40mail.gmail.com.

Brian Brazil

unread,

Jan 7, 2019, 5:21:12 AM1/7/19

to Ben Kochie, Paul Seymour, Prometheus Users, Matthias Rampke

On Mon, 7 Jan 2019 at 10:16, Ben Kochie <sup...@gmail.com> wrote:

We should not be breaking scrapes, even if they're not technically correct, without a major version bump.

No, it's fine to break things that are broken. We first need to figure out what's going on here though.

Brian

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 5:57:33 AM1/7/19

to Prometheus Users

What's the last few lines of the output, including newlines?

--
Brian Brazil
www.robustperception.io

solace_msgvpns_bridges_is_bound_to_bridge_queue{server="MPDIM502-mgt1",vpn="200004246_SOL_VPN",bridge="200004246_SOL_VPN_B_200001342_SOL_VPN"} 0.0$

solace_derived_data_svc_ok{server="MPDIM502-mgt1"} 0.0$

solace_derived_msg_spool_ok{server="MPDIM502-mgt1"} 0.0$

solace_scrape_duration{server="MPDIM502-mgt1"} 15.535$

Paul Seymour

unread,

Jan 7, 2019, 6:13:07 AM1/7/19

to Prometheus Users

I have also rolled this back to 2.5.0 and forward to 2.6.0 a few times and it's only 2.6.0 that throws the EOF error.

We do have some crazy lines like this:-

solace_msgvpns_queues_quota_in_mb{server="MPDIM502-mgt1",vpn="#config-sync",queue="#CFGSYNC/OWNER/MPDIM501/VPN/200005076 t_anaid10_E0SOLVPN/CFG"} 200.0

solace_msgvpns_queues_consumer_count{server="MPDIM502-mgt1",vpn="#config-sync",queue="#CFGSYNC/OWNER/MPDIM501/VPN/200005076 t_anaid10_E0SOLVPN/CFG"} 0.0

solace_msgvpns_queues_is_enabled{server="MPDIM502-mgt1",vpn="#config-sync",queue="#CFGSYNC/OWNER/MPDIM501/VPN/200005076 t_anaid10_E0SOLVPN/CFG"} 1.0

But why they are fine on 2.5.0 and not 2.6.0 is an issue for us.

Thanks for all the help.

Brian Brazil

unread,

Jan 7, 2019, 8:33:24 AM1/7/19

to Paul Seymour, Prometheus Users

What are the headers going each way, and which java client is it?

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 9:12:23 AM1/7/19

to Prometheus Users

What are the headers going each way, and which java client is it?

--
Brian Brazil
www.robustperception.io

It's someone else's code but it's based off of this:-

https://github.com/koverton/solace-monitoring-extension

Which is public. Will see if I can get the headers.

Thanks

Brian Brazil

unread,

Jan 7, 2019, 9:18:05 AM1/7/19

to Paul Seymour, Prometheus Users

I can't find any Prometheus-related code in there, or even a http server. Can you point me at the relevant parts?

Brian

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ee33273d-e690-4794-baa7-0f2cb6f6bb61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 9:40:43 AM1/7/19

to Prometheus Users

I can't find any Prometheus-related code in there, or even a http server. Can you point me at the relevant parts?

Sadly our's isn't public and I don't think I can share a whole bunch of code this is a parser for some data:-

public String convertToPrometheusString(Map<String, Object> metrics, String prefix, Map<String, String> labels) {

StringBuilder sb = new StringBuilder();

metrics.entrySet().forEach(entry -> {

String metric = metricFixer(prefix, entry.getKey(), labels);

try {

Double value = entry.getValue() == null ? 0 : Double.valueOf(entry.getValue().toString());

sb.append(combineMetricsAndLabels(metric, labels)).append(' ').append(value).append("\n");

} catch (NumberFormatException | NullPointerException e) {

logger.info("FAILED: " + metric + " --- " + entry.getValue());

}

});

return sb.toString();

}

private String metricFixer(String prefix, String metric, Map<String, String> labels) {

String newMetric = "";

if (prefix.toLowerCase().startsWith("services") && metric.toLowerCase().endsWith("restportup")) {

// Takes care of the VPN inside of the metric name for services -- msg vpns rest port up

labels.put("vpn", metric.substring(metric.indexOf('|')+1, metric.lastIndexOf('|')));

newMetric = "MsgVpnsRestPortUp";

} else {

newMetric = metric;

}

String start = prefix.toLowerCase().startsWith("solace") ? prefix : "solace_" + prefix;

start = start.replaceAll("([A-Z])", "_$1").toLowerCase();

String metricFixed = newMetric.replaceAll("[^a-zA-Z0-9_]", "")

.replaceAll("([A-Z])", "_$1")

.replaceAll("^s_m_f", "smf")

.toLowerCase();

return ((start.startsWith("_") ? start.substring(1) : start) +

"_" + metricFixed)

.replaceAll("_+", "_")

.replaceAll("m_b$", "mb")

.replaceAll("msg_vpns", "msgvpns");

}

private String combineMetricsAndLabels(String metric, Map<String, String> labels) {

StringBuilder metricAndLabels = new StringBuilder(metric);

metricAndLabels.append('{');

labels.entrySet().forEach(entry ->

metricAndLabels.append(entry.getKey())

.append("=\"")

.append(entry.getValue().replaceFirst("^\\\\\\$", ""))

.append("\","));

metricAndLabels.setLength(metricAndLabels.length()-1); // remove trailing ','

metricAndLabels.append('}');

return metricAndLabels.toString();

}

Not sure if this helps at all.

Paul Seymour

unread,

Jan 7, 2019, 10:59:40 AM1/7/19

to Prometheus Users

Convinced this is because this exporter doesn't appear to be sending metrics data back in plaintext encoding. But it's not my code so cannot say for sure.

Brian Brazil

unread,

Jan 7, 2019, 12:57:30 PM1/7/19

to Paul Seymour, Prometheus Users

On Mon, 7 Jan 2019 at 15:59, 'Paul Seymour' via Prometheus Users <promethe...@googlegroups.com> wrote:

Convinced this is because this exporter doesn't appear to be sending metrics data back in plaintext encoding. But it's not my code so cannot say for sure.

It should be encoded as utf-8, what is the content-type?

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 12:59:20 PM1/7/19

to Prometheus Users

On Monday, 7 January 2019 15:59:40 UTC, Paul Seymour wrote:

Convinced this is because this exporter doesn't appear to be sending metrics data back in plaintext encoding. But it's not my code so cannot say for sure.

This was true. It was the encoding. It's not gzip and the type wasn't coming in as text/plain

I think this commit was the one that changed the behaviour.

https://github.com/prometheus/prometheus/pull/4912/files

The error message was a little odd. But makes sense ultimately.

Thanks again to Brian for pointing me in the right direction on this. Using https and the endpoint on k8s and going through a few layers made it fun :)

Brian Brazil

unread,

Jan 7, 2019, 1:06:50 PM1/7/19

to Paul Seymour, Prometheus Users

On Mon, 7 Jan 2019 at 17:59, 'Paul Seymour' via Prometheus Users <promethe...@googlegroups.com> wrote:

On Monday, 7 January 2019 15:59:40 UTC, Paul Seymour wrote:

Convinced this is because this exporter doesn't appear to be sending metrics data back in plaintext encoding. But it's not my code so cannot say for sure.

This was true. It was the encoding. It's not gzip and the type wasn't coming in as text/plain

I think this commit was the one that changed the behaviour.

https://github.com/prometheus/prometheus/pull/4912/files

Something is still not right here, if we don't recognise the content-type we should be treating it the same as text/plain. What exact headers were you sending back?

Brian

The error message was a little odd. But makes sense ultimately.

Thanks again to Brian for pointing me in the right direction on this. Using https and the endpoint on k8s and going through a few layers made it fun :)

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7cdf1a7c-8b18-48e6-9e76-32543b800af8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 1:07:26 PM1/7/19

to Prometheus Users

It should be encoded as utf-8, what is the content-type?

The charset was UTF-8 but the was text/html, forcing plain text made it work as it's not gzip'ed I think the commit I referenced in the other mail was the one that changed the behaviour between versions.

Thanks so much for the help, and your time.

Brian Brazil

unread,

Jan 7, 2019, 1:09:40 PM1/7/19

to Paul Seymour, Prometheus Users

On Mon, 7 Jan 2019 at 18:07, 'Paul Seymour' via Prometheus Users <promethe...@googlegroups.com> wrote:

It should be encoded as utf-8, what is the content-type?

The charset was UTF-8 but the was text/html, forcing plain text made it work as it's not gzip'ed I think the commit I referenced in the other mail was the one that changed the behaviour between versions.

That should still have worked.

Thanks so much for the help, and your time.

--

Brian Brazil

www.robustperception.io

Paul Seymour

unread,

Jan 7, 2019, 1:17:41 PM1/7/19

to Prometheus Users

Something is still not right here, if we don't recognise the content-type we should be treating it the same as text/plain. What exact headers were you sending back?

Brian

It's fixed now but this is all I have from fiddling with it before (attached)

The springboot code now forces it with this kind of thing:-

@GetMapping(produces = MediaType.TEXT_PLAIN_VALUE)

This wasn't forced previously

reqresp.png

Reply all

Reply to author

Forward