Prometheus collect the metric fail

[易Richard]

Prometheus collect endpoint fail. The error msg is "expected equal, got INVALID"

I checked the stderr output but didn't find any clue about the error. 

The metric content  from the endpoint fail to collect seems normal.

Do I need upload the whole metric content  cause it's too long？

[Brian Candler]

First try a couple of simple things:

curl http://10.70.20.10:8080/metrics -o metrics.txt

grep INVALID metrics.txt

If the grep doesn't find anything, then examine the file manually for structural errors.  Maybe you have a label name with an invalid character? The valid characters in label names are documented here.  Or even a label or value which has a newline in it?

foo{bar

="baz"} 123

You can also try a binary chop:

- split the file metrics.txt into two halves

- serve one half via a webserver (e.g. apache) as a static file

- scrape this via prometheus

- if there's no error, try the second half

- when you find the bad half, repeat the process (split that piece into two)

[Mike Spreitzer]

Would it be so hard to make the error message include the position where the problem was?

Thanks,
Mike

promethe...@googlegroups.com wrote on 11/12/2021 01:06:11 AM:

> From: "易Richard" <ricyy...@gmail.com>

> To: "Prometheus Users" <promethe...@googlegroups.com>
> Date: 11/12/2021 01:06 AM
> Subject: [EXTERNAL] [prometheus-users] Prometheus collect the metric fail
> Sent by: promethe...@googlegroups.com

>
> Prometheus collect endpoint fail. The error msg is "expected equal,
> got INVALID" I checked the stderr output but didn't find any clue

> about the error. The metric content from the endpoint fail to
> collect seems normal. ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

> Prometheus collect endpoint fail. The error msg is "expected equal,
> got INVALID"
> I checked the stderr output but didn't find any clue about the error.

>
> [image removed]

>
> The metric content  from the endpoint fail to collect seems normal.
> Do I need upload the whole metric content  cause it's too long？

> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/prometheus-users/
> a9b67778-4657-4dd9-9a01-044094f7bc3an%40googlegroups.com.

[Brian Candler]

Probably not - it just needs to keep a count of the line number for each metric.  I expect a PR implementing this would be considered favourably.

However, are you generating your metrics via a standard client library, or ad-hoc?  It might be worth doing better validation at the client side, so that bad metrics are caught before being emitted.

[易Richard]

Thx for the reply.

The grep didn't find anything. And I didn't find any invalid character.

The metric was generated in Apache Pulsar Broker Service 2.8.1 (Via Prometheus Java Client). org/apache/pulsar/broker/stats/prometheus/metrics

As for the binary chop method, it's really a bit complicated....
I uploaded the whole metric text.

[Brian Candler]

> As for the binary chop method, it's really a bit complicated....

I'm not going to do it for you.

If this is broken metric output from pulsar, then I suggest you take your problem to the pulsar mailing list.  I suggest you'll likely get a quicker response if you can help them by proving which metric(s) are invalid, which means doing the binary chop.

[Dan S]

You can use promtool for this:

```

cat metrics.txt | promtool check metrics

error while linting: text format parsing error in line 444: unexpected end of label value "persistent://public/default/"

```

these metrics have values with quotes in them, there's a several:

pulsar_subscriptions_count{cluster="pulsar-cluster-zk-1",namespace="public/default",topic="persistent://public/default/"qwerdf""} 1.0 1636946709604

pulsar_producers_count{cluster="pulsar-cluster-zk-1",namespace="public/default",topic="persistent://public/default/"qwerdf""} 0.0 1636946709604

Dan

[Dan S]

I should include the relevant article: 

https://www.robustperception.io/invalid-is-not-a-valid-start-token-and-other-scrape-errors

(and in other news I need to figure out how to paste code blocks)

[Brian Candler]

Thanks Dan, that's very neat!

(For code blocks, in groups.google.com web interface anyway, I just select the text and set font to 'Courier New'.  I can't see a less dumb way of doing this)