invalid UTF-8 label value on Swedish text

[John Dexter]

I understand that metric and label names are standard ASCII but values support Unicode, at least according to docs.

I am working on a Swedish-language system and just realised several of my targets are reporting this error, however their endpoints work just fine.

I have a metric with:

obfuscated_metric_name{location="Törn"} 0.00000000000000000

message_bytes_total{direction="sent",subsystem="ÖTPP"} 0.00000000000000000

I copy-pasted this from the endpoint apart from hiding the metric name for privacy reasons so what is wrong? Our code is quite old C++ so quite how these characters are handled is not obvious to me but we haven't had this issue before (we use these strings all over the place) and when I paste this text into a UTF8 validity checked, it reports no problems.

Does Prometheus have any configuration I can tweak otherwise I'm a bit stuck!

Thanks for any help.

[Ben Kochie]

Sounds like you may have some data that's encoded in a pre-UTF-8 format like https://en.wikipedia.org/wiki/Windows-1257 or https://en.wikipedia.org/wiki/ISO/IEC_8859.

You'll have a character encoding converter in your exporter in order to ingest the data.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGJRanhVcN%3DgsyF_QDPckNopVpUm1_4NY6cDPbxdjjKw4ZFGpA%40mail.gmail.com.

[John Dexter]

I wouldn't be surprised though it does validate online (unless copy-paste in chrome automatically changes the encoding!)

I'm using the prometheus-cpp library and I cannot find any mention of character support. Does Prometheus not have any way to convert incoming data?

The strings I am using as labels come from very old code which is quite fragile, so fixing it outside our code would be far preferable - other code may rely on it being in whatever encoding it is in.

[Julien Pivotto]

Do you have an up to date version of prometheus? I tryied this and it
workd for me.

Are you sure that those are the offending metrics?

Thanks.

On 25 Mar 10:47, John Dexter wrote:
> I wouldn't be surprised though it does validate online (unless copy-paste
> in chrome automatically changes the encoding!)
> I'm using the prometheus-cpp library and I cannot find any mention of
> character support. Does Prometheus not have any way to convert incoming
> data?
> The strings I am using as labels come from very old code which is quite
> fragile, so fixing it outside our code would be far preferable - other code
> may rely on it being in whatever encoding it is in.
>
> On Thu, 25 Mar 2021 at 10:38, Ben Kochie <sup...@gmail.com> wrote:
>
> > Sounds like you may have some data that's encoded in a pre-UTF-8 format
> > like https://en.wikipedia.org/wiki/Windows-1257 or
> > https://en.wikipedia.org/wiki/ISO/IEC_8859.
> >
> > You'll have a character encoding converter in your exporter in order to
> > ingest the data.
> >
> > On Thu, Mar 25, 2021 at 11:32 AM John Dexter <jo...@jdxsolutions.com>
> > wrote:
> >

> >> I understand that metric and label *names* are standard ASCII but
> >> *values* support Unicode, at least according to docs.

> >> I am working on a Swedish-language system and just realised several of my
> >> targets are reporting this error, however their endpoints work just fine.
> >>
> >> I have a metric with:
> >>
> >> obfuscated_metric_name{location="Törn"} 0.00000000000000000
> >>
> >> message_bytes_total{direction="sent",subsystem="ÖTPP"} 0.00000000000000000
> >>
> >> I copy-pasted this from the endpoint apart from hiding the metric name
> >> for privacy reasons so what is wrong? Our code is quite old C++ so quite
> >> how these characters are handled is not obvious to me but we haven't had
> >> this issue before (we use these strings all over the place) and when I
> >> paste this text into a UTF8 validity checked, it reports no problems.
> >> Does Prometheus have any configuration I can tweak otherwise I'm a bit
> >> stuck!
> >>
> >> Thanks for any help.
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Prometheus Users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an
> >> email to prometheus-use...@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/prometheus-users/CAGJRanhVcN%3DgsyF_QDPckNopVpUm1_4NY6cDPbxdjjKw4ZFGpA%40mail.gmail.com

> >> <https://groups.google.com/d/msgid/prometheus-users/CAGJRanhVcN%3DgsyF_QDPckNopVpUm1_4NY6cDPbxdjjKw4ZFGpA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGJRania%3DmzAgg%3DvSD0hjfd6%3D-CEswoMKkJPboPy8PK-L1J%3DTg%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie

[John Dexter]

It's the latest version as of a few weeks ago.

Other modules which are identical except their label values contain no special characters work fine so I am fairly confident this is the reason - should I be able to find more verbose logs telling me exactly what it didn't like because I could not find anything so far?

I have wondered if the clipboard is fixing this when I copy-paste so you can't reproduce!

[John Dexter]

A little more digging (I am not a web-dev so excuse any dumb things):

- when I look at the endpoint URL in Chrome, Developer tools->Network->Header I get the following response header:

- when I check what encoding Chrome has actually used (https://superuser.com/questions/1187374/check-the-character-encoding-of-a-given-page-in-chrome) I get:

document.characterSet
"windows-1252"

So it appears that my exporter (prometheus-cpp) is not specifying the encoding, Google Chrome is detecting to display in Windows-1252. Further inspection appears to show that it is encoded in Windows 1252.

So now what? Does Prometheus not support this encoding at all? Are there any ways to convert - say you were using an old bespoke exporter that used an older encoding and couldn't be modified?

ngth: 2091

[David Leadbeater]

Can you try:

curl host/metrics | promtool check metrics

Which will avoid any issues due to copying and means Prometheus sees
the raw bytes.

C++ std::string types (which is what prometheus-cpp uses:
https://jupp0r.github.io/prometheus-cpp/structprometheus_1_1ClientMetric_1_1Label.html)
don't have character set information associated with them, so it's
likely you need to convert them yourself as the API doesn't have
enough info to know what they are, e.g. using iconv() from ISO-8859-1
to UTF-8, which you then can put in a std::string.

David

On Thu, 25 Mar 2021 at 11:07, John Dexter <jo...@jdxsolutions.com> wrote:
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGJRanjLA7p-VUM8%3DefCUDiDA0DLMKUyZBRVvimA%2BoR484qCSw%40mail.gmail.com.

[John Dexter]

Thanks David,

That indeed flags the error and shows the special character is all messed up. I don't know iconv but if this is the issue and there's no way around it than fixing the raw text input, I can find a code solution. Shame Prometheus doesn't support any other encodings but heigh ho.

Thanks for the useful information.

[Julien Pivotto]

On 25 Mar 12:48, John Dexter wrote:
> Thanks David,
>
> That indeed flags the error and shows the special character is all messed
> up. I don't know iconv but if this is the issue and there's no way around
> it than fixing the raw text input, I can find a code solution. Shame
> Prometheus doesn't support any other encodings but heigh ho.

Please be careful with your words here :)

UTF-8 is currently quite standard in this area and this is properly
documented. I don't see it as a limitation of Prometheus.

Regards

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGJRanguAwkDryrkatDgkifiqCBPRRt%2Bhw3hZegP1j%3DUWRZFdw%40mail.gmail.com.

--
Julien Pivotto
@roidelapluie