Limiting the blast radius of OTel / UTF-8 support for normal Prometheus users?
95 views
Skip to first unread message
Julius Volz
May 24, 2024, 3:00:29 AMMay 24
to Prometheus Developers
Hi,
While others are figuring out how to add UTF-8 support and other OTel compatibility functionality to Prometheus, my brain has been trying to figure out what all of this will mean for the normal Prometheus user, how we should explain this new optionality in Prometheus, and what we should recommend with regards to UTF-8 usage outside of the OTel compatibility scenario.
What (literally) keeps me up at night the most is the new selector syntax that metric names with previously illegal characters will now require: https://docs.google.com/document/d/1yFj5QSd1AgCYecZ9EJ8f2t4OgF2KBZgJYVde-uzVEtI/edit#heading=h.yxv3hzhog2l2.
In this proposal, Björn did the best he could given the difficult situation that OTel compatibility support push puts us in, and I also don't have better ideas for nicer syntaxes. However, it looks like all current selector syntax suggestions require putting the metric name not only in quotes, but also within the curly braces, and possibly even introducing new (IMO) very odd short-hand syntaxes for it.
Given that I think this new selector syntax is very unelegant, inferior, and confusing (harder to type, harder to read and understand intuitively) compared to Prometheus' really clean and nice existing selector syntax that we've had for over 10 years, how should we approach UTF-8 support when explaining it to users? Should we say, "Yes, the Prometheus data model now allows any UTF-8 characters in metric names, but you should really ignore this fact completely and stick to the old character set, unless you have to deal with UTF-8 for the purpose of OTel metrics because otherwise your PromQL will start looking really weird"? Or do we not give any recommendations around it at all?
I just fear that UTF-8 characters will start creeping in everywhere because people see that it's supported. And then all users will eventually just have to start using the new syntax for everything, because "that's the one that always works", and because they didn't even know or think about that fact when they were creating their metrics. In any case, all this will require a whole lot of new explanations for every new Prometheus user, and the same is true for some of the other OTel features.
There's other things that worry me about UTF-8 support and OTel support in general, but the selector syntax and what this will ultimately do to Prometheus users down the road is the main one. Any opinions on how we should advertise this to the regular (non-OTel) Prometheus user going forward?
Cheers,
Julius
Bryan Boreham
May 24, 2024, 5:52:25 AMMay 24
to Julius Volz, Prometheus Developers
I think we should allow dot (.) in names without quoting.
It gives up a possible operator, but makes the syntax nicer for the vast majority of people who need a change.
--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YowuTe6tLp1N0jEQc5zhAQE0RDpdME4tP4m-uXEmRKqwPA%40mail.gmail.com.
Fabian Stäber
May 24, 2024, 7:31:44 AMMay 24
to Bryan Boreham, Julius Volz, Prometheus Developers
What Bryan said. OpenTelemetry's semantic conventions don't use all of UTF-8. They are restricted to the same character set as Prometheus plus the dot.
If we simply allow the dot, we cover all of OTel's semantic conventions for metric names and label names.
I don't care if custom metrics that use arbitrary UTF-8 characters require a weird escaping syntax.
The goal should be to make OpenTelemetry's semantic conventions work nicely with Prometheus.
Fabian
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/C96DD934-9A56-4A1E-B1CA-E45AAD1B1E41%40gmail.com.
Julius Volz
May 24, 2024, 9:44:17 AMMay 24
to Fabian Stäber, Bryan Boreham, Prometheus Developers
Yeah, that's true, that would help a lot already. Maybe it's time to open up the dot after all. I know we've kept it reserved for "reasons" (first as a potential "job" label separator, then later for other potential future ideas like ".sum" / ".count" for native histograms), but maybe it's worth finally using it for this. I know Björn has the strongest opinions on the dot :)
George Robinson
May 27, 2024, 4:07:00 AMMay 27
to Prometheus Developers
Hi! I worked with Björn and Josh to add support for UTF-8 in Alertmanager, and I thought I would share my opinion on this matter as we had to solve similar problems there.
I'm with Bryan and Fabian on the dot operator. If dot is all that's needed to support the OTel character set for metric names in Prometheus, and we don't expect further relaxations of this character set, then I think we should add it to Prometheus. We can then defer the problem of supporting UTF-8 in metrics names and how it would work with the existing PromQL grammar.
I also wouldn't worry about future use of dot as a syntactic sugar for label matchers or native histograms. I personally don't think it adds much to PromQL over the more verbose foo{job="bar"} or histogram_sum(sum(request_duration_seconds)), and I would argue it makes PromQL more difficult to learn.
On Björn's suggestion of __name__, I don't share Julius' concerns. In fact, I think it's quite nice because:
1. You can and sometimes have to use __name__ in Prometheus if the metric name conflicts with a reserved word. It is also a documented feature in Prometheus and extending it for metric names with UTF-8 seems very sensible to me.
2. In TSDB, the metric name is just a label anyway, is it not? If anything, once you understand how Prometheus stores metrics, querying a metric using __name__ feels more natural than the current PromQL grammar where it is separate from all other label matchers.
However, I'm not convinced that we should adopt the short {="metric_name"} or optional = as in {"metric_name"}. In this case I think consistency and uniformity of the grammar is more important than verbosity.
However, this is all irrelevant if we choose to accept dot anyway :)
George
I'm with Bryan and Fabian on the dot operator. If dot is all that's needed to support the OTel character set for metric names in Prometheus, and we don't expect further relaxations of this character set, then I think we should add it to Prometheus. We can then defer the problem of supporting UTF-8 in metrics names and how it would work with the existing PromQL grammar.
I also wouldn't worry about future use of dot as a syntactic sugar for label matchers or native histograms. I personally don't think it adds much to PromQL over the more verbose foo{job="bar"} or histogram_sum(sum(request_duration_seconds)), and I would argue it makes PromQL more difficult to learn.
On Björn's suggestion of __name__, I don't share Julius' concerns. In fact, I think it's quite nice because:
1. You can and sometimes have to use __name__ in Prometheus if the metric name conflicts with a reserved word. It is also a documented feature in Prometheus and extending it for metric names with UTF-8 seems very sensible to me.
2. In TSDB, the metric name is just a label anyway, is it not? If anything, once you understand how Prometheus stores metrics, querying a metric using __name__ feels more natural than the current PromQL grammar where it is separate from all other label matchers.
However, I'm not convinced that we should adopt the short {="metric_name"} or optional = as in {"metric_name"}. In this case I think consistency and uniformity of the grammar is more important than verbosity.
However, this is all irrelevant if we choose to accept dot anyway :)
George
Julius Volz
May 27, 2024, 7:45:44 AMMay 27
to George Robinson, Prometheus Developers, Bjoern Rabenstein
On Mon, May 27, 2024 at 10:07 AM 'George Robinson' via Prometheus Developers <prometheus...@googlegroups.com> wrote:
Hi! I worked with Björn and Josh to add support for UTF-8 in Alertmanager, and I thought I would share my opinion on this matter as we had to solve similar problems there.
I'm with Bryan and Fabian on the dot operator. If dot is all that's needed to support the OTel character set for metric names in Prometheus, and we don't expect further relaxations of this character set, then I think we should add it to Prometheus. We can then defer the problem of supporting UTF-8 in metrics names and how it would work with the existing PromQL grammar.
I also wouldn't worry about future use of dot as a syntactic sugar for label matchers or native histograms. I personally don't think it adds much to PromQL over the more verbose foo{job="bar"} or histogram_sum(sum(request_duration_seconds)), and I would argue it makes PromQL more difficult to learn.
From what I've heard so far, I would tend to agree with all that. However, I'd like to make sure that Björn also weighs in on the dot character question, since he was the main person raising concerns about it in the past. CC-ing him here :)
On Björn's suggestion of __name__, I don't share Julius' concerns. In fact, I think it's quite nice because:
1. You can and sometimes have to use __name__ in Prometheus if the metric name conflicts with a reserved word. It is also a documented feature in Prometheus and extending it for metric names with UTF-8 seems very sensible to me.
While this is technically true, I've never seen such a reserved-word collision in practice, and the problem would only occur for a few reserved words, like "on"/"ignoring", but e.g. not for just "rate" or "by" or "sum" etc. when they are grammatically used as a metric name. Other than that, you only ever need to know about "__name__" if your metrics are structured weirdly and you need to do regexes against them or if you're doing admin-level queries.
2. In TSDB, the metric name is just a label anyway, is it not? If anything, once you understand how Prometheus stores metrics, querying a metric using __name__ feels more natural than the current PromQL grammar where it is separate from all other label matchers.
Yeah, I think it's good for people to know that it's just a special label, but you should almost never need to access "__name__" unless your metrics are structured badly (encoding things in the metric name that should really be in a label) or if you are doing admin-level Prometheus stuff (e.g. figuring out which dimensional groupings produce how many time series in your Prometheus server). So I think requiring power users to use "__name__" for those exceptional queries is acceptable, but requiring it for just every-day normal user queries would make the query language less approachable.
However, I'm not convinced that we should adopt the short {="metric_name"} or optional = as in {"metric_name"}. In this case I think consistency and uniformity of the grammar is more important than verbosity.
Yeah, I'm split on that. On one hand, I don't want people to have to type the internal "__name__" label, on the other hand I like things to be explicit as well. Maybe one could have a special string syntax (e.g. prefixed somehow like in Python) for metric names that doesn't collide with our existing strings, so you could just write something like:
m"some.metric.name.with.dots"{foo="bar"}
I'm not sure if I like that much more though 😅
However, this is all irrelevant if we choose to accept dot anyway :)
As long as people stick to OTel conventions at least, yeah :)
Cheers,
Julius
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a402cc06-651d-4a32-950b-b58fa33d6775n%40googlegroups.com.
Bjoern Rabenstein
May 28, 2024, 6:17:10 PMMay 28
to Julius Volz, George Robinson, Prometheus Developers
I'm trying to keep things short, as all of this had been discussed
at length before.
WRT "how to explain UTF-8 support to users": I actually don't think
this is a huge problem. I would frame it "this is like file
names". You can use blanks and slashes in Unix file names, and if you
do, it requires weird quoting or escaping, but that's not a huge
problem in practice. People just don't use them if they care. And if
they have to interact with other file sources, where blanks are
common, they cope. And yes, that means that names from OTel semantic
conventions will always be considered weird, but that's a problem of
OTel, not all the other languages where a dot has a special
meaning. Segue to the next paragraph...
WRT the dot in OTel semantic conventions: Personally, I'm more
convinced than ever that it was a grave mistake to use dots in the
semantic conventions. I understand the history thereof, but the moment
that OTel self-declared as the overarching standard for all kind of
telemetry, they should have realized that using a character that has a
special meaning or is even an operator in sooooo many languages is a
really really bad idea. This is not just PromQL specific. Originally,
I thought it's infeasible to change the semantic conventions at this
point, but by now, that's exactly what I think OTel should do. If the
dot were an actual operator in OTel (let's say a separator of actual
1st class namespaces) rather than just a convention within a
technically opaque string, I could see some merit. But as it is not,
it's just annoying and has no benefits whatsoever.
Despite having said all of that, I don't realistically expect that
OTel is going to change the semantic conventions. So next question is
how to deal with it. There are many reasons why it's a bad idea to
allow the dot in Prometheus metric names, most of them weren't
mentioned in this thread. I won't enumerate them all again. We can do
that if we really want to open that can of worms again. Segue to the
next paragraph...
In all the discussions we had before, my impression was that the
consensus (in the spirit of RFC 7282) was to not add the dot to the
characters that don't require quoting. As the saying goes, in OSS, a
"no" is temporary and a "yes" is forever. So we can re-open this
debate as often as anyone wishes. If the result is different at some
point in the future, so be it. It's unlikely that I will change my
mind (in fact, as alluded to above, I'm more convinced than ever that
Prometheus should resist the urge). But that doesn't necessarily
prevent an RFC-7282-style consensus. (Or we could also just have a
vote, like in the old days, although that should be a last resort.)
Despite the opinions expressed so far, I would doubt that I'm the only
one who will be opposed.
Julius has previously described quite nicely how OTel conventions and
practices creep into the Prometheus ecosystem, undermining original
properties of Prometheus as "simple, light-weight, and
opinionated". The whole quoting syntax that opened this thread is for
me a way of allowing what OTel needs but also of containing the damage
and keep things in spirit for normal Prometheus users. Maybe another
thing to include when explaining the syntax to normal Prometheus
users.
--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in
at length before.
WRT "how to explain UTF-8 support to users": I actually don't think
this is a huge problem. I would frame it "this is like file
names". You can use blanks and slashes in Unix file names, and if you
do, it requires weird quoting or escaping, but that's not a huge
problem in practice. People just don't use them if they care. And if
they have to interact with other file sources, where blanks are
common, they cope. And yes, that means that names from OTel semantic
conventions will always be considered weird, but that's a problem of
OTel, not all the other languages where a dot has a special
meaning. Segue to the next paragraph...
WRT the dot in OTel semantic conventions: Personally, I'm more
convinced than ever that it was a grave mistake to use dots in the
semantic conventions. I understand the history thereof, but the moment
that OTel self-declared as the overarching standard for all kind of
telemetry, they should have realized that using a character that has a
special meaning or is even an operator in sooooo many languages is a
really really bad idea. This is not just PromQL specific. Originally,
I thought it's infeasible to change the semantic conventions at this
point, but by now, that's exactly what I think OTel should do. If the
dot were an actual operator in OTel (let's say a separator of actual
1st class namespaces) rather than just a convention within a
technically opaque string, I could see some merit. But as it is not,
it's just annoying and has no benefits whatsoever.
Despite having said all of that, I don't realistically expect that
OTel is going to change the semantic conventions. So next question is
how to deal with it. There are many reasons why it's a bad idea to
allow the dot in Prometheus metric names, most of them weren't
mentioned in this thread. I won't enumerate them all again. We can do
that if we really want to open that can of worms again. Segue to the
next paragraph...
In all the discussions we had before, my impression was that the
consensus (in the spirit of RFC 7282) was to not add the dot to the
characters that don't require quoting. As the saying goes, in OSS, a
"no" is temporary and a "yes" is forever. So we can re-open this
debate as often as anyone wishes. If the result is different at some
point in the future, so be it. It's unlikely that I will change my
mind (in fact, as alluded to above, I'm more convinced than ever that
Prometheus should resist the urge). But that doesn't necessarily
prevent an RFC-7282-style consensus. (Or we could also just have a
vote, like in the old days, although that should be a last resort.)
Despite the opinions expressed so far, I would doubt that I'm the only
one who will be opposed.
Julius has previously described quite nicely how OTel conventions and
practices creep into the Prometheus ecosystem, undermining original
properties of Prometheus as "simple, light-weight, and
opinionated". The whole quoting syntax that opened this thread is for
me a way of allowing what OTel needs but also of containing the damage
and keep things in spirit for normal Prometheus users. Maybe another
thing to include when explaining the syntax to normal Prometheus
users.
--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in
George Robinson
May 29, 2024, 9:26:27 AMMay 29
to Prometheus Developers
Thanks Björn for writing this up, and also writing up Collected reasons why Prometheus doesn't allow dot as a regular character in metric and label names. I think it adds a huge amount of value for people looking to participate in discussion! Having read through it all I withdraw my original support for just adding dot to metric names. There are a lot of considerations I did not know about that I agree with thanks to your document.
Julius Volz
May 29, 2024, 11:41:20 AMMay 29
to George Robinson, Prometheus Developers
Hah, I knew it would be a good idea to check with Björn :D Thanks Björn, that's a great write-up!
Yes, that also convinced me about not allowing the dot as a normal character for now. Lots of good arguments, but number 3 actually resonates the most with me - why allow two different separator characters if they have no semantic difference (no true namespacing).
--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d14d59cf-204b-4215-afb8-3b5adee96be4n%40googlegroups.com.
Arthur Silva Sens
Jun 5, 2024, 11:31:18 AMJun 5
to Julius Volz, George Robinson, Prometheus Developers
The results of Otel-Prometheus interoperability are out, which is probably relevant to the discussion here (specially the questions regarding UTF-8)
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CA%2BT6YoxypJRtxY%2B_rCspTEjvZi%3D2LD0Gn75f_vLKNfM-ReMPSQ%40mail.gmail.com.
Fabian Stäber
Jun 5, 2024, 12:07:48 PMJun 5
to Prometheus Developers
Thanks a lot for sharing!
So, is the prefered solution to keep things as they are, i.e. keep replacing dots with underscores?
> why allow two different separator characters if they have no semantic difference (no true namespacing).
This argument seems to resonate with the Prometheus team. If this is the main concern, we don't solve it by allowing dots in quotes. We solve this by replacing dots with underscores.
From the survey it looks like most users prefer the current naming scheme as well:
Shall we just drop the idea of adding UTF-8 support?
Fabian
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAJqZosxwoWTqgLfEmhEyMjvidW6-hFxJzzz3ev3A%3D4ZnV_deDQ%40mail.gmail.com.
Owen Williams
Jun 5, 2024, 12:30:37 PMJun 5
to Prometheus Developers
I would love to understand how to come to a decision as a project in a way that will preclude regular second-guessing. We went through the proposal process (https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md) so I though the question of both "if" and "how" were already closed.
Bjoern Rabenstein
Jun 5, 2024, 12:55:46 PMJun 5
to Fabian Stäber, Prometheus Developers
On 05.06.24 18:07, 'Fabian Stäber' via Prometheus Developers wrote:
>
> So, is the prefered solution to keep things as they are, i.e. keep
> replacing dots with underscores?
I don't think the purpose of the survey was to find a "preferred
>
> So, is the prefered solution to keep things as they are, i.e. keep
> replacing dots with underscores?
solution". First of all, this is a technical decision, not a
democratic one. And even if it were, an online survey is inherently
biased.
The idea behind the survey was (I hope) to get a broad idea what
people find surprising or annoying, what they expect, what they like,
... and then we can use those inputs in a responsible fashion to
inform decisions.
> > why allow two different separator characters if they have no
> > semantic difference (no true namespacing).
>
> This argument seems to resonate with the Prometheus team. If this is the
> main concern, we don't solve it by allowing dots in quotes. We solve this
> by replacing dots with underscores.
into name collisions, and it is hard to match a name from one side of
the conversion wall to the corresponding name on the other side.
The previous discussion lead to the conclusion that we want allow all
of UTF-8, because OTel does, but that everything that is not a
valid conventional Prometheus name will require quoting.
We kept open the option of later allowing more characters in the
unquoted names, after we have seen how the quoting goes.
> >From the survey it looks like most users prefer the current naming scheme
> as well:
>
> [image: screenshot_2024-06-05_18:04:05_908234003.png]
> [image: screenshot_2024-06-05_18:04:14_304430186.png]
schemas without providing any context. This can only give us some idea
about people's gut feeling, but not much more.
> Shall we just drop the idea of adding UTF-8 support?
of UTF-8 in names. If somebody names a metric in Chinese or Cyrillic,
we cannot convert it to "______________". That's the whole point. We
need UTF-8 support _anyway_. So let's do it and see how it goes before
running the umptieth reiteration of "can we just allow dots in metric
names".
Bartłomiej Płotka
Jun 10, 2024, 6:07:43 AMJun 10
to Bjoern Rabenstein, Fabian Stäber, Prometheus Developers
First of all, while it can be painful, I think it's ok to double check the already approved proposal for the big blockers. I don't think I see any big blocker, given this discussion.
Just 2c, I wonder if it would be helpful to split metric/label naming discussion into:
A) What protocol, storage and PromQL supports (more flexibility for future)
B) What makes sense to use for everyone (recommendation or mandated for some systems)
Essentially we can (and probably should) add versatile escaping mechanisms. Does it mean people should call their metrics "❤️🔥123🧠_total"? I don't think it's useful to accept those. It might be useful to accept (escaped) dots though given Otel users want (for example) without translation, use the same matchers for traces, logs, profiles and metrics and they cannot really change that decision.
Essentially we can (and probably should) add versatile escaping mechanisms. Does it mean people should call their metrics "❤️🔥123🧠_total"? I don't think it's useful to accept those. It might be useful to accept (escaped) dots though given Otel users want (for example) without translation, use the same matchers for traces, logs, profiles and metrics and they cannot really change that decision.
We could keep (recommend or ensure) the conversion schema for Otel metrics (move to _) AND move with generic UTF-8 support. Both could be done in parallel, as long as it's officially agreed and documented (:
Kind Regards,
Bartek Płotka (@bwplotka)
--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ZmCYjf9yXTLMqNbW%40mail.rabenste.in.
Owen Williams
Jun 10, 2024, 10:53:56 AMJun 10
to Bartłomiej Płotka, Bjoern Rabenstein, Fabian Stäber, Prometheus Developers
the committed code in /common includes support for 3 escaping formats:
* all legacy-invalid characters to underscores (cannot round-trip)
* dots are escaped to _dot_ and underscores are converted to double underscores, all other legacy-invalid characters become single underscores. (partial round trip)
* a new U__ encoding for unicode code points so names can be fully round-tripped
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-developers/ftnfizjXOmk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAMssQwa87r9%2B3tAKRSwG6L8YAAu6QQwk4mdt7E%2BGgYR7%3DvOjfQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages