Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible.

[Chris Burke]

Can someone please help explain the following Alertmanager warning.
I know it's trying to tell me that my value needs to be double-quoted, but it already is, so I do not understand what it is complaining about.
My config: applicationid=~"^SNSVC\d{7}$"

Thanks for any help

ts=2024-12-06T00:44:13.964Z caller=parse.go:176 level=warn msg="Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible. Alertmanager has instead parsed the input using the classic matchers parser as a fallback. To make this input compatible with the UTF-8 matchers parser please make sure all regular expressions and values are double-quoted. If you are still seeing this message please open an issue." input="applicationid=~\"^SNSVC\\d{7}$\"" origin=config err="15:29: \"^SNSVC\\d{7}$\": invalid input" suggestion="applicationid=~\"^SNSVC\\\\d{7}$\""

[Brian Candler]

A bit more context would be helpful. Did you write:

matchers:
  - applicationid=~"^SNSVC\d{7}$"

or something else?

Testing with alertmanager 0.27, I think the problem is around handling of the backslash. The following is accepted by amtool check-config:

matchers:
  - applicationid=~"^SNSVC\\d{7}$"

but I've not checked if it matches as expected - you don't want to match a literal backslash and d!

This is also what the "suggestion" was telling you from the error message, but then it's also having to escape things for logfmt logging, which means double-quotes are preceded by backslash, and backslashes are doubled. When it says:

suggestion="applicationid=~\"^SNSVC\\\\d{7}$\""

I think what it's actually suggesting is:

applicationid=~"^SNSVC\\d{7}$"

Unfortunately, YAML also has its own rules for backslashes. Because of this complexity, I avoid backslashes in regexps where possible, for example using [.] instead of \. to match a literal dot.  In your case, you could write:

matchers:
  - applicationid=~"^SNSVC[0-9]{7}$"

and there would be no ambiguity.

[Brian Candler]

I checked at https://prometheus.io/docs/alerting/latest/configuration/#matcher

AFAICS, it doesn't explicitly mention the behaviour of backslashes in UTF-8 matches. There is a specific note about the fallback behaviour of classic matchers though:

"However, literal line feed characters are tolerated, as are single \ characters not followed by \, n, or ". They act as a literal backslash in that case."

Assuming that UTF-8 matchers strictly require literal backslashes to be doubled (\\), this would explain why your expression was accepted by classic matchers but not UTF-8 matchers. The warning message "To make this input compatible with the UTF-8 matchers parser please make sure all regular expressions and values are double-quoted" could perhaps be improved to mention this possibility.

[George Robinson]

I believe Brian is correct. In the new parser, backslash is an escape character, so to write a literal backslash (for \d) it would need to be escaped as \\. You'll find the same behavior in languages like Go where "^SNSVC\d{7}$" is not a valid string (unknown escape sequence) and must be written as "^SNSVC\\d{7}$".

The suggestion in the log line gave you the correct matcher, but the wrong explanation.

We can add this to the documentation to make it clear.

George

[George Robinson]

I just checked the code, and the reason `"\d"` needs to be escaped is we call strconv.Unquote to unquote the right hand side of the expression. In other words, to turn `"\d"` into just `\d`.

That means while the lexer accepts `"\d"`,  it is not a valid double quoted string that follows the Go escaping rules, and so it is rejected. However `"\\d"` is because when it is unquoted the second backslash is treated as an escape character and we end up with `\d`.

The tl;dr here is that when using double quotes you need to follow Go's rules in double quoted strings.

As Brian mentioned, this is separate from YAML escape rules.

We can update the docs to make this clear.

[George Robinson]

Just to wrap this up, one of the reasons for this is to be consistent.

If you have a matcher like applicationid=~"^SNSVC\d{7}$" and you ask Alertmanager to write it back to a file, it will do it as applicationid=~"^SNSVC\\d{7}$" (note the double escape). This is because again it follows the Go escaping rules for strings when encoding it.

By enforcing double quotes in the new parser, we now have consistency between what you the user can write and what you see output from Alertmanager. The behavior though is unchanged.

I hope this makes sense

George

[George Robinson]

Here is the PR that updates the docs https://github.com/prometheus/alertmanager/pull/4157

[Chris Burke]

Thanks for this, really appreciate it.

Was a bit thrown by the warning message focusing on the double quotes even though the value had them.

This clarifies thing nicely.

Thanks again!