Replacing a specific character in a label value

[Yuriy Bash]

Hi,

I have a prometheus metric that contains a label with a value that contains a character I’d like to replace with another. Specifically, the label can look like this (omitting irrelevant labels):

some_metric{destination_job=”us-east-1/foo-bar/prod/some_job}”

or like this:

some_metric{destination_job=”us-east-1/foo-bar-baz/prod/some_job}”}

Note the variable number of hyphens in the second segment (foo-bar vs foo-bar-baz).

I’d like to replace this character with an underscore, so that:

some_metric{destination_job="us-east-1/foo-bar/prod/some_job"} 

becomes: 

some_metric{destination_job="us-east-1/foo_bar/prod/some_job}"

and 

some_metric{destination_job=”us-east-1/foo-bar-baz/prod/some_job}”}

becomes:

some_metric{destination_job=”us-east-1/foo_bar_baz/prod/some_job}”}

I attempted to use label_replace, but the problem is that we’re using a variable number of hyphens, so I couldn’t use something like (.+)-(.+) to capture the segment and then use $1_$2 (or whatever numbers the capture groups end up being), since some cases I’d need to do $1_$2_$3, etc.

Is there a way to do this? I’m attempting to do this at query time, but also happy to use metric_relabel_configs - but that appears to suffer from the same limitation.

I might be missing something simple, in which case I apologize in advance. As a last resort, we can update our application code to publish a separate (or modified label) value, but I’d prefer to avoid that if possible.

Thanks!

- Yuriy

[Brian Candler]

Is the number of characters to replace bounded, e.g. a maximum of 4? Then you can just use label_replace 4 times (or repeat the whole rewriting rule 4 times).

Otherwise, I think getting the labels right at the exporter is the right answer.

[Yuriy Bash]

Unfortunately, it's not bounded. I think what we'll likely do is update the application to add another label with the correctly-formatted values before publishing. Seems a bit silly, but doesn't seem like there's much an alternative.