Federation: blacklist vs. whitelist

[Jordan Sokolic]

Hello,

I'm trying to find the best way to use federation to match a metric blacklist. Currently we are using a single query string as one long array element to accomplish this, i.e.:

params:

  'match[]':

    - >-

      {__name__=~".+",

       __name__!~"prefix_1.+",

       __name__!~"prefix_2.+"}

Breaking up this string into multiple elements does not work, as each element is parsed into a separate match[] parameter in the HTTP request. The response appears as if each array element was queried individually and combined together after query execution, so one element only excludes one prefix, the next element only excludes the next prefix, etc. with the result that all matching timeseries are eventually returned.

What is the recommend way to provide a query blacklist when federating Prometheus servers?

[Brian Candler]

{__name__!~"(prefix_1|prefix_2).+"}

[Jordan Sokolic]

Thank you Brian.

That's definitely a more compact format, but the downside to that approach is that it cannot be split up into multiple lines for readability since it breaks YAML folding due to conversion of newlines to spaces. If using more than a few matching blacklist expressions, tracking changes in a single line becomes cumbersome.

[Brian Candler]

Maybe you could store the matches in a readable file (e.g. YAML or just a plain text file), then write a simple tool to read this in and convert it to the long regexp.

[Jordan Sokolic]

Yes, we're using Ansible already so this is definitely doable using a Jinja template -- thanks for the suggestion. I just wanted to make sure I wasn't missing something obvious re: the federation syntax.