Relabel action suggestions

[Tristan Colgate]

Hi,

  I'd like to find a way to overcome the issue with duplicate scapes

when using the prometheus.io/port annotation in kubernetes.

Essentially, because regexp can't do backtracking we can't do any kind of

"discard this target if the value of label2 does not match the value of label1"

What currently happens is that we get repeated discovered targets all with the same address. These /may/ get deduped, depending on labeling, but does not in all cases (for instance, we use the container name as the primary "job" label, so this changes per target).

I would like to suggest an alternation to keep and drop that would match

against target_label's value if specified.

This would be enough to prevent the duplicate scrapes mentioned about, and should be pretty easy to implement. I'll do this if people don't have any major objections.

As a complete long shot, I also have one other suggestion.

We could introduce a split action which would take each source label, and split by regex (possibly, or similar), and produce a secries of labels with a suffix (e.g. _0, _1). In addition, a "sourcesRegex" could be introduced to all rules that allowed source label selection based on regex rather than explicit configuration. These two rules together, combined with the above keep/drop modification would allow something like "prometheus.io/ports":"1234,5678" to select multiple ports.

I think all three suggestions are reasonably orthogonal to the current feature set and would add value.

Again, if people are amenable, I am happy to do the work. I just don't want to start if there's no chance of it getting accepted.

Thoughts? 

[Brian Brazil]

On 8 March 2018 at 11:44, Tristan Colgate <tcol...@gmail.com> wrote:

Hi,

  I'd like to find a way to overcome the issue with duplicate scapes

when using the prometheus.io/port annotation in kubernetes.

Essentially, because regexp can't do backtracking we can't do any kind of

"discard this target if the value of label2 does not match the value of label1"

What currently happens is that we get repeated discovered targets all with the same address. These /may/ get deduped, depending on labeling, but does not in all cases (for instance, we use the container name as the primary "job" label, so this changes per target).

I would like to suggest an alternation to keep and drop that would match

against target_label's value if specified.

This would be enough to prevent the duplicate scrapes mentioned about, and should be pretty easy to implement. I'll do this if people don't have any major objections.

That seems a bit specific as an action. Why would it only apply to drop?

 

As a complete long shot, I also have one other suggestion.

We could introduce a split action which would take each source label, and split by regex (possibly, or similar), and produce a secries of labels with a suffix (e.g. _0, _1). In addition, a "sourcesRegex" could be introduced to all rules that allowed source label selection based on regex rather than explicit configuration. These two rules together, combined with the above keep/drop modification would allow something like "prometheus.io/ports":"1234,5678" to select multiple ports.

I think all three suggestions are reasonably orthogonal to the current feature set and would add value.

That sounds quite intricate, and hard to use. 

I think the core issue here is that users are taking what was meant to just be a demo, and not adapting it for their particular deployment. I've never been a major fan of this annotation, and would prefer to have a demo that just scraped everything for example.

As it stands you can work with this style of annotation, you just need as many scrape configs as the pod with the highest number of ports to scrape. I'm generally against adding features for things that are handled via existing methods, such as a small bit of duplication in your prometheus.yml. 

I'm wary of adding any additional actions as there's enough confusion over relabelling as-is without adding more ways to combine, particularly to only over one use case for which there are other known approaches.

The best solution would be if Kubernetes supported per-port annotations, as all of this is trying to effectively use per-pod annotations as per-port annotations.

Brian

 

Again, if people are amenable, I am happy to do the work. I just don't want to start if there's no chance of it getting accepted.

Thoughts? 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAPGZSGLsiKon7UFoGpwz-XBqsReR8wo-vwfVv61C_hVb%2Bnwq6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Tristan Colgate]

On Thu, 8 Mar 2018 at 11:57 Brian Brazil <brian....@robustperception.io> wrote:

I would like to suggest an alternation to keep and drop that would match

against target_label's value if specified.

This would be enough to prevent the duplicate scrapes mentioned about, and should be pretty easy to implement. I'll do this if people don't have any major objections.

That seems a bit specific as an action. Why would it only apply to drop?

Sorry, I worded that poorly. I was suggesting adding this as a feature to the existing keep and drop actions.

re: existing example config.

This is true to some extent. Testing for label equality though seems like a completely reasonable thing to be able to do. We could do it if Go's regexp did backtracking, but they don't (for good reasons).

re: split, sourcesRegex,  

That sounds quite intricate, and hard to use. 

 I tend to agree, I also don't think it would add that much. One exporter per-pod is by far the most common case. We do prefer to scrape all ports but that is not always feasible for lots of reasons

- Not everything is HTTP, you don't always know purely from the port spec

- Sadly, not everything is going to have /metrics.

-  Coffee just makes the sound of developers complaining at me about "all these requests to /metric" more annoying, and I'm not likely to stop drinking coffee.

 

The best solution would be if Kubernetes supported per-port annotations, as all of this is trying to effectively use per-pod annotations as per-port annotations.

 

I think you mean per-container annotations, and I am inclined to agree, but  that ship may have sailed.

[Brian Brazil]

On 8 March 2018 at 12:16, Tristan Colgate <tcol...@gmail.com> wrote:

On Thu, 8 Mar 2018 at 11:57 Brian Brazil <brian.brazil@robustperception.io> wrote:

I would like to suggest an alternation to keep and drop that would match

against target_label's value if specified.

This would be enough to prevent the duplicate scrapes mentioned about, and should be pretty easy to implement. I'll do this if people don't have any major objections.

That seems a bit specific as an action. Why would it only apply to drop?

Sorry, I worded that poorly. I was suggesting adding this as a feature to the existing keep and drop actions.

It still seems a bit limited, it really sounds like a new action so that it could be composed.

 

re: existing example config.

This is true to some extent. Testing for label equality though seems like a completely reasonable thing to be able to do. We could do it if Go's regexp did backtracking, but they don't (for good reasons).

I have previously investigated if there was a non-re2 engine for Go, with no success.

 

re: split, sourcesRegex,  

That sounds quite intricate, and hard to use. 

 I tend to agree, I also don't think it would add that much. One exporter per-pod is by far the most common case. We do prefer to scrape all ports but that is not always feasible for lots of reasons

- Not everything is HTTP, you don't always know purely from the port spec

- Sadly, not everything is going to have /metrics.

-  Coffee just makes the sound of developers complaining at me about "all these requests to /metric" more annoying, and I'm not likely to stop drinking coffee.

One option is to work from the port names, though as with everything here it's not perfect.

 

 

The best solution would be if Kubernetes supported per-port annotations, as all of this is trying to effectively use per-pod annotations as per-port annotations.

 

I think you mean per-container annotations, and I am inclined to agree, but  that ship may have sailed.

Has anyone actually asked them if they'd consider it?

--

Brian Brazil

www.robustperception.io

[Tristan Colgate]

On Thu, 8 Mar 2018 at 11:57 Brian Brazil <brian....@robustperception.io> wrote:

I would like to suggest an alternation to keep and drop that would match

against target_label's value if specified.

This would be enough to prevent the duplicate scrapes mentioned about, and should be pretty easy to implement. I'll do this if people don't have any major objections.

That seems a bit specific as an action. Why would it only apply to drop?

Sorry, I worded that poorly. I was suggesting adding this as a feature to the existing keep and drop actions.

It still seems a bit limited, it really sounds like a new action so that it could be composed.

 

I'm not sure if being the same action makes it any less composable. It gives keep and drop two separate modes of operation. If it were a separate action (for the sake of argument, compare), I think you'd actually want two actions, compare_keep and compare_drop, and that seems worse to me than extending the semantics of the existing actions.

 

One option is to work from the port names, though as with everything here it's not perfect.

yes, having something like a "-promscrape" suffix could be done, but this definitely makes it harder to work with third party manifests. Unless it got accepted as a norm by the larger kube community. 

The best solution would be if Kubernetes supported per-port annotations, as all of this is trying to effectively use per-pod annotations as per-port annotations.

 

I think you mean per-container annotations, and I am inclined to agree, but  that ship may have sailed.

Has anyone actually asked them if they'd consider it?

Not that I can find reference to. Though one objection I can foresee is that Containers are not actually objects in the kube sense of API objects, and it is API objects that get annotations and labels.  If Kubernetes decided Containers needed labels and annotations, that would then need to flow down to all the container run times.

Regardless, some label equality functionality in relabel rules seems reasonable. If you do not like repurposing target_label in keep/drop, what would be your preference?

[Tristan Colgate]

I hope that showing the code might help, I have raised https://github.com/prometheus/prometheus/pull/3929

[Brian Brazil]

On 8 March 2018 at 12:35, Tristan Colgate <tcol...@gmail.com> wrote:

On Thu, 8 Mar 2018 at 11:57 Brian Brazil <brian.brazil@robustperception.io> wrote:

I would like to suggest an alternation to keep and drop that would match

against target_label's value if specified.

This would be enough to prevent the duplicate scrapes mentioned about, and should be pretty easy to implement. I'll do this if people don't have any major objections.

That seems a bit specific as an action. Why would it only apply to drop?

Sorry, I worded that poorly. I was suggesting adding this as a feature to the existing keep and drop actions.

It still seems a bit limited, it really sounds like a new action so that it could be composed.

 

I'm not sure if being the same action makes it any less composable. It gives keep and drop two separate modes of operation. If it were a separate action (for the sake of argument, compare), I think you'd actually want two actions, compare_keep and compare_drop, and that seems worse to me than extending the semantics of the existing actions.

That precludes any further interactions with replace or the like.

 

 

One option is to work from the port names, though as with everything here it's not perfect.

yes, having something like a "-promscrape" suffix could be done, but this definitely makes it harder to work with third party manifests. Unless it got accepted as a norm by the larger kube community. 

The best solution would be if Kubernetes supported per-port annotations, as all of this is trying to effectively use per-pod annotations as per-port annotations.

 

I think you mean per-container annotations, and I am inclined to agree, but  that ship may have sailed.

Has anyone actually asked them if they'd consider it?

Not that I can find reference to. Though one objection I can foresee is that Containers are not actually objects in the kube sense of API objects, and it is API objects that get annotations and labels.  If Kubernetes decided Containers needed labels and annotations, that would then need to flow down to all the container run times.

Regardless, some label equality functionality in relabel rules seems reasonable. If you do not like repurposing target_label in keep/drop, what would be your preference?

One thing I had been pondering was a lookup action to let you lookup the value of the label called replacement, but that also concerns me complexity wise. I'd rather not make relabelling more complex for the sake of one niche use case for which there are other solutions.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 8 March 2018 at 14:06, Tristan Colgate <tcol...@gmail.com> wrote:

I hope that showing the code might help, I have raised https://github.com/prometheus/prometheus/pull/3929

Looking at your PR, I fail to see how this allows for multiple ports in one pod to be scraped with one scrape_config.

Brian

--

Brian Brazil

www.robustperception.io

[Tristan Colgate]

On Thu, 8 Mar 2018 at 14:10 Brian Brazil <brian....@robustperception.io> wrote:

That precludes any further interactions with replace or the like.

I think I'm missing something. The existing keep/droop have no existing interaction with replace.

 

One thing I had been pondering was a lookup action to let you lookup the value of the label called replacement, but that also concerns me complexity wise.

This sounds interesting, though I'd need to understand how that would be used.

 

I'd rather not make relabelling more complex for the sake of one niche use case for which there are other solutions.

 

I don't think any of the workarounds are practical. At least not in our environment.

- relying on all named/listed ports exposing metrics is not going to work (not everything is http)

- removing ports from configured list (they are optional), removes the very useful feature of using them to document a pods functionality, and precludes using named ports in services.

- naming ports explicitly to control metrics collection is going to be ugly and hard to add retrospectively, also impact service names.

- using multiple scrape configs will essentially degenerate into a scrape config per application (or a scrape config per application that doesn't fit a specific form, which still gets down to micromanaging scrape configs)

- asking kubernetes to add labels/annotations to containers is a very big conversation involving an awful lot of people so far as I know. 

Also, more generally, it seems to me like some ability to compare label values, beyond explicit comparison to static strings would be useful in relabeling. PromQL is sort of built on label matching, if it is useful there, why not in relabeling?

[Tristan Colgate]

On Thu, 8 Mar 2018 at 14:58 Brian Brazil <brian....@robustperception.io> wrote:

Looking at your PR, I fail to see how this allows for multiple ports in one pod to be scraped with one scrape_config.

I updated the example config. It allows for

- all ports on pods with no prometheus.io/port annotation to be scraped

- only the specifically request named port on pods with prometheus.io/port to be scraped.

My earlier sourcesRegex and split proposals were an attempt to support multiple ports. I agreed those were complicated. Supporting a scraping an arbitrary subset of ports would definitely require something more, though I don't have any other ideas what that might look like.

 

[Brian Brazil]

On 8 March 2018 at 14:58, Tristan Colgate <tcol...@gmail.com> wrote:

On Thu, 8 Mar 2018 at 14:10 Brian Brazil <brian.brazil@robustperception.io> wrote:

That precludes any further interactions with replace or the like.

I think I'm missing something. The existing keep/droop have no existing interaction with replace.

Exactly, they're orthogonal. With what you're proposing you can only drop or keep, not do anything else based on that information.

 

 

One thing I had been pondering was a lookup action to let you lookup the value of the label called replacement, but that also concerns me complexity wise.

This sounds interesting, though I'd need to understand how that would be used.

You'd use the port name to lookup a map of annotations.

 

 

I'd rather not make relabelling more complex for the sake of one niche use case for which there are other solutions.

 

I don't think any of the workarounds are practical. At least not in our environment.

- relying on all named/listed ports exposing metrics is not going to work (not everything is http)

- removing ports from configured list (they are optional), removes the very useful feature of using them to document a pods functionality, and precludes using named ports in services.

- naming ports explicitly to control metrics collection is going to be ugly and hard to add retrospectively, also impact service names.

I believe you can have ports with the same number but different names.

 

- using multiple scrape configs will essentially degenerate into a scrape config per application (or a scrape config per application that doesn't fit a specific form, which still gets down to micromanaging scrape configs)

That's not the case, if you can add a io.prometheus/port annotation you can add a io.prometheus/port2 annotation. Any solution in this space requires this ability.

 

- asking kubernetes to add labels/annotations to containers is a very big conversation involving an awful lot of people so far as I know. 

Also, more generally, it seems to me like some ability to compare label values, beyond explicit comparison to static strings would be useful in relabeling. PromQL is sort of built on label matching, if it is useful there, why not in relabeling?

 Relabelling is only intended to go so far, once the use case gets complex enough you are going to end up using config generation and/or file_sd.

--

Brian Brazil

www.robustperception.io

[Brian Brazil]

On 8 March 2018 at 15:02, Tristan Colgate <tcol...@gmail.com> wrote:

That's the problem users are asking about, not using names instead of numbers. 

--

Brian Brazil

www.robustperception.io

[Tristan Colgate]

That's the problem users are asking about, not using names instead of numbers. 

I think you are at least partially misunderstanding the problem.

- pod with 5 ports

- role: endpoints scrape

- I add the prometheus.io/port annotaion  to a pod

I get 5 scrapes of that port. The address is overwritten every time.

If I leave off the annotation, I get 5 scrapes, of the individual ports, some of which will fail (most likely all bar one)

In your example of 

- prometheus.io/portN

I will end up with Nx5 scrapes, for for each of my target ports, a bunch of which will be duplicates, because I still can't do any kind of comparison.

[Brian Brazil]

On 8 March 2018 at 15:20, Tristan Colgate <tcol...@gmail.com> wrote:

That's the problem users are asking about, not using names instead of numbers. 

I think you are at least partially misunderstanding the problem.

- pod with 5 ports

- role: endpoints scrape

- I add the prometheus.io/port annotaion  to a pod

I get 5 scrapes of that port. The address is overwritten every time.

If I leave off the annotation, I get 5 scrapes, of the individual ports, some of which will fail (most likely all bar one)

That's not right, the targets should be de-duplicated. Are you sure they're all ending up with the same target labels and scrape settings?

 

In your example of 

- prometheus.io/portN

I will end up with Nx5 scrapes, for for each of my target ports, a bunch of which will be duplicates, because I still can't do any kind of comparison.

--

Brian Brazil

www.robustperception.io

[Tristan Colgate]

I mentioned earlier that they end up with different variations in labels due to other relabeling actions. I can look at trying to reduce those, but in that past that has not worked (these are combination of containers with multiple ports, pods with multiple containers). I can try harder to make the labels collide perhaps.

Some kind of label comparison support in relabel'ing makes sense to me. I hoped my suggestion was a fairly simple and safe option. In lieu of backreferences in regex, this seems like an obvious alternative.

I can't say I understand the objections re: replace, you seem to be suggesting that this doesn't offer enough new functionality, and simultaneously suggesting that it is too much extra complexity.

If the lookup function you have mentioned can resolve similar issue, then show me some unit tests and I'll gladly implement it (though you have also pretty much ruled that out too as adding too much complexity).

I'm left in a situation where I can't add a new relabel rule (seemingly for any purpose), and can't adjust the ones that are there. I can only assume that I'm doing a bunch of stuff wrong, but that there are no examples of how I can get it right.

Thanks for your time.

[Brian Brazil]

On 8 March 2018 at 16:02, Tristan Colgate <tcol...@gmail.com> wrote:

I mentioned earlier that they end up with different variations in labels due to other relabeling actions. I can look at trying to reduce those, but in that past that has not worked (these are combination of containers with multiple ports, pods with multiple containers). I can try harder to make the labels collide perhaps.

Some kind of label comparison support in relabel'ing makes sense to me. I hoped my suggestion was a fairly simple and safe option. In lieu of backreferences in regex, this seems like an obvious alternative.

I can't say I understand the objections re: replace, you seem to be suggesting that this doesn't offer enough new functionality, and simultaneously suggesting that it is too much extra complexity.

I'm saying more that the design isn't clean, and even with a clean design this would still be extra complexity.

 

If the lookup function you have mentioned can resolve similar issue, then show me some unit tests and I'll gladly implement it (though you have also pretty much ruled that out too as adding too much complexity).

It's something I've thought about, I'm not saying that it's a good idea.

 

I'm left in a situation where I can't add a new relabel rule (seemingly for any purpose), and can't adjust the ones that are there. I can only assume that I'm doing a bunch of stuff wrong, but that there are no examples of how I can get it right.

It sounds like you might be pulling in more target labels than you should be, and some are varying on a per-port or per-pod basis. I'd check for an over-zealous labelmap.

https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas/ has general advice in this area.

Brian

 

Thanks for your time.

On Thu, 8 Mar 2018 at 15:30 Brian Brazil <brian.brazil@robustperception.io> wrote:

On 8 March 2018 at 15:20, Tristan Colgate <tcol...@gmail.com> wrote:

That's the problem users are asking about, not using names instead of numbers. 

I think you are at least partially misunderstanding the problem.

- pod with 5 ports

- role: endpoints scrape

- I add the prometheus.io/port annotaion  to a pod

I get 5 scrapes of that port. The address is overwritten every time.

If I leave off the annotation, I get 5 scrapes, of the individual ports, some of which will fail (most likely all bar one)

That's not right, the targets should be de-duplicated. Are you sure they're all ending up with the same target labels and scrape settings?

 

In your example of 

- prometheus.io/portN

I will end up with Nx5 scrapes, for for each of my target ports, a bunch of which will be duplicates, because I still can't do any kind of comparison.

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io

[Tristan Colgate]

In one case, the issue is while the __address__ port is renamed, the port in the instance label isn't. Consequently they don't match, which causes the duplication. There are other cases, this one I think I can anyway (my guess is that the original example config suffers from the same problem).

On Thu, 8 Mar 2018 at 17:03 Brian Brazil <brian....@robustperception.io> wrote:

On 8 March 2018 at 16:02, Tristan Colgate <tcol...@gmail.com> wrote:

I mentioned earlier that they end up with different variations in labels due to other relabeling actions. I can look at trying to reduce those, but in that past that has not worked (these are combination of containers with multiple ports, pods with multiple containers). I can try harder to make the labels collide perhaps.

Some kind of label comparison support in relabel'ing makes sense to me. I hoped my suggestion was a fairly simple and safe option. In lieu of backreferences in regex, this seems like an obvious alternative.

I can't say I understand the objections re: replace, you seem to be suggesting that this doesn't offer enough new functionality, and simultaneously suggesting that it is too much extra complexity.

I'm saying more that the design isn't clean, and even with a clean design this would still be extra complexity.

 

If the lookup function you have mentioned can resolve similar issue, then show me some unit tests and I'll gladly implement it (though you have also pretty much ruled that out too as adding too much complexity).

It's something I've thought about, I'm not saying that it's a good idea.

 

I'm left in a situation where I can't add a new relabel rule (seemingly for any purpose), and can't adjust the ones that are there. I can only assume that I'm doing a bunch of stuff wrong, but that there are no examples of how I can get it right.

It sounds like you might be pulling in more target labels than you should be, and some are varying on a per-port or per-pod basis. I'd check for an over-zealous labelmap.

https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas/ has general advice in this area.

Brian

 

Thanks for your time.

On Thu, 8 Mar 2018 at 15:30 Brian Brazil <brian....@robustperception.io> wrote:

On 8 March 2018 at 15:20, Tristan Colgate <tcol...@gmail.com> wrote:

That's the problem users are asking about, not using names instead of numbers. 

I think you are at least partially misunderstanding the problem.

- pod with 5 ports

- role: endpoints scrape

- I add the prometheus.io/port annotaion  to a pod

I get 5 scrapes of that port. The address is overwritten every time.

If I leave off the annotation, I get 5 scrapes, of the individual ports, some of which will fail (most likely all bar one)

That's not right, the targets should be de-duplicated. Are you sure they're all ending up with the same target labels and scrape settings?

 

In your example of 

- prometheus.io/portN

I will end up with Nx5 scrapes, for for each of my target ports, a bunch of which will be duplicates, because I still can't do any kind of comparison.

--

Brian Brazil

www.robustperception.io

--

Brian Brazil

www.robustperception.io