remote_write and federation

[Thibaut Sarion]

Hello everyone !
I've got a question regarding remote_write and federation.
I have two levels of prometheus : level 1 with classic prometheus instances and level 2 with prometheus federating some metrics from my level 1.
Both levels are remote_writing to my backend, but I need to find a way not to write the same metrics twice in my backend.
From level 1, I remote_write every metrics to the backend, while from the level 2, I need to only remote_write the new ones (calculated on level 2) so I was planning to drop metrics on level 2 based on the federate job name.

remote_write:
- url: "mybackend"
write_relabel_configs: # attempt to drop every federated metric
- source_labels: ['job_name']
regex: "federate"
action: drop

In most cases, this would work, but if a user writes this rule on level 2

rules:
- record: new_metric_on_level_2
expr: sum without(random_label) (counter_from_level_1)

new_metric_on_level_2 will inherit the job=federate,  thus being dropped at remote_write but it shouldn't as it's a new metric on level 2 and needs to be remote_write to my backend.
Note : I cannot modify the rule for adding without(random_label,job) even if it solves the problem (because I don't own the rules)
Do you have any suggestion on how I could solve this use case ? Thank you so much for reading

[Laurent C.]

Hi Thibault,

This puzzles me since I read your message.

In level 2, can't you immediately remove the label federate in job definition? But the problem is still at level 2. So all recording rules at level 2 should add a dedicated label "iwanttoremotewritethis". This does not seem to be a feature. If you don't manage level 1 and 2, can't you use some post-processing automation to inject labels "iamlevel1" / "iamlevel2" in all recording rules?

Then if you have only label "iamlevel2", you know that you can remote_write this metric.

--

Laurent CREPET.