Ensuring Persistent Labels in Ingested Metrics

[mohan garden]

Hi Everyone,

I need to handle a scenario where I add a label to ingested metrics, but only if the label does not already exist in the collected data. If the label is present in a metric, it should remain unchanged.

For example, given a set of metrics:

custom_metrics_from_server{ source="server1.example.com",status="run" } 3213  
custom_metrics_from_server{ source="server1.example.com",status="pend" } 3215

When additional metrics are collected from a node exporter, some will already have a "source=" label, while others will not:

custom_metrics_from_server{ source="server1.example.com",status="run" } 3213  
custom_metrics_from_server{ source="server1.example.com",status="pend" } 3215  
node_cooling_device_max_state{name="7",type="Processor"} 0

To ensure that all metrics contain the "source=" label without overriding existing ones, I figured the following approach using honor_labels and relabel_configs:

honor_labels: true  
relabel_configs:  
  - source_labels: [ _address_ ]  
    regex: "(.*):(.*)"  
    target_label: "source"  
    replacement: '${1}'

Is this the best way to handle this, or is there a more optimal approach?
Please advice.

Thanks!

[mohan garden]

here's some additional context - 

the following metrics are exported by the node exporter via text file:

custom_metrics_from_server{ source="server1.example.com",status="run" } 3213  
custom_metrics_from_server{ source="server1.example.com",status="pend" } 3215

hence they have source= label, whereas rest of the metrics are collected from the system by node exporter. i.e.

node_cooling_device_max_state{name="7",type="Processor"} 0

[Brian Candler]

It's better to avoid honor_labels unless you really need it. It allows the exporter to do bad things like overriding the "job" and "instance" labels.

Better to use metric_relabel_configs to relabel, at which point you have the combined set of labels from service discovery and the scrape. A missing label is the same as a label whose value is the empty string. So you can do (untested):

metric_relabel_configs:  
  - source_labels: [ source, __address__ ]
    regex: ";(.*)(:.*)?"

    target_label: source
    replacement: "${1}"

This works as long as the source label's value cannot start with a semicolon - if it can, define a different "separator" character. If you don't like that, another approach is to set a temporary label as a flag:

metric_relabel_configs:  
  - source_labels: [ source ]
    regex: ""
    target_label: __tmp_source_empty
    replacement: Y

  - source_labels: [ __tmp_source_empty, __address__ ]

    regex: "Y;(.*)(:.*)?"

    target_label: source

    replacement: "${1}"

  - target_label: __tmp_source_empty
    replacement: ""

[mohan garden]

Thank you for the response Brian,
with both config a and config b, i was unable to see the source label. 

suggested config a)

  metric_relabel_configs:
    - source_labels: [ source, __address__ ]
      regex: ";(.*)(:.*)?"
      target_label: source

      replacement: '${1}'

output: 
query: custom_metrics_from_server
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source="server1.example.com", status="run"} 3213

query: up
response:  
up{instance="localhost:9104", job="nodeexporter"}  1

query: node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", type="Processor"}
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", type="Processor"} 0


source= metrics was not available.


now to debug, i changed the query slightly , and it seems that the __address__ is not available with  metric_relabel_configs?
modified config a)   
    - source_labels: [ source, __address__ ]
      regex: "(.*)"

      target_label: source
      replacement: '${1}'

query: custom_metrics_from_server
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source="server1.example.com;", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source="server1.example.com;", status="run"}    3213

query: up
up{instance="localhost:9104", job="nodeexporter"}

query: node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", type="Processor"}
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", source=";", type="Processor"}  0 

so from the last query, it seems that __address__ is blank, and since source= is also blank, we see source=";" which appears to be concatenation of 2 blank values with ; as delimiter.
Q1: here i am unable to understand that how ; was appended after source value in  custom_metrics_from_server  .




config b) :

    metric_relabel_configs:
    - source_labels: [ source ]
      regex: ""
      target_label: __tmp_source_empty
      replacement: Y
    - source_labels: [ __tmp_source_empty, __address__ ]
      regex: "Y;(.*)(:.*)?"
      target_label: source
      replacement: "${1}"
    - target_label: __tmp_source_empty
      replacement: ""

output:
query: custom_metrics_from_server
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source="server1.example.com", status="run"} 3213

query: up
response:  
up{instance="localhost:9104", job="nodeexporter"}  1

query: node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", type="Processor"}
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", type="Processor"} 0


source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the __address__ is not available with  metric_relabel_configs with config b?


modified config b)



query: custom_metrics_from_server
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source=";", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter", source=";", status="run"} 3213     

query: up
up{instance="localhost:9104", job="nodeexporter"}

query: node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", type="Processor"}
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter", name="0", source="Y;", type="Processor"}


so from the last query, it seems that __address__ is blank, and since source= is also blank, we see source=";" which appears to be concatenation of 2 blank values with ; as delimiter.
so far , it appears that the __address__ field is not available with metrics_relabel_config?

Please advice.

[mohan garden]

Please ignore my previous email.
I am repeating previous message again as i missed out setting under modified configuration b):

    - target_label: __tmp_source_empty
      replacement: ""

3215