drop all some metrics based on regex

[mel]

Hello I am using node_exporter and I am trying to drop all node_systemd_unit_state metrics except for a handful of services like (e.g.,) ssh and apache. How would I do this? I came up with the following, but I don't think this is correct because it will drop other metrics as well (metrics that are not related to systemd service)

metric_relabel_configs:

    - source_labels: [__name__, name]

      regex: 'node_systemd_unit_state;(ssh|apache).*'

      action: keep

How do I drop all service metrics except for ssh and apache service?

[Ben Kochie]

relabel actions are exclusive. Drop means keep everything but X. Keep means drop everything but X.

For your exact problem, there is already a node_exporter flag to handle this.

./node_exporter --collector.systemd.unit-include="(ssh|apache)"

This will also be more efficient because it it will only gather data about those two units.

[mel]

You are absolutely correct but I don't have access to a lot of the servers so I am trying to drop them on the prometheus side

[Brian Candler]

You can use temporary variables. Something like this (untested):

metric_relabel_configs:

    - source_labels: [__name__, name]

      regex: 'node_systemd_unit_state;(ssh|apache).*'

      target_label: __tmp_keep

      replacement: y

    - source_labels: [__name__, __tmp_keep]

      regex: 'node_systemd_unit_state;'

      action: drop

[Ben Kochie]

The other way you can do this is with the "__tmp_keep" pattern. This is where you positively tag the metrics you want to keep, and then use a drop that matches that temporary label doesn't exist.

metric_relabel_configs:
  - source_labels: [__name__, name]

    regex: 'node_systemd_.*;(ssh|apache).*'
    target_label: __tmp_keep
    replacement: yes
  - source_labels: [__tmp_keep,__name__]
    regex: ';node_systemd_.*'
    action: drop
  - regex: __tmp_keep
    action: labeldrop

[Brian Candler]

Thanks. I always forget that labels starting with __ are automatically dropped after target relabelling, but not metric relabelling.