Remote write write_relabel_configs not working

[Alex Zeleznikov]

I'm trying to configure remote write to Grafana Cloud, without write_relabel_configs everything works and gets sent fine, if I try to add more then one keep actions then nothing gets sent
Working example

remote_write:
- url: https://example.grafana.net/api/prom/push
  remote_timeout: 30s
  write_relabel_configs:
  - source_labels: [job]
    regex: (spark-exporter|sqs-exporter|prometheus)
    action: keep

When I add another keep action nothing gets sent.

remote_write:
- url: https://example.grafana.net/api/prom/push
  remote_timeout: 30s
  write_relabel_configs:
  - source_labels: [job]
    regex: (spark-exporter|sqs-exporter|prometheus)
    action: keep
  - source_labels: [__name__]
    regex: (container_last_seen|container_memory_usage_bytes|container_cpu_user_seconds_total|container_network_receive_bytes_total|container_network_transmit_bytes_total|node_uname_info|node_cpu_seconds_total|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_network_receive_bytes_total|node_network_transmit_bytes_total)
    action: keep

[Brian Candler]

You seem to have a break in your long second regex line, is that true in your original as well?

I guess you're aware that keep action is not cumulative - keep means "drop if does not match", and then follows through to the following rule.  Those rules will only keep if *both* conditions are true.  Is that what you were expecting?

Note that you can write a single rule which checks both at once, using

- source_labels: [job, __name__]

The string will consist of both job and name separated by a semicolon, so you can write a regex which matches both.

[Alex Zeleznikov]

I was not aware that keep action is not cumulative, I cannot find any good documentation/examples on how relabel_configs works.

I ended up using multiple remote writes each for a different label

remote_write:
- url: https://example.grafana.net/api/prom/push
  remote_timeout: 30s
  write_relabel_configs:

  - source_labels: [__name__]
    regex: (container_last_seen|container_memory_usage_bytes|container_cpu_user_seconds_total|container_network_receive_bytes_total|container_network_transmit_bytes_total|node_uname_info|node_cpu_seconds_total|node_memory_MemFree_bytes|node_memory_MemTotal_bytes|node_network_receive_bytes_total|node_network_transmit_bytes_total)
    action: keep

- url: https://example.grafana.net/api/prom/push
  remote_timeout: 30s
  write_relabel_configs:
  - source_labels: [job]

    regex: (sqs-exporter|prometheus)
    action: keep