recording rules duplication

[deln...@gmail.com]

Using different jobs to scrape different groups of same exporters. At the same time I would like to make alerting rules be clean by introducing recording rules. My question is following approach fine? I understand that in expr you can use logical operators, but than if recording metric is used for calculation some other recorded metrics it's going to break or get to complicated. Is there a clean way?
  groups:
    - name: node-exporter.rules
      rules:
      - expr: |
          count without (cpu) (
            count without (mode) (
              node_cpu_seconds_total{job="node_1"}
            )
          )
        record: instance:node_num_cpu:sum
    - name: node-exporter.rules
      rules:
      - expr: |
          count without (cpu) (
            count without (mode) (
              node_cpu_seconds_total{job="node_2"}
            )
          )
        record: instance:node_num_cpu:sum

[Brian Candler]

On Friday, 30 October 2020 13:18:11 UTC, deln...@gmail.com wrote:

Using different jobs to scrape different groups of same exporters.

Any particular reason for separating them over multiple jobs?

In any case, what problem does it give you?  Your alerting rules can be written without reference to the job label, e.g.

expr: count without (cpu, job) ( ... )

although it may be clearer to say what label(s) you *do* want to group by, e.g.

expr: count by (instance) ( ... )

 

At the same time I would like to make alerting rules be clean by introducing recording rules. My question is following approach fine?

I don't like having two recording rules writing to the same metric, and I'm not sure if it's allowed.  But again, I don't see why you want to separate those rules by job anyway, when a single rule can cover both jobs.  Why not:

    - name: node-exporter.rules
      rules:
      - expr: |
          count without (cpu) (
            count without (mode) (
              node_cpu_seconds_total

            )
          )
        record: instance:node_num_cpu:sum

or:

    - name: node-exporter.rules
      rules:
      - expr: |
          count without (cpu) (
            count without (mode) (

              node_cpu_seconds_total{job=~"node_1|node_2"}

            )
          )
        record: instance:node_num_cpu:sum

?

 

I understand that in expr you can use logical operators, but than if recording metric is used for calculation some other recorded metrics it's going to break or get to complicated. Is there a clean way?

I'm not sure what you're asking, as you didn't give any example of how you wanted to use a logical operator.

Logical operators come in two flavours:

foo > 0   is a filter.  It allows the timeseries through only if its value is > 0, and filters it out otherwise.

foo > bool 0   is a value.  It always passes some value through, but that value is either 0 or 1 depending on the condition.

Normally filters are used in alerting rules.  Alerting rules trigger if one or more timeseries passes through the filters.

[Nemanja Delic]

Brian,

Different jobs are necessary as server groups are using different certs for authentication. My concern is if we go

    - name: node-exporter.rules
      rules:
      - expr: |
          count without (cpu) (
            count without (mode) (
              node_cpu_seconds_total{job=~"node_1|node_2"}

            )
          )

        record: instance:node_num_cpu:sum

than for instance:

     - expr: |
          (
            node_load1{job=~"node_1|node_2"}
          /
            instance:node_num_cpu:sum{job="node_1|node_2"}
          )
        record: instance:node_load1_per_cpu:ratio

There's a chance that we can have more than one record on top of this. Things get complicated quite easily.

Thanks for helping on this.

 

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/AGp_TeBLAJ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/683dd70e-215a-42ff-bfa7-ded9901b62b1o%40googlegroups.com.