evaluation interval cannot be recognized in seperate job

[nina guo]

Hi, I received following error.

- job_name: TLS Connection
        scrape_interval: 1d
        evaluation_interval: 1d
        metrics_path: /probe
        params:
          module: [smtp_starttls]
        file_sd_configs:
        - files:xxx

 kubectl logs prometheus -c prometheus -n monitoring
level=error ts=2022-08-12T03:03:50.120Z caller=main.go:347 msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n  line 54: field evaluation_interval not found in type config.ScrapeConfig"

[nina guo]

what i want to implement is to scrape and evaluate one kind of metrics not so frequently, I want to adjust the interval to 1d or 2d, something like this.

[Stuart Clark]

Two things here:

1. There is no entry called "evaluation_interval" within a scrape config, so that needs removing to clear the unmarshal error.

2. The maximum sensible scrape interval is around 2-3 minutes, so 1 day is far too long. With a longer interval you will end up with stale time series and "gaps" in all your graphs.

-- 
Stuart Clark

[nina guo]

OK. So if I want to scrape the metrics for 1 day interval, which way is better to implement?

[Stuart Clark]

On 12/08/2022 08:09, nina guo wrote:
> OK. So if I want to scrape the metrics for 1 day interval, which way
> is better to implement?

Some options would be:

- Scrape it every 2 minutes instead of daily
- Use the textfile collector of the node exporter, with a scheduled job
to update the file daily
- Use the push gateway with a scheduled job that updates the API daily

For the second two options you will lose the ability to use the "up"
metric (as that will now refer to the node exporter/push gateway
instead), but both add their own additional metrics containing timstamps
of the last time the metric was updated.

--
Stuart Clark

[nina guo]

So if with push gateway or textfile collector, we need to also to customize our metrics, am I right?

[Stuart Clark]

On 12/08/2022 08:29, nina guo wrote:
> So if with push gateway or textfile collector, we need to also to
> customize our metrics, am I right?

What do you mean?

--
Stuart Clark

[nina guo]

now I'm using blackbox exporter exposed metrics.

if i use pushgateway or textfile collector, i need to write a script  first to generate the metrics and write them with prometheus metrics format to textfile collector folder. 

[Brian Candler]

Perhaps the most import question to ask is: why do you only want to scrape once per day?

If you're worrying about storage space consumption: don't.  Prometheus uses differential encoding.  If you record the same metric repeatedly with the same value, i.e. the difference between the adjacent values is zero, then the amount of storage space used is miniscule.  Even if it's not the same value, it's extremely highly compressed.  And disk space is cheap.

If you only want to *check* a value every 1 day for alerting, then it's perfectly fine to have an alerting rule group with a 1 day evaluation interval - even if the underlying metric is collected more frequently. The alerting rule will use the most recent value at evaluation time.

If you're worried that the exporter itself is expensive to run on each scrape, then you should do one of the other things that Stuart suggested (i.e. using textfile collector or push gateway).  On the target node use a cron job to update the metric at whatever interval makes sense - but let prometheus scrape it as often as it likes.  But you should still scrape the cached value every 1-2 minutes.

If it's blackbox_exporter that you're talking about, those scrapes are cheap.  So you may as well scrape every 2 minutes.

If you're worried about transient errors causing spurious alarms, then adjust your alerting rules (e.g. in the simplest case, add "for: 30m" which means that you'll only get a notification if the alert has been active for 30m *continuously*).

[nina guo]

Thank you. Brian.

I'm worried about the blackbox_exporter is expensive or not. 

I also have other jobs, other jobs are with global 30s scrape intervals and 30s evaluation invervals, when I set the special job with scrape interval 2m, will it cause any problem?

"

If you only want to *check* a value every 1 day for alerting, then it's perfectly fine to have an alerting rule group with a 1 day evaluation interval - even if the underlying metric is collected more frequently. The alerting rule will use the most recent value at evaluation time.

"

regarding this point, is it possible to set alerting rule group for a special rule or a job?

[Brian Candler]

On Friday, 12 August 2022 at 08:50:28 UTC+1 ninag...@gmail.com wrote:

I'm worried about the blackbox_exporter is expensive or not. 

You are, or are not, worried about this?

 

I also have other jobs, other jobs are with global 30s scrape intervals and 30s evaluation invervals, when I set the special job with scrape interval 2m, will it cause any problem?

Each job defines its own scrape interval.  The global default is used only if it doesn't define its own scrape interval.

Similarly, each alerting rule group defines its own evaluation interval.  The global default is used only if it doesn't define its own evaluation interval.

Scraping and rule evaluation run entirely independently.

 

"

If you only want to *check* a value every 1 day for alerting, then it's perfectly fine to have an alerting rule group with a 1 day evaluation interval - even if the underlying metric is collected more frequently. The alerting rule will use the most recent value at evaluation time.

"

regarding this point, is it possible to set alerting rule group for a special rule or a job?

I don't understand the question.  You can create as many alerting rule groups as you like, and each rule group can have as many alerting rules as you like. Each alerting expression can access *all* the metrics in the TSDB - it is not tied to a specific scrape job.

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/