how to troubleshoot alert rule not loading

[Derek Lai]

Hi

We are evaluating using Prometheus v. 2.15.2. Here's my prometheus.yml:

global:

  scrape_interval: 15s # By default, scrape targets every 15 seconds.

  evaluation_interval: 15s # By default, scrape targets every 15 seconds.

  external_labels:

    monitor: "example"

rule_files:

  - "alert.rules"

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      - 'localhost:9093'

scrape_configs:

  - job_name: "prometheus"

    scrape_interval: 5s

    scrape_timeout: 5s

    static_configs:

      - targets:

        - localhost:9090

  - job_name: node

    static_configs:

      - targets:

         - localhost:9100

Here's the alert rule file, alert.rules:

groups:

- name: alert.rules

  rules:

  - alert: InstanceDown

    expr: up == 0

    for: 1m

    labels:

      severity: "critical"

    annotations:

      summary: "Endpoint {{ $labels.instance }} down"

      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes."

I've ran promtool check rule on this rule file and it passes. Looking thru docker log:

level=info ts=2020-04-17T21:36:28.087Z caller=main.go:734 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml

....

level=info ts=2020-04-17T21:36:28.092Z caller=main.go:762 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-17T21:36:28.092Z caller=main.go:617 msg="Server is ready to receive web requests."

but in the UI under Rules it says: "No rules defined". How can I go about troubleshooting what is wrong?

Thanks,

Derek

[Augustin Husson]

Hello,

I guess that's because prometheus doesn't find the file.

It's quite a weird extension you have here. Why did you put ".rules" instead of .yaml ? 

Maybe it's coming from there. And looking at the path you put, check that the alertRule file is in the same directory than the prometheus binary.

Cheers,

Augustin. 

_____________________________________________________________

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/74b9a4f7-d327-47c7-ae01-a9b14f15e849%40googlegroups.com.

[Brian Candler]

On Saturday, 18 April 2020 01:15:12 UTC+1, Augustin Husson wrote:

check that the alertRule file is in the same directory than the prometheus binary.

If you use relative paths, I believe they are relative to the directory containing the prometheus.yml file (not the directory containing the binary, nor the cwd)

I have:

/opt/prometheus/prometheus  # the binary

/etc/systemd/system/prometheus.service

...

WorkingDirectory=/opt/prometheus

/etc/prometheus/prometheus.yml

...

rule_files:

  - "rules/*.yml"

/etc/prometheus/rules/whatever.yml

[Augustin Husson]

Well a way to avoid these questions would be to use first an absolute path.

At least it will help you to understand why Prometheus doesn't find your file.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0dd8287e-70c5-408a-927c-4202996ced63%40googlegroups.com.

[Derek Lai]

Well, I found out that there is an option for debugging by adding command line option "--log.level=debug". However, I had to use the executable version instead of running from a docker container. When I do that, it was able to load my rules. So I am able to proceed with testing the alert rules.

Thanks for the suggestions. Augustin, if you look in the default prometheus.yml that comes with the software, it shows using "first.rules", "second.rules". I also googled and found there were discussions about people having trouble with using /etc/prometheus/rules.d/*/*.yml and the discussion seems to suggest that the software was expecting the rule files to be name *.rules. Originally I used "alert.rules.yml".

Derek

---

From: promethe...@googlegroups.com <promethe...@googlegroups.com> on behalf of Augustin Husson <husson....@gmail.com>
Sent: Saturday, April 18, 2020 2:09 AM
To: Brian Candler <b.ca...@pobox.com>
Cc: Prometheus Users <promethe...@googlegroups.com>
Subject: Re: [prometheus-users] how to troubleshoot alert rule not loading

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOJizGfM9E%3DUO04H9v1XYoULAPUTTQNEPGEpzxG8Oe_acccKAw%40mail.gmail.com.