Simple alert rule logic not working

[Unni Sathyarajan]

Hello Guys, 

Following is alert rule I have created from my instrumented app:

- alert: LastSectionPullTimeExceeded

    expr: (time()- pull_sectionlists{instance="example.org:8000"}) > 2155

    for: 2m

    labels:

      severity: warning

    annotations:

      impact: 'Last pull time of content exceeded by 20mins'

      summary: Last pull time of contents  exceeded by 20mins

pull_sectionlists : returns an epoch time so to convert to current time I have used time()  and it works fine to display the metric in Grafana. 

But to create an alert, the prometheus expression does not work. 

The value of  (time() - pull_sectionlists{instance="example.org:8000") has gone up way above 2155 but still the alerts are not yet triggered and prometheus does not show it as FIRING state.

Any idea why the expression does not work?

Thanks 

[Brian Brazil]

What version of Prometheus are you running, and how often are you reloading the config? This might be a bug we're just fixing.

 

Thanks 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5c488f47-dcfc-4fc6-b15d-bcf2825d14db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

Hi Brian, 

I am running prometheus 2.0 

prometheus, version 2.0.0-rc.1 (branch: HEAD, revision: 5ab8834befbd92241a88976c790ace7543edcd59)

  build user:       root@1f56dd8b6f7b

  build date:       20171017-12:34:15

  go version:       go1.9.1

Prometheus.yml:  

global:

  scrape_interval:     15s 

  evaluation_interval: 15s 

Thanks

[Brian Brazil]

How often are you reloading the config file?

Brian

On 1 November 2017 at 13:34, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Hi Brian, 

I am running prometheus 2.0 

prometheus, version 2.0.0-rc.1 (branch: HEAD, revision: 5ab8834befbd92241a88976c790ace7543edcd59)

  build user:       root@1f56dd8b6f7b

  build date:       20171017-12:34:15

  go version:       go1.9.1

Prometheus.yml:  

global:

  scrape_interval:     15s 

  evaluation_interval: 15s 

Thanks

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

Sorry, where can I find this info? 

I execute the following command to reload my config file  (prometheus.yml) in Prometheus:

curl -X POST http://localhost:9090/-/reload

and also make sure to check if the rule updates appear on http://prometheus.example.org/alerts - page. 

And this is how I start prometheus: 

./prometheus --storage.tsdb.retention=15d --web.enable-lifecycle --config.file=prometheus.yml

[Brian Brazil]

On 1 November 2017 at 13:41, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Sorry, where can I find this info? 

changes(prometheus_config_last_reload_success_timestamp_seconds[5m]) should tell you.

Brian

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

changes(prometheus_config_last_reload_success_timestamp_seconds[5m]) = 0 

[Brian Brazil]

What do you see if you graph (time()- pull_sectionlists{instance="example.org:8000"}) > 2155  ?

Brian

On 1 November 2017 at 13:50, Unni Sathyarajan <unnisa...@gmail.com> wrote:

changes(prometheus_config_last_reload_success_timestamp_seconds[5m]) = 0 

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/57oGnAvpZlw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLrrcakQ%3DO9-_E0_qokzWA%3DiYRQpd-5U4rvEUZW%2BSSTeqg%40mail.gmail.com.

[Brian Brazil]

Can you share your full configuration?

On 1 November 2017 at 14:00, Unni Sathyarajan <unnisa...@gmail.com> wrote:

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

Hi Brian, 

Sure, please find the configuration details below:

FILE:prometheus.yml

global:

  scrape_interval:     15s 

  evaluation_interval: 15s 

  external_labels:

      monitor: 'codelab-monitor'

 

rule_files:

  - "./alert-rules/dockerhost.yml"

  - "./alert-rules/pub1.yml"

 

alerting:

  alertmanagers:

    - static_configs:

      - targets: ["localhost:9093"]

scrape_configs:

  - job_name: 'prometheus'

    static_configs:

      - targets: ['localhost:9090']

  - job_name: 'pub1'

    static_configs:

      - targets: ['pub1.example.net:9100','pub1.example.net:9121','pub1.example.net:8000']

        labels:

         name: 'pub1.example.net'

         env: 'prod'

  - job_name: 'dockerhost'

    static_configs:

      - targets: ['1.2.3.4:4194','1.2.3.4:9100','1.2.3.4:9121']

        labels:

         name: 'dockerhost'

         env: 'test'                                                                                                                                

 

FILE:./alert-rules/pub1.yml

- name: alert.rules

  rules:

  - alert: LastSectionPullTimeExceeded

    expr: round(time()- pull_sectionlists{name="pub1.example.net",instance="pub1.example.net:8000"}) > 2155

    for: 2m

    labels:

      severity: warning

    annotations:

      impact: 'Last pull time content  exceeded by 20mins'

      summary: Last pull time content  exceeded by 20mins

 

 

FILE: alertmanager.yml

global:

templates: 

- '/etc/alertmanager/template/*.tmpl'

route:

  group_wait: 30s

  group_interval: 5m

  repeat_interval: 3h 

  receiver: first-level-notifiers 

  routes:

inhibit_rules:

- source_match:

    severity: 'critical'

  target_match:

    severity: 'warning'

  equal: ['alertname', 'cluster', 'service']

receivers:

- name: 'first-level-notifiers'

  email_configs:

  - to: 'unnisa...@gmail.com'

    send_resolved: true

    from: 'ad...@prometheus.example.cc'

    smarthost: 'ses-endpoint:25'

    auth_username: 'abc'

    auth_password: 'abc'

  - to: 'a...@example.net'

    send_resolved: true

    from: 'ad...@prometheus.example.cc'

    smarthost: 'ses-endpoint:25'

    auth_username: 'abc'

    auth_password: 'abc124'

  slack_configs:

  - api_url: https://abc.1234./

    send_resolved: true

    channel: '#prometheus-alerts'

I had set up alerts for other server metrics such as CPU, DISK, NETWORK I/O which works fine 

[Brian Brazil]

That all looks okay. Can you share the Alerts page on Prometheus?

Brian

On 1 November 2017 at 14:23, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Hi Brian, 

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

[Brian Brazil]

This is smelling like a bug. Does this also happen with Prometheus 1.8.1?

Brian

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

I have not tried it yet. 

[Unni Sathyarajan]

Is it possible to convert the alert rules from proemtheus 2.0 to prometheus 1.8 compatible syntax using promtool ?

On Wed, Nov 1, 2017 at 6:49 PM, Unni Sathyarajan <unnisa...@gmail.com> wrote:

I have not tried it yet. 

[Brian Brazil]

On 1 November 2017 at 15:06, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Is it possible to convert the alert rules from proemtheus 2.0 to prometheus 1.8 compatible syntax using promtool ?

No, it only goes the other direction. But for one rule it shouldn't be difficult to convert by hand.

The conversion is more intended to help those where there's too many rules to convert by hand.

Brian

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

I have tried with prometheus 1.8.0 , same issue. Alert is not getting triggered. :-(

[Brian Brazil]

On 1 November 2017 at 15:24, Unni Sathyarajan <unnisa...@gmail.com> wrote:

I have tried with prometheus 1.8.0 , same issue. Alert is not getting triggered. :-(

This is probably an issue on your end then. Can you share the results of pull_sectionlists[10m] in the expression browser table view?

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

[Brian Brazil]

Thanks, no oddities in the raw data.

How about: (time()- pull_sectionlists) > 2155

Brian

On 1 November 2017 at 15:31, Unni Sathyarajan <unnisa...@gmail.com> wrote:

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

[Brian Brazil]

Hmm, how about: ALERTS

That's a special time series used to record alerts.

Brian

On 1 November 2017 at 15:40, Unni Sathyarajan <unnisa...@gmail.com> wrote:

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

Becoming an image editing expert now 😂

[Brian Brazil]

I meant what does the expression "ALERTS" show in the expression browser?

If you have any recording rules, are their time series getting updated?

Brian

On 1 November 2017 at 15:52, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Becoming an image editing expert now 😂

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

I get "No Datapoints Found" error !!

[Unni Sathyarajan]

Brian, 

Could you please share a link to know about the "special times series for alerts"

On Wed, Nov 1, 2017 at 8:07 PM, Unni Sathyarajan <unnisa...@gmail.com> wrote:

I get "No Datapoints Found" error !!

[Brian Brazil]

On 1 November 2017 at 16:09, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Brian, 

Could you please share a link to know about the "special times series for alerts"

https://prometheus.io/docs/alerting/rules/#inspecting-alerts-during-runtime

My best guess is that recording rules aren't being evaluated. Does a restart of Prometheus fix this?

--

Brian Brazil

www.robustperception.io

[Unni Sathyarajan]

Yes Brian, I have tried restarting it. It's still the same.

[Brian Brazil]

Is the prometheus_rule_evaluation_duration_seconds_count time series increasing?

Brian

On 2 November 2017 at 04:50, Unni Sathyarajan <unnisa...@gmail.com> wrote:

Yes Brian, I have tried restarting it. It's still the same.

--

Brian Brazil

www.robustperception.io