ssl expiry notification

[barnyb...@gmail.com]

Hello my friends.

I'm using ribbybibby/ssl_exporter  for checking ssl expiry for some services. All works fine  but I would like to add more information to the slack message. Specifically, add to the messages instance on which the certificates expire.

Now the slack receives messages with the name of the alert, the number of instances with expiring certificates and a link to the prom. I'm trying to change the config but to no avail so far.

```

  - alert: ssl_cert_expire

    expr: ssl_cert_not_after{ - time() < 86400 *7

    for: 2m

    labels:

      severity: 'warning'

    annotations:

      title: 'Warning: SSL cert will expire soon for the site (instance {{ $labels.instance }})'

      description: 'SSL cert expires in 7 days\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}'
```

[Christian Hoffmann]

Hi,

On 10/23/20 5:54 PM, barnyb...@gmail.com wrote:
> Hello my friends.

> I'm using ribbybibby <https://github.com/ribbybibby>/*ssl_exporter
> <https://github.com/ribbybibby/ssl_exporter>  *for checking ssl expiry

I think we would need some more details in order to help:

1) Can you verify the PromQL expression (there seems to be a "{" too
much or something else missing)?
2) Can you share an example result from your PromQL?
3) Can you show your Alertmanager config, especially the slack receiver
part (be sure to delete any secrets)?
4) Can you verify that your config reloads of Prometheus and
Alertmanager have been successful? Try checking the logs (stderr) and/or
the config metrics.

Kind regards,
Christi

[barnyb...@gmail.com]

I apologize for providing incomplete information. If the license expires for one instance, then its name is sent to the slack, if there are 2 or more of them, then it is sent without instance hostnames, but as in the screenshot above, just the number of instances.

docker-compose.yml

  prometheus_alertmanager:

    image: prom/alertmanager:v0.21.0

    container_name: prometheus-alertmanager

    ports:

      - '9093:9093'

    volumes:

      - prometheus-alertmanager-volume:/alertmanager

      - ./prometheus-alertmanager/conf/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro

    command:

      - '--config.file=/etc/alertmanager/alertmanager.yml'

      - '--storage.path=/alertmanager'

      - '--cluster.advertise-address=10.10.5.187:9093'

    networks:

      vpcbr:

        ipv4_address: 172.10.0.5

alertmanager.yml

global:

  slack_api_url: 'https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxxxxx/xxxxxxxxxxxxxxxxx'

route:

  group_by: [alertname]

  # Send all notifications to me.

  receiver: notify-me

receivers:

- name: notify-me

  email_configs:

  - to: m...@domain.com

    from: sen...@server.com

    send_resolved: true

    smarthost: smtp.eu.mailgun.org:587

    auth_username: "acc...@uname.com"

    auth_identity: "so...@identity.com"

    auth_password: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

  slack_configs:

  - channel: '#alerts'

    text: 'https://prometheus/alerts'

    send_resolved: true

Alert_rules.yml 

 - alert: ssl_cert_expire

    expr: ssl_cert_not_after - time() < 86400 *7

    for: 2m

    labels:

      severity: 'warning'

    annotations:

      title: 'Warning: SSL cert will expire soon for the instance {{ $labels.instance }}'

      description: 'SSL cert expires in 7 days {{ $labels.instance }} of job {{ $labels.job }}'

PromQl query:
ssl_cert_not_after{cn="server.com",dnsnames=",server.com,",instance="server.com",issuer_cn="Let's Encrypt Authority X3",job="ssl",serial_no="some serial"}

[Christian Hoffmann]

Hi,

On 10/26/20 5:09 PM, barnyb...@gmail.com wrote:
> I apologize for providing incomplete information. If the license expires
> for one instance, then its name is sent to the slack, if there are 2 or
> more of them, then it is sent without instance hostnames, but as in the
> screenshot above, just the number of instances.

[...]

>   slack_configs:
>   - channel: '#alerts'
>     text: 'https://prometheus/alerts'
>     send_resolved: true

It looks like you are mostly using the default slack config. I don't use
Slack myself, but according to the docs [1], the template
slack.default.title [2] kicks in, which will use a generic __subject
template [3]. That template uses CommonLabels, which may explain the
difference in behavior you are seeing regarding one/multiple alerts.

Long story short, I think you should configure a custom title template
for slack if you want to modify the existing message. Another option
would be to keep the existing title and add the details to the "text"
field, which may be more appropriate.

You can either work through the docs yourself (the notification template
syntax will help [4], along with the examples [5]) or try to find some
blog posts such as [6] where people document how they tuned their Slack
alerts for their needs.

Kind regards,
Christian

[1] https://prometheus.io/docs/alerting/latest/configuration/#slack_config
[2]
https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl#L15
[3]
https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl#L4
[4] https://prometheus.io/docs/alerting/latest/notifications/
[5] https://prometheus.io/docs/alerting/latest/notification_examples/
[6] https://harthoover.com/pretty-alertmanager-alerts-in-slack/