Alert Manger Issues: "x509: certificate is not valid for any names" AND "context deadline exceeded"

148 views
Skip to first unread message

Hyojun Kim

unread,
Sep 28, 2022, 6:49:16 AM9/28/22
to Prometheus Users
I have installed kube-prometheus-stack v 0.59.1 from helm chart on my K3S cluster with the following values.

```
kubeTargetVersionOverride: '1.25'
namespaceOverride: 'monitoring'
grafana:
  namespaceOverride: 'monitoring'
```

My ubuntu version is below and it is running as a virtual machine in Proxmox.

```
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:    20.04
Codename:    focal
```

After installing Prometheus stack to my K3S cluster, I set my alertmanagner secret as below for the slack notification
```
global:
  slack_api_url: 'https://hooks.slack.com/services/API_KEY......'

route:
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 1m
  repeat_interval: 10m
  receiver: 'slack'

receivers:
  - name: 'slack'
    slack_configs:
      - channel: '#codepot'
        icon_emoji: ':bell:'
        send_resolved: true
        text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}\nmessage: {{ .CommonAnnotations.message }}"
```
With this config I am getting "x509: certificate is not valid for any names, but wanted to match hooks.slack.com"
So, I have added insecure_skip_verify = true to secret as below
```
global:
  slack_api_url: 'https://hooks.slack.com/services/API_KEY......'
  http_config:
    tls_config:
      insecure_skip_verify: true
route:
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 1m
  repeat_interval: 10m
  receiver: 'slack'

receivers:
  - name: 'slack'
    slack_configs:
      - channel: '#codepot'
        icon_emoji: ':bell:'
        send_resolved: true
        text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}\nmessage: {{ .CommonAnnotations.message }}"
```
Now, I am getting "Post \"<redacted>\": context deadline exceeded" error.
I am also getting the same error with the Pagerduty.
```
receivers:
  - name: 'slack'
    pagerduty_configs:
      - service_key: API_KEY
```
Odd thing is, if I install exactly the same helm chart and secret to my Macbook using Minikube or Kubernetes that comes within the docker dashboard, it works!
Any idea what am I doing wrong? 
Reply all
Reply to author
Forward
0 new messages