I have installed kube-prometheus-stack v 0.59.1 from helm chart on my K3S cluster with the following values.
```
kubeTargetVersionOverride: '1.25'
namespaceOverride: 'monitoring'
grafana:
namespaceOverride: 'monitoring'
```
My ubuntu version is below and it is running as a virtual machine in Proxmox.
```
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal
```
After installing Prometheus stack to my K3S cluster, I set my alertmanagner secret as below for the slack notification
```
global:
slack_api_url: '
https://hooks.slack.com/services/API_KEY......'
route:
group_by: ['alertname']
group_wait: 5s
group_interval: 1m
repeat_interval: 10m
receiver: 'slack'
receivers:
- name: 'slack'
slack_configs:
- channel: '#codepot'
icon_emoji: ':bell:'
send_resolved: true
text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}\nmessage: {{ .CommonAnnotations.message }}"
```
With this config I am getting "x509: certificate is not valid for any names, but wanted to match
hooks.slack.com"
So, I have added insecure_skip_verify = true to secret as below
```
global:
slack_api_url: '
https://hooks.slack.com/services/API_KEY......'
http_config:
tls_config:
insecure_skip_verify: true
route:
group_by: ['alertname']
group_wait: 5s
group_interval: 1m
repeat_interval: 10m
receiver: 'slack'
receivers:
- name: 'slack'
slack_configs:
- channel: '#codepot'
icon_emoji: ':bell:'
send_resolved: true
text: "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}\nmessage: {{ .CommonAnnotations.message }}"
```
Now, I am getting "Post \"<redacted>\": context deadline exceeded" error.
I am also getting the same error with the Pagerduty.
```
receivers:
- name: 'slack'
pagerduty_configs:
- service_key: API_KEY
```
Odd thing is, if I install exactly the same helm chart and secret to my Macbook using Minikube or Kubernetes that comes within the docker dashboard, it works!
Any idea what am I doing wrong?