Integrating Prometheus with Splunk and ServiceNow for automated ticket creation.

[Er Ravikiran Paatil]

while Integrating Prometheus with Splunk,

      - name: 'splunk-webhook' # Adding Splunk Webhook
        webhook_configs:
          - url: 'http://glchbs-st1001.eu.org.net:8088/services/collector'
            send_resolved: true
            http_config:
              basic_auth:
                username: x
                password: 2etr5t549d43-d2d0-417e-ver7-1234569a2f9a2
  - name: 'splunk-webhook'
        webhook_configs:
          - url: 'http://prometheus-alertmanager-splunkbot.isdt-sbxprom.svc.cluster.local:44553/alerts'
            send_resolved: true



i am getting...
2024-02-23T05:51:13.408Z caller=dispatch.go:515 level=debug component=dispatcher aggrGroup="{}/{severity=~\"^(?:critical|Critical|info|Critica)$\"}:{alertname=\"TEST\", namespace=\"isdt-sbxtest\"}" msg=flushing alerts=[TEST[e6f0eaf][active]]
2024-02-23T05:51:13.408Z caller=dispatch.go:515 level=debug component=dispatcher aggrGroup="{}/{severity=~\"^(?:critical|Critical|info|Critica)$\"}:{alertname=\"TEST\", namespace=\"isdt-sbxtest\"}" msg=flushing alerts=[TEST[e6f0eaf][active]]
2024-02-23T05:51:13.420Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="splunk-webhook/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 401: http://glchbs-st21234.eu.org.net:8088/services/collector: {\"text\":\"Invalid authorization\",\"code\":3}"

[Brian Candler]

> Invalid authorization

Seems you're not authorizing to Splunk properly. Can you point to their documentation which says how you need to authenticate to their API?

I note you're using http rather than https, so HTTP basic auth is probably not allowed (it's insecure, it sends the username and password in cleartext along with every request). But even with https, they may require you to authenticate in some other way.

[Aditya Sharma]

Hi Team,

Now we are authenticating successfully with username and password(while the password is given as Bearer Token of Splunk). but facing an issue while sending Alert data from Prometheus to Splunk, giving the error "No DATA" seems like Splunk is looking for an event header in the Data block as well we're currently encountering a roadblock in our efforts to integrate Prometheus alerts into Splunk. Every attempt at integration results in an error labeled "NO DATA," accompanied by error code 5.



Here's a snippet of the data we're trying to integrate:

{
    "receiver": "splunk-webhook",
    "status": "firing",
    "alerts": [{
        "status": "firing",
        "labels": {
            "alertname": "TEST",
            "env": "isdt-sbx",
            "namespace": "isdt-sbxtest",
            "severity": "critical"
        },
        "annotations": {
            "description": "description of the alert",
            "runbook": "http://runbook.biz",
            "summary": "summary of the alert"
        },
        "startsAt": "2024-02-26T12:38:53.724141255Z",
        "endsAt": "0001-01-01T00:00:00Z",
        "generatorURL": "",
        "fingerprint": "e6f0eaf72b9d568c"
    }],
    "groupLabels": {
        "alertname": "TEST",
        "namespace": "isdt-sbxtest"
    },
    "commonLabels": {
        "alertname": "TEST",
        "env": "isdt-sbx",
        "namespace": "isdt-sbxtest",
        "severity": "critical"
    },
    "commonAnnotations": {
        "description": "description of the alert",
        "runbook": "http://runbook.biz",
        "summary": "summary of the alert"
    },
    "externalURL": "https://monitoring.server.net/alertmanager",
    "version": "4",
    "groupKey": "{}/{severity=~\"^(?:critical|Critical|info|Critica)$\"}:{alertname=\"TEST\", namespace=\"isdt-sbxtest\"}",
    "truncatedAlerts": 0
}

Is there any possibility we can add a receiver with a template like Slack webhook or some other parameter that we can pass event while sending Data to Splunk?

If anyone in the group has encountered a similar issue or has expertise in Prometheus to Splunk integration, we would greatly appreciate your insights and recommendations on resolving this challenge.

Thanks & Regards,
Aditya Sharma