Prometheus - Alert Manager - Webhook (Splunk) - 401 Unauthorized error

[Nevethan SS]

Hello,

I am new to Prometheus (v2.13.1) and Alert manager (v0.19.0). I am trying to send the alerts to splunk using webhook_configs and I am getting HTTP 401 error. Please help:

Splunk webhook setup guide: https://docs.splunk.com/Documentation/Splunk/8.0.0/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_Splunk_Enterprise

Alertmanager yml:

global:
  resolve_timeout: 1m
route:
  receiver: custom
  group_by:
  - alertname
  routes:
  - receiver: custom
    continue: true
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 1h
receivers:
- name: custom
  webhook_configs:
  - send_resolved: true
    http_config:
      bearer_token: 'Splunk kd34662b-78e7-4923-b211-b26dab08cc4f'
      tls_config:
        insecure_skip_verify: true
    url: 'https://splunk1234:8088/services/collector'

Error message:

level=error ts=2019-11-21T21:26:49.534Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 401: http://splunk1234:8088/services/collector/event" context_err=null

[Simon Pasquier]

Already replied on IRC but for completeness:

Splunk expects a header with "Authorization: Splunk <your token>" but
your Alertmanager sends "Authorization: Bearer <your token>" (that's
the "bearer_token" setting) so Splunk rejects the request
You'd need a shim proxy between Alertmanager and the Splunk endpoint.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0dfbd801-93e2-4ede-9079-1cd056b91998%40googlegroups.com.

[Nevethan SS]

Thanks Simon. I will check that one. Also i noticed there is basic auth setting and i am getting 404 error. But i am able to send curl cmd with basic auth and i see the events in splunk search.

receivers:
- name: custom
  webhook_configs:
  - send_resolved: true
    http_config:

      basic_auth:
        username: x
        password: kd34662b-78e7-4923-b211-b26dab08cc4f
      tls_config:
        insecure_skip_verify: true
    url: 'https://splunk1234:8088/services/collector/'

Error:

level=error ts=2019-11-22T16:40:49.552Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 404: https://splunk1234:8088/services/collector/" context_err=null

Curl cmd:

curl -k -u "x:kd34662b-78e7-4923-b211-b26dab08cc4f" "https://splunk1234:8088/services/collector/event" \
    -d '{"event": "hello world from prometheus1"} {"text": "Success", "code": 0}'

[Simon Pasquier]

Your curl URL (https://splunk1234:8088/services/collector/event) is
different from the configured webhook URL
(https://splunk1234:8088/services/collector/).

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/87423e2d-95a0-491f-9e18-f63426ffa960%40googlegroups.com.

[Nevethan SS]

Hi Simon. Sorry for the confusion. I changed the curl url same as webhook url and it works fine with basic auth. But it fails via alertmanager only.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Simon Pasquier]

The code managing HTTP Basic Auth is also used in Prometheus
(especially the scraper part) so it's unlikely that there's a problem
with it. I'd double check your configuration and the Splunk
collector's logs if possible.

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/79099ef3-fa2a-4c87-981d-d3d5051e92f3%40googlegroups.com.

[Nevethan SS]

Hi Simon,

I am still stuck with this. Do you have any example how to setup the proxy for splunk HTTP event collector?

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/79099ef3-fa2a-4c87-981d-d3d5051e92f3%40googlegroups.com.

[Brian Candler]

404 means the URL is wrong.

"I changed the curl url same as webhook url and it works fine with basic auth. But it fails via alertmanager only."

Can you show exactly the curl command and the webhook config that you are now using?  The URLs need to be exactly the same.  Trailing slash may be significant.

Did you killall -HUP alertmanager after changing its config? 

[Nevethan SS]

Hi Brian,

Thanks for your reply. Here is the info. Also i have killed all the instances and tried restarting the server.

configuration:

receivers:
- name: custom
  webhook_configs:
  - send_resolved: true
    http_config:
      basic_auth:
        username: x
        password: kd34662b-78e7-4923-b211-b26dab08cc4f
      tls_config:
        insecure_skip_verify: true

    url: 'https://splunk1234:8088/services/collector'

curl command:

curl -k -u "x:kd34662b-78e7-4923-b211-b26dab08cc4f" "https://splunk1234:8088/services/collector" \
    -d '{"event": "hello world from prometheus"} {"text": "Success", "code": 0}'

[Brian Candler]

    -d '{"event": "hello world from prometheus"} {"text": "Success", "code": 0}'

Are you *sure* you copy-pasted that verbatim?  Becaues I don't think you did.

It shows you sending a single string containing two adjacent JSON objects.  What I think more likely is that the first is what you sent, and the second is the response.  Except the way the quotes are placed, that's not what it shows.

If you don't copy-paste from your screen, then we can't see what you see, so everything is pure guesswork.

Try adding -v to the curl request.  It will show you the full set of request and response headers.  See if there's any header which curl is adding which might be significant.

What about http logs at the splunk side? Do they show the request URL and method (GET/POST)? Do you see any difference between the curl and alertmanager requests?

Another possibility is to set up a fake HTTPS server and change the URL (of both curl and alertmanager) to write to that instead.  See what the difference is between the requests.