Unable to template room_id in hipchat_config

[dees...@gmail.com]

I've been trying to set up alerts in prometheus with a 'hipchatroom' label to enable routing of alerts to the appropriate room. In my alertmanager route configuration I have:

routes:

      - group_by:

        - hipchatroom

        group_interval: 1m

        group_wait: 10s

        match:

          severity: hipchat

        receiver: hipchat-notifier

        repeat_interval: 5m

My prometheus alert is configured as:

- alert: HighCPU

        expr: ((sum(rate(node_cpu{mode!='idle'}[5m]))/ count(node_cpu{mode="system"})) * 100) > 1

        for: 1m

        labels:

          severity: hipchat

          hipchatroom: 8675309

        annotations:

          summary: "HighCPU"

And finally in my alertmanager receiver config:

receivers:

    - hipchat_configs:

      - api_url: https://secret.hipchat.com/

        auth_token: secret

        from: FRIENDLY.ALERT.MANAGER

        message: '{{ .GroupLabels.hipchatroom }}'

        message_format: text

        notify: true

        room_id: '{{ .GroupLabels.hipchatroom }}'

        send_resolved: true

      name: hipchat-notifier

And this all results in the following errors (when log.level=debug):

level=debug ts=2018-07-11T00:57:22.557263748Z caller=notify.go:605 component=dispatcher msg="Notify attempt failed" attempt=1 integration=hipchat receiver=hipchat-notifier err="unexpected status code 404"

level=error ts=2018-07-11T00:57:22.557372944Z caller=notify.go:303 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"hipchat\" due to unrecoverable error: unexpected status code 404"

For debugging I have hardcoded the correct room id into the receiver as room_id: 8675309 and then the messages go to hipchat and the templating in the message field (the same template as I try to used in room_id) correctly prints the room_id from the label I set in my alert. According to the docs at https://prometheus.io/docs/alerting/configuration/#%3Chipchat_config%3E the room_id field should be a tmpl_string same as the message field but for some reason it doesn't seem to work. Is this a bug or am I doing something wrong?

[dees...@gmail.com]

I've also attempted defining a template for room_id in a separate file and including that in my alertmanager config:

/etc/config # cat templates.tmpl

{{ define "hipchat.roomid.text" }}{{ .GroupLabels.hipchatroom }}{ end }}

And then assign room_id as:

room_id: '{{ template "hipchat.roomid.text" . }}'

Again, this template works when used in the message field but used in room_id it seems to fail. I even tried hardcoding the template like:

/etc/config # cat templates.tmpl

{{ define "hipchat.roomid.text" }}amtest{ end }}

And when used in room_id it causes the 404 errors but works as expected in the message field. On reviewing the code at https://github.com/prometheus/alertmanager/blob/a736a90dd031b4d2b2ee277346c2573961232e9f/notify/impl.go#L755 I don't see anywhere where tmplText method is called on the room_id field.

[Simon Pasquier]

There's definitely a discrepancy between the documentation and the code. Please file an issue at https://github.com/prometheus/alertmanager/issues/new

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/722ffc96-cb82-4607-9690-7b6824fd14bd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simon Pasquier]

Ah I missed that you've already opened a PR: https://github.com/prometheus/alertmanager/pull/1463