Monitoring Mixins with Jsonnet

[Tom Wilkie]

Hello Everyone

Frederic + I have been kicking round the idea of reusable monitoring packages (mixins) for a month or two, inspired by lots of the problems Julius described in his monitoring bundles design doc[1].  We're quite keen on the idea of using Jsonnet for this, as it allows us to make the configurable, extensible and reusable.

I've written up a motivation + design doc that goes into a bit more details, and I really appreciate any feedback you anyone might have:  

https://docs.google.com/document/d/1A9xvzwqnFVSOZ5fD3blKODXfsat5fg6ZhnKu9LK3lB4/edit#

At the bottom of the doc are the 4-5 examples we've built[2].  I'll be talking about this at KubeCon[3] and hanging out in Berlin the week after if anyone wants to meet in person for a coffee + chat.

Thanks

Tom!

[1] https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/prometheus-developers/yNFdBG3dSrU/oYym0l4kBwAJ

[2] https://docs.google.com/document/d/1A9xvzwqnFVSOZ5fD3blKODXfsat5fg6ZhnKu9LK3lB4/edit#heading=h.w4clwwhamp66

[3] https://kccnceu18.sched.com/event/Dqvp/prometheus-monitoring-mixins-using-jsonnet-to-package-together-dashboards-alerts-and-exporters-tom-wilkie-kausal-intermediate-skill-level?iframe=no&w=100%&sidebar=yes&bg=no

[Julius Volz]

Probably equally relevant for -users?

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAB58Z104%3DVnQ0ZcUsjbap%3D5_Uhvd6ncgYaKcjTbtuYWb4q7ZgA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Krasimir Georgiev]

Hi Tom , looks quite interesting.

I will run one of the examples and will send some feedback.

Or if you have time for a short demo/video that would be even better.

	Krasi Georgiev

--

You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

To post to this group, send email to prometheus...@googlegroups.com.

[Tom Wilkie]

Thanks Krasi! I think putting together a short demo is a great idea - hopefully I’ll get a chance to do that this week.

Tom

[Tom Wilkie]

Thank you everyone so far for you feedback on the doc - much appreciated.

I wanted to follow up and take some of the discussion off the the comments in the doc.  Frederic + I spent the day putting together a proof of concept for the Kubernetes mixin:

https://github.com/kubernetes-monitoring/kubernetes-mixin

It is still very early.  It combines the KLUMPs module from Kausal with some of the dashboards from kubernetes-grafana[1].  I intend to roll in more of those dashboards and alerts from the kube-prometheus[2] over the next few days. Feedback is very much welcome.

In particular there was some discussion on the doc about string substitution vs embedded DSL. Now that we have a more fully worked example that tries to use string substitution, we can see how well it works. In the config, you can specify a bit of a selector as config (currently using the defaults for my prometheus-jsonnet module[3]):

// Selectors are inserted between {} in Prometheus queries.
cadvisor_selector: 'job="kube-system/cadvisor"',
kubelet_selector: 'job="kube-system/kubelet"',

(https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/config.libsonnet#L3)

These are then substituted in dashboards, alerts and recording rules:

expr: |||
rate(kube_pod_container_status_restarts_total{%(kube_state_metrics_selector)s}[15m]) > 0
||| % $._config,

(https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/apps_alerts.libsonnet)

Generally, it seem to work well.  There are some places where is strains imagination (see `not_kube_dns_selector` [4][5]), but in general I think its more than enough for now, so I'm going to shelve the DSL idea.  I'll update the design doc to match.

We've also tried to standardise on a way of providing config for mixins, via a top-level, non-materialed, reserved `_config` field.  I'll be added this to the design doc in the next few days.

Thanks again

Tom

[1] https://github.com/brancz/kubernetes-grafana

[2] https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus

[3] https://github.com/kausalco/public/tree/master/prometheus-ksonnet

[4] https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/config.libsonnet#L8

[5] https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/system_alerts.libsonnet#L23

[Henri DF]

Late to the party but a high-level question on the original scope/background: Was there ever any consideration given to encompassing relabelling config as part of this?

Superficially the idea seems nice: by providing a relabelling config, you could potentially avert the need to re-configure selectors in "downstream" dashboard/rule/alert configurations. But the issue then becomes one of reconciling potentially different relabelling configs (e.g. mixins A and B both provide relabel configs for kubernetes-endpoint service discovery...).

I'd be super interested in historical context on this and/or thoughts. Thanks!

Henri

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.