Mtail best practices with k8s

Skip to first unread message

Jonathan M. Bardin

Oct 16, 2018, 1:56:54 AM10/16/18
to mtail-users
Hello everyone,

We are new to mtail and use it to collect metrics from our nginx log.

We plan to run mtail as a daemonset for each of our k8s nodes.

Our mtail daemon pod mount `/var/lib/docker/containers` from the k8s node. This folder contains the log files from the node we which to monitor.
However all those log files are in a sub-directory:


I was wondering if you anyone have some best practice regarding the usage of mtail with k8s.

Would it make more sens to run mtail as a sidecar container for each of our nodes.
Or keeping the daemonset approach but with a different approach to be able to watch all the log files directly with mtail? (for example, use symlink to flatten the folder hierachy).



Jonathan M. Bardin

Oct 16, 2018, 8:46:14 AM10/16/18
to mtail-users
Hi again,

Just as a follow up we ended up going with the daemonset approach.

We mount the host node /var/log/ and /var/lib/docker/containers into our mtail container.
We then rely on mtail glob: -logs /var/log/containers/myservice*.log

It's important to mount /var/lib/docker/containers in addition to /var/log/containers since the files in /var/log/containers are symlink that eventually points to file located /var/lib/docker/containers/{id}/{id}.log

This works well for us, we hadn't run into any issue so far with this approach.

Cheers to the mtail team!

Jamie Wilkinson

Oct 25, 2018, 9:00:32 PM10/25/18
Sorry I haven't' replied; I'm currently on vacation.

Without giving it any thought; using daemonset for mtail seems appropriate for docker because it's on every node, and sidecar seems appropriate for applications that don't have their own instrumentation.

I'll be interested to hear how this works out for you, and if there's any config examples you can share with the rest of us that would be great.

You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To post to this group, send email to
To view this discussion on the web visit
For more options, visit

Jonathan M. Bardin

Oct 29, 2018, 3:32:42 AM10/29/18
to mtail-users
No problem, thanks for the follow up.

It works very well for us so far. I attached our deamonset conf alongside our Dockerfile and the mtail service conf.

Basically all of our containers (pods) use the sysout for logging, which k8s will capture into log files in the host. The mtail deamonset tail those file using a glob pattern. Each file being prefix by the container/pod name in the same directory (/var/log/containers), that makes thing easier.

Without tailing any log, an mtail container have a footprint of ~2MB, we pick ~40MB when tailing a bunch of log files. We have ~26 host in our cluster.

Thanks for making such a great tool!

Jamie Wilkinson

Oct 30, 2018, 9:21:43 AM10/30/18
to Jonathan M. Bardin, mtail-users
You're welcome, it's my pleasure. Thanks for the feedback!

Reply all
Reply to author
0 new messages