Mtail best practices with k8s

[Jonathan M. Bardin]

Hello everyone,

We are new to mtail and use it to collect metrics from our nginx log.

We plan to run mtail as a daemonset for each of our k8s nodes.

Our mtail daemon pod mount `/var/lib/docker/containers` from the k8s node. This folder contains the log files from the node we which to monitor.

However all those log files are in a sub-directory:

`/var/lib/docker/containers/<container_id>/<container_id>.log`

I was wondering if you anyone have some best practice regarding the usage of mtail with k8s.

Would it make more sens to run mtail as a sidecar container for each of our nodes.

Or keeping the daemonset approach but with a different approach to be able to watch all the log files directly with mtail? (for example, use symlink to flatten the folder hierachy).

Thanks!

Jonathan

[Jonathan M. Bardin]

Hi again,

Just as a follow up we ended up going with the daemonset approach.

We mount the host node /var/log/ and /var/lib/docker/containers into our mtail container.

We then rely on mtail glob: -logs /var/log/containers/myservice*.log

It's important to mount /var/lib/docker/containers in addition to /var/log/containers since the files in /var/log/containers are symlink that eventually points to file located /var/lib/docker/containers/{id}/{id}.log

This works well for us, we hadn't run into any issue so far with this approach.

Cheers to the mtail team!

[Jamie Wilkinson]

Sorry I haven't' replied; I'm currently on vacation.

Without giving it any thought; using daemonset for mtail seems appropriate for docker because it's on every node, and sidecar seems appropriate for applications that don't have their own instrumentation.

I'll be interested to hear how this works out for you, and if there's any config examples you can share with the rest of us that would be great.

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/5d902964-7ff6-4ce2-89db-e51f55a2bb38%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan M. Bardin]

No problem, thanks for the follow up.

It works very well for us so far. I attached our deamonset conf alongside our Dockerfile and the mtail service conf.

Basically all of our containers (pods) use the sysout for logging, which k8s will capture into log files in the host. The mtail deamonset tail those file using a glob pattern. Each file being prefix by the container/pod name in the same directory (/var/log/containers), that makes thing easier.

Without tailing any log, an mtail container have a footprint of ~2MB, we pick ~40MB when tailing a bunch of log files. We have ~26 host in our cluster.

Thanks for making such a great tool!

[Jamie Wilkinson]

You're welcome, it's my pleasure. Thanks for the feedback!

To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/ba78d103-cdd0-4a72-9ee0-6bcda3183ed8%40googlegroups.com.