Log rotation to avoid a disk full

[poiuytrez]

Hello, 

I am quite new in the container world. I have an app that is generating logs in /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log until the disk is full.

1. Do you have a quick fix to avoid storing any logs

2. Is there a way to rotate logs? I have only a coreos machine (no stuff with fleet or any cluster management system).

Thank you for your help

poiuytrez

[Brandon Philips]

Hello Poiuytrez-

One thing you could try is using the `rkt` container runtime instead, rkt relies on systemd's journal daemon which does log rotation. To use rkt with a docker image see these docs:

https://coreos.com/rkt/docs/0.5.4/running-docker-images.html

Docker is improving its logging system but there is nothing to help in the current releases.

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sara Akgul]

So for the current alpha release, to configure logrotate, you'll need to add your custom config to the user-writable file located at /etc/logrotate.d/logrotate.conf... For any older releases, you need to do a bit more work (See more at the bottom.)

Open up this file, and add your config! These config files allow * as a wildcard character, which is nice for this use case. For example, below is a sample set of config options that will rotate each log up to 20 times before overwriting. A rotation will happen when each file reaches a size of 5 MB. Just add this to the file.

    /var/lib/docker/containers/*/*json.log {

    rotate 20
    maxsize 5M

    }

        (more  reading/options:

        http://stackoverflow.com/questions/20162176/centos-linux-setting-logrotate-to-maximum-file-size-for-all-logs

        http://www.linuxcommand.org/man_pages/logrotate8.html)

The second thing to keep in mind is that by default, logrotate only runs once a day. So you will need to take extra steps potentially to prevent disk space filling, depending on how fast your docker app is pumping out logs. These steps will be the same regardless of what version of CoreOS you are running.

CoreOS uses systemd timer units to schedule logrotate executions. To do this, copy the logrotate.timer unit file that systemd defaults to on startup, and place it in the user's units. logrotate default timing is here:

/usr/lib/systemd/system/logrotate.timer

Copy whats in there to here:

/etc/systemd/system/logrotate.timer

And modify the file to desired frequency. This will override the system defaults.

For example, your new custom file can look like this:

    [Unit]
    Description=Docker App Log Rotation

    [Timer]
    OnCalendar=30m
    AccuracySec=1m
    Persistent=true

See systemd man pages for more information about unit/timer files.


http://jason.the-graham.com/2013/03/06/how-to-use-systemd-timers/
https://wiki.archlinux.org/index.php/Systemd/Timers#Caveats

-------------------------

For our current beta and stable releases, you'll need to use a work around to get logrotate working. Adding a snippet config to logrotate.d doesn't work, and you have to do a full copy of the system's config in addition to the user-customized portion, in addition to other things.

First, do the following to add a user-defined logrotate unit:

sudo cp /usr/lib/systemd/system/logrotate.service /etc/systemd/system/logrotate.service

Open up the file at the new location and modify the line "ExecStart=/usr/sbin/logrotate /usr/share/logrotate/logrotate.conf" to instead say "ExecStart=/usr/sbin/logrotate /etc/logrotate.d/logrotate.conf"

This is to point logrotate to a configuration file that user writable, since the /usr directory on CoreOS is read-only.

Now, you'll be able to store custom user-defined configurations for logrotate to use here, but the difference here is you need to include the system default in this file aswell. Copy the default logrotate config file over to /etc/ aswell:

/usr/share/system/logrotate.conf to /etc/logrotate.d/logrotate.conf

and open up the file at the new location, appending the custom settings to the bottom of the config. If you need to adjust the frequency of how often the log is checked, follow the steps for modifying the timer unit, similar to above.

Hope this helps! :)

--

[poiuytrez]

Hello Sara, 

Thank you for your in-depth answer! I have seen that docker 1.8 was released a few days ago with the support of a log rotation:

https://docs.docker.com/reference/logging/overview/

I will try this option and keep you updated. 

poiuytrez