docker hogging huge space from /var/lib/docker/aufs/mnt

[sagar...@gmail.com]

Hello,

We've been using docker instances while running test cases on out Jenkins slaves.

We take care of docker cleanup (docker rm -vf ...) after every test-case.

Yet after all the test cases are run, we notice that there are some stale docker containers hanging.

But there's a bigger problem.

We see that /var/lib/docker/aufs/mnt is full of many directories (which look like they correspond to stale containers)

It's taking up 500GB of space!!! Looks like it's been building cumulatively overtime.

"docker ps -a" does not show these container-ids

Worst is that we are unable to delete this too.

I stopped the docker service & tried "rm -rf /var/lib/docker/aufs/mnt/*"

It says "Operation not permitted" though I'm the root.

How do I clean this space up? "docker rm" for these container ids say "container doesn't exist"

Any help is appreciated.

PS: Our docker version is 1.9.1

~Sagar

[Adam McMaster]

It seems to keep old volumes around forever. We run this at the end of each build:

docker volume ls -q | xargs --no-run-if-empty docker volume rm

It'll refuse to remove any volumes which are still in use so it seems to be safe to just run that whenever.

Adam.

> --
> You received this message because you are subscribed to the Google Groups "docker-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[sagar...@gmail.com]

Thanks for this information Adam,

Somehow in our case this command doesn't work.

docker volume ls -q 

returns empty!

~Sagar

=============================================================

[Jaron Kennel]

We ran into a similar problem and ended up writing a job that completely deletes the docker directory on each Jenkins worker.  I have the scripts here: https://gist.github.com/jaronkk/22038a917cbb9a1c4a20e5628396182e

To start off with I would recommend running the commands in cleanup_docker.sh on one of your slaves to make sure it works correctly. Then you can customize the cleanup_docker_on_all_workers.sh to fit your Jenkins setup and schedule either via a cron job or a Jenkins job that is pinned to the master.

[Michael Crosby]

Well the good news is that the mnt dir is not using up 500GB.  This is the mount dir for container filesystems, no data is stored in that dir, it only holds mountpoints.  

Lets say you have 1 image that is 500mb on your system.  Now you create 1000 containers from that image.  If you where to du . -sh the mnt dir it will say that you are using 500GB but that is not the case.  Its just because the overlay filesystems are mounted there but they are sharing the same underlying data.

The real issue is that these mountpoints are not being cleaned up.  You should reboot and that will clean them up but you need to look into your jenkins scripts to see if something is hanging onto references after the container has exited. 

A good place to start debugging is to reboot to start fresh, run your build/operations and see if it leaves the mount point around.  If it does, you can use something like lsof to see if any other processes are hanging onto open fds in that directory.  It will help give us a better idea about what is going on.  Also the EPERM is weird when you try to remove.  It could be an bug in the filesytem.  What distro and distro version are you running?  What is your kernel version? 

Help this helps.

--

-----------------------------------------

Michael Crosby

@crosbymichael

[sagar...@gmail.com]

Thanks Jaron & Michael for these pointers.

We did find one process holding onto the mount-points.

So we ended up killing all instances on this process which allowed us to umount & clear up these volumes.

PS: We are using Ubuntu 14.04.

Regards

~Sagar

=====================================================

[Mads]

Could you elaborate on this? I am facing the same issues

Due to my Docker ignorance, I am afraid to delete the directory as I have a database running inside a container