docker hangs at commiting when no enough disk space

[yigong liu]

Hello,

Is this a known issue? "docker commit" hangs when system is short of disk space. From some initial debugging, it seems the process of "tar the container file-tree and untar it at destination tmp directory" will hang and not failed.

If it is unknown, should i add it the bug list?

I forked the code base at (https://github.com/yigongliu-docker/docker) and attempt to check file-tree size and available space at container.go::ExportRw() and bail out if not enough space available. Please have a look.

Thanks
Yigong

[Jérôme Petazzoni]

Interesting. Can you try to reproduce with e.g. mounting a fixed-size tmpfs on /var/lib/docker before starting the Docker daemon?

(That should be an easy way to simulate limited disk space.)

I can provide some help with the tmpfs setup if needed.

The goal would be to provide a very simple, repeatable way, to reproduce the issue.

Thanks!

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

@jpetazzo

Latest blog post: http://jpetazzo.github.io/2013/10/16/configure-docker-bridge-network/

[Jérôme Petazzoni]

(I'm re-adding docker-devel since we removed at some point, it seems :-))

On UNIX systems, some space is reserved for root (by default, it's 5%).

It means that when the disk is 95% full, "df" will report 100% usage, but if you do the math, you will realize that it amounts to 95% instead of 100%.

This reserved space cannot be used by normal users, but can be used by root.

To read more about this, see here:

http://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why

Would that explain the difference you're seeing?

On Thu, Oct 17, 2013 at 10:28 PM, yigong liu <yigongli...@gmail.com> wrote:

Hello,

Thanks for your information, i mounted the 10M ramdisk and pull a busybox image; run a container and did some update and try to commit it. It did come up with all the error messages we expect from comandline. Yes docker works fine!

After a second thought, it occured to me that maybe other system facilities behave abnormally when no space available, which could cause docker's response change? I have a ubuntu13.04 partition with a few GB left. So i copied several tgz files and easily run out of space. Strange things happened: "df ." shows 0 space left. I cannot create file and the error says that no space available. However docker daemon keeps running, i can run docker command commit image with NO errors i saw as in RAM disk case, then i can "docker run" this new images to create a container to do more things. It seems that docker is not influenced, Magic! It is the latest code i checked out from github last night.

I tried to take a snapshot of the screen but failed since no space is available. So i used my camera take a picture. Please see it attached.

I am totally puzzlled. I'll try my ubuntu installation in another partition tomorrow in case something wrong with my current partition.

Just a thought, should we add some space checking before commiting to avoid all these situations?

Thanks

Yigong

On Thu, Oct 17, 2013 at 4:45 PM, Jérôme Petazzoni <jerome.p...@dotcloud.com> wrote:

Sure!

The idea would be:

- stop docker

- mount -t tmpfs -o size=10M none /var/lib/docker

- start docker

This will mount a 10M ramdisk (initially empty) over /var/lib/docker. Docker should initialize it automatically when it starts.

Later, when you're done and want to recover the old content of /var/lib/docker, just stop docker and "umount /var/lib/docker".

Let me know if you need more details!

On Thu, Oct 17, 2013 at 2:48 PM, yigong liu <yigongli...@gmail.com> wrote:

Yes, mounting a fixed/small size tmpfs is a good idea. Please advise me how to do it.

Thanks

Yigong

[yigong liu]

Thanks for the info and link. It does explain what i found when i run test code tar.go (extracted from archive.go) and it behaved differently when using in "sudo" or as normal user.

Right now, docker depends on info piped back from "untar" child process to report failure, which means it will happen at very last point when disk did get full. Since we can find out the tree size and avail disk size before hand, should we check it before doing real commit work? I understand this could be a corner case, since most people build images from Dockerfile and each commit is a small "delta"?

Thanks

[Jérôme Petazzoni]

Good question!

On the one hand, it means that we have to "walk" the filesystem a first time to compute the diff size, and a second time when generating the tarball.

On the other hand, it shouldn't affect performance too much, because when we walk the filesystem the second time, all the data should be in cache (as a result of the first walk).

I think it's a good idea; but we still need to handle the "disk full" scenario, since the disk usage could change between the moment when we start the tar, and the moment when we finish it.

@jpetazzo

Latest blog post: http://jpetazzo.github.io/2013/10/20/secure-connection-docker-api/