Docker image hard links

[Joe Kemp]

I'm having an issue that I think is somewhat specific to the Prometheus and Alertmanager docker images.  I am running these in Kubernetes v1.4.6.  After several deployments of these images to a node I get the "too many links" error when docker attempts to pull the image.  I'm not running coreos but here is a description of the issue.  For instance, after just a few deployments the files in /bin have 33,210 hard links (output of "find /var/lib/docker/overlay/ -links +30000 -printf '%p %n\n'" /var/lib/docker/overlay/de114d505616ce62e3ef6e1fd560949cd3c303322a48872bfcd0f0606f76a66c/root/bin/cut 33210).  Ext4 hard link limit is 65,000.

I'm not sure what is going on with these images.  I run a few other ubuntu and alpine images without this issue.  I noticed the base prom/busybox image is built with Docker 1.9.  Any chance to start using 1.10 to build the image?  Any other clues what might be causing this?

Thanks,

-Joe

[Joe Kemp]

Figured out a little more info.  If you do a "ls -l /bin | wc" you will see there are 369 files.  33210 / 369 = 90.  I think the link count explosion is due to the way the overlay storage driver handles all of the layers in the images.  I checked a different Alpine image that uses busy box and it uses symbolic links for everything in /bin to /bin/busybox instead of the hardlinks used in the prometheus image.  This issue can only be observed if running with the Overlay storage drive.  Aufs and Overlay2 would not have this issue (but I'm stuck with Overlay for now).