CoreOS automatically updated and now docker run doesn't work?

[Cory Keane]

Hello.

I am experiencing an issue whenever trying to use the "docker run" command on one of my Fleet CoreOS machines. It is only happening to this specific CoreOS machine in the cluster, no other ones. The only difference between this CoreOS machine and the others is that it seems to have automatically updated its CoreOS version. I am thinking this might be related.

The issue is: Whenever I try and use the "docker run" command, no matter how I format it (I've tried everything, and am positive that I am not using the command incorrectly), the command fails and I get the following message "docker: Error response from daemon: Error relabeling upper directory: operation not supported.".

I cannot find anything on the Internet with this exact error message. I was thinking I could try and revert the CoreOS version for this machine, but no matter which release channel I try (stable, beta, or alpha), I still get the same issue. Is there a way to revert to a specific version of CoreOS? All my other CoreOS machines are on a version below the current stable release. Ideally, I'd like to figure out the issue and be able to use this current stable release for all our CoreOS machines.

Thanks!

[Brandon Philips]

Hello Cory-

What does `journalctl -u docker` say?

My hunch is that `/var/lib/docker` got corrupted in some way on this host. There are several issues over several releases of Docker that can cause this to happen and the issues are itermittent or racy. If you don't care about any of the volumes or data in your containers I would suggest stopping docker (`systemctl stop docker.service docker.socket`), remove all of the files (`rm -Rf /var/lib/docker/*`), and restart the machine.

Hope that helps.

Thank You,

Brandon

[Cory Keane]

Hi Brandon. Luckily, I do not need to retain any data in the docker containers, so I ran the commands for stopping the docker service and socket, and removed the files in /var/lib/docker, on the faulty machine. I then restarted the machine, and things seem to work again.

For the sake of helping fix this issue, since it was very frustrating, can you let me know what I am looking for in the journal? I ran `fournalctl -u docker` and there is a LOT of results.

Thanks very much for your help!!

[Cory Keane]

Brandon, I apologize. It appears that this worked for one of the machines experiencing this issue, but not the other machine. Since I posted this, another one of our fleet machines auto updated to what appears to be this faulty version.

So please let me know what I am looking for specifically when running the `journalctl -u docker` command.

[Brandon Philips]

Hey Cory-

It is really hard to know what to look for in the logs. If you don't want to share the logs publicly you can email them to me personally and I will take a look.

Cheers,

Brandon

[Cory Keane]

No worries Brandon. I will upload them. But what's the easiest way to export them? The command you told me to run makes it so I have to press the space bar to cycle through all the lines of the file.

Thanks so much.

--

Cory Keane

Co-Founder & CTO

ThreadMeUp.com

[Brandon Philips]

journalctl --no-pager -u docker.service > foo.log

I think it turns on no-pager by default if it knows stdout isn't a tty

[Brandon Philips]

Hello Cory-

There are a number of errors about the mounts which should be non-fatal: 

Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: time="2016-08-09T15:40:08.520711349Z" level=error msg="Clean up Error! Cannot destroy container 26537dcd69eeb25dcd3a4867f81db909e5ca992efa1e6861d9e804770ec11239: rmdriverfs: Driver overlay failed to remove root filesystem 26537dcd69eeb25dcd3a4867f81db909e5ca992efa1e6861d9e804770ec11239: mount still active"

Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: time="2016-08-09T15:40:08.520749851Z" level=error msg="Handler for POST /v1.22/containers/create returned error: Error relabeling upper directory: operation not supported"

Could you grab the output of `mount` from these machines?

And it doesn't look like the daemon dies for several minutes after:

Aug 09 15:43:41 ip-172-31-16-118.ec2.internal systemd[1]: Stopping Docker Application Container Engine...

I really don't know what is getting messed up to cause this. What are the arguments to docker run?

It looks sort of a like a repeat of https://github.com/coreos/bugs/issues/1301 which was about SELinux and OverlayFS so I am going to cc in Nick, Matthew, and Crawford.

Cheers,

Brandon

On Tue, Aug 9, 2016 at 10:41 AM Cory Keane <co...@threadmeup.com> wrote:

Hey Brandon,

I decided to just email the logs directly to you.

Thanks!

[Brandon Philips]

Hello Cory-

Also, what versions of CoreOS is this happening under? `cat /etc/os-release`.

Brandon

[Cory Keane]

The docker command I try running takes the format: `docker run --rm --name=blahblahblah -e APP_ENV=production quay.io/path/tocontainer:stable php aphpfile.php`

Below you will find the following:

1. CoreOS Info

2. Output from running `mount`

1. CoreOS Info:

NAME=CoreOS

ID=coreos

VERSION=1081.5.0

VERSION_ID=1081.5.0

BUILD_ID=2016-07-16-2255

PRETTY_NAME="CoreOS 1081.5.0 (MoreOS)"

ANSI_COLOR="1;32"

HOME_URL="https://coreos.com/"

BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Output from running `mount`:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

devtmpfs on /dev type devtmpfs (rw,nosuid,size=2008876k,nr_inodes=502219,mode=755)

securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)

cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)

pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)

cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)

cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)

cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)

cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)

cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)

cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)

cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)

cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)

cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)

/dev/xvda9 on / type ext4 (rw,relatime,data=ordered)

/dev/xvda3 on /usr type ext4 (ro,relatime,block_validity,delalloc,barrier,user_xattr,acl)

selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)

systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11137)

tmpfs on /tmp type tmpfs (rw)

tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime)

hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)

debugfs on /sys/kernel/debug type debugfs (rw,relatime)

xenfs on /proc/xen type xenfs (rw,relatime)

systemd-1 on /boot type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11461)

mqueue on /dev/mqueue type mqueue (rw,relatime)

/dev/xvda6 on /usr/share/oem type ext4 (rw,nodev,relatime,commit=600,data=ordered)

/dev/xvda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

tmpfs on /run/user/500 type tmpfs (rw,nosuid,nodev,relatime,size=404996k,mode=700,uid=500,gid=500)

Thank you,

Cory

[Cory Keane]

Hi guys,

I was wondering if you have any updates on this. Thanks for your help.

[Cory Keane]

Hey guys,

Is this an open issue?

[Kyle Brown]

Hello Cory,

I can't find any issues relating specifically to the "docker: Error response from daemon: Error relabeling upper directory: operation not supported." error. If you could update to the latest beta and see if this error continues before submitting an issue to coreos/bugs that would be ideal.

Cheers,

Kyle Brown