[gentoo-user] Error while starting Docker daemon

[Hubert Hauser]

I am using Gentoo as Host OS for Docker containers. I have compiled
kernel using instructions on page
https://wiki.gentoo.org/wiki/Docker#Kernel and I have installed Docker
from Gentoo repository.

Host system informations:

    pecan@tux ~ $ uname -a
    Linux tux 4.12.12-gentoo #8 SMP Sat Oct 7 13:58:47 CEST 2017 x86_64
Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz GenuineIntel GNU/Linux

Docker version:

    pecan@tux ~ $ docker version
    Client:
     Version:      17.03.2-ce
     API version:  1.27
     Go version:   go1.9.1
     Git commit:   f5ec1e2
     Built:        Sat Oct  7 14:50:59 2017
     OS/Arch:      linux/amd64
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

Look at "Cannot connect to the Docker daemon at
unix:///var/run/docker.sock. Is the docker daemon running?". The same
message appears if I try get docker system-wide informations:

    pecan@tux ~ $ docker info
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

The same error appears if I try run the same command as sudo, so this
error applies to daemon. I tried to check if there a mistake in Docker
daemon privileges.

    pecan@tux ~ $ sudo docker info
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

Based on the message I am able to say that maybe Docker daemon not
running. I checked daemon status to make sure:

    pecan@tux ~ $ sudo service docker status
     * status: crashed

Docker daemon is crashed. To see the reason, I looked at the logs:

    pecan@tux ~ $ cat /var/log/docker.log
    time="2017-10-07T14:52:13.178261811+02:00" level=info
msg="libcontainerd: new containerd process, pid: 32311"
    time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph
migration to content-addressability took 0.00 seconds"
    time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your
kernel does not support cgroup blkio weight"
    time="2017-10-07T14:52:14.434423960+02:00" level=warning msg="Your
kernel does not support cgroup blkio weight_device"
    time="2017-10-07T14:52:14.434759986+02:00" level=info msg="Loading
containers: start."
    time="2017-10-07T14:52:14.437180876+02:00" level=info msg="Firewalld
running: false"
    Error starting daemon: Error initializing network controller: list
bridge addresses failed: no available network

Currently, that is a point in that I do not know what should I do to be
able run Docker daemon.

Useful informations:

- I am connected to OpenVPN through UDP.
- I have disabled iptables and ip6tables.
- I have set 8.8.8.8 and 8.8.4.4 DNS providers.
- I have running privoxy and tor daemons.
- I use OpenRC init system.

Can you help me?

[Mick]

It seems you have not yet started docker.

> The same error appears if I try run the same command as sudo, so this
> error applies to daemon. I tried to check if there a mistake in Docker
> daemon privileges.
>
> pecan@tux ~ $ sudo docker info
> Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
> Is the docker daemon running?
>
> Based on the message I am able to say that maybe Docker daemon not
> running. I checked daemon status to make sure:
>
> pecan@tux ~ $ sudo service docker status
> * status: crashed

Did you try starting it from the CLI? Any useful messages there?

> Docker daemon is crashed. To see the reason, I looked at the logs:
>
> pecan@tux ~ $ cat /var/log/docker.log
> time="2017-10-07T14:52:13.178261811+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 32311"
> time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph
> migration to content-addressability took 0.00 seconds"
> time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your
> kernel does not support cgroup blkio weight"

OK, start from checking your kernel has all the necessary modules compiled in,
rebuild it and reboot.

> time="2017-10-07T14:52:14.434423960+02:00" level=warning msg="Your
> kernel does not support cgroup blkio weight_device"
> time="2017-10-07T14:52:14.434759986+02:00" level=info msg="Loading
> containers: start."
> time="2017-10-07T14:52:14.437180876+02:00" level=info msg="Firewalld
> running: false"
> Error starting daemon: Error initializing network controller: list
> bridge addresses failed: no available network
>
> Currently, that is a point in that I do not know what should I do to be
> able run Docker daemon.
>
> Useful informations:
>
> - I am connected to OpenVPN through UDP.
> - I have disabled iptables and ip6tables.
> - I have set 8.8.8.8 and 8.8.4.4 DNS providers.
> - I have running privoxy and tor daemons.
> - I use OpenRC init system.
>
> Can you help me?

I don't use docker to know any operational peculiarities of it, but others
with more experience will hopefully chip in. From what I see above you need
to rebuild your kernel with the necessary modules, reboot and then try
starting docker if it hasn't started on its own.

HTH.
--
Regards,
Mick

[Mick]

On Sunday, 8 October 2017 12:20:00 BST Hubert Hauser wrote:

> I can't start it because failure occurs when starting Docker daemon.

>
> On 07/10/17 18:50, Mick wrote:
> >> The same error appears if I try run the same command as sudo, so this
> >> error applies to daemon. I tried to check if there a mistake in Docker
> >> daemon privileges.
> >>
> >> pecan@tux ~ $ sudo docker info
> >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
> >>
> >> Is the docker daemon running?
> >>
> >> Based on the message I am able to say that maybe Docker daemon not
> >>
> >> running. I checked daemon status to make sure:
> >> pecan@tux ~ $ sudo service docker status
> >>
> >> * status: crashed
> >
> > Did you try starting it from the CLI? Any useful messages there?
>

> Yes.
>
> The result of command sudo dockerd:
> https://paste.pound-python.org/show/AyjLQEBGABpjo57E9WMa/
> and of sudo dockerd -s overlay2:
> https://paste.pound-python.org/show/ulyqb7D363I51JqkvrEk/.
>
> You can also look at attachments dockerd.txt (sudo dockerd) and
> dockerd-overlay2.txt (sudo dockerd -s overlay2).
>
> I must run dockerd without connecting into VPN.
>
> I've reset iptables and ip6tables rules to defaults:
>
> pecan@tux ~ $ sudo iptables -L
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> target prot opt source destination
> DOCKER-ISOLATION all -- anywhere anywhere
> DOCKER all -- anywhere anywhere
> ACCEPT all -- anywhere anywhere ctstate
> RELATED,ESTABLISHED
> ACCEPT all -- anywhere anywhere
> ACCEPT all -- anywhere anywhere
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain DOCKER (1 references)
> target prot opt source destination
>
> Chain DOCKER-ISOLATION (1 references)
> target prot opt source destination
> RETURN all -- anywhere anywhere
>
> (tensorflow) pecan@tux ~ $ sudo ip6tables -L
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> Here's my Docker logs:
>
> (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker-err.log
> time="2017-10-07T14:54:24.561887891+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 994"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T14:56:44.682987498+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 4983"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T17:00:16.938724808+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 7254"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T17:03:58.151814419+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 7857"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T17:05:09.492506131+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 7964"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T17:06:27.126674008+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 8202"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T17:09:47.482580356+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 8730"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-07T21:16:26.088461863+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 6414"
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-08T10:20:56.011379547+02:00" level=debug msg="docker group
> found. gid: 987"
> time="2017-10-08T10:20:56.023468730+02:00" level=debug msg="Listener
> created for HTTP on unix (/var/run/docker.sock)"
> time="2017-10-08T10:20:56.023686388+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 26284"
> time="2017-10-08T10:20:56.030713286+02:00" level=debug msg="containerd:
> read past events" count=0
> time="2017-10-08T10:20:56.030808207+02:00" level=debug msg="containerd:
> supervisor running" cpus=4 memory=7849 runtime=docker-runc
> runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd"
> time="2017-10-08T10:20:56.030871453+02:00" level=debug msg="containerd:
> grpc api on /var/run/docker/libcontainerd/docker-containerd.sock"
> time="2017-10-08T10:20:56.524998596+02:00" level=debug
> msg="libcontainerd: containerd health check returned error: rpc error:
> code = 14 desc = grpc: the connection is unavailable"
> time="2017-10-08T10:20:57.035916104+02:00" level=debug msg="Using
> default logging driver json-file"
> time="2017-10-08T10:20:57.035960712+02:00" level=debug msg="Golang's
> threads limit set to 56430"
> time="2017-10-08T10:20:57.036012700+02:00" level=debug
> msg="[graphdriver] trying provided driver: btrfs"
> time="2017-10-08T10:20:57.036027034+02:00" level=debug msg="Cleaning up
> old mountid : start."
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> time="2017-10-08T10:21:05.597682241+02:00" level=debug msg="docker group
> found. gid: 987"
> time="2017-10-08T10:21:05.597724031+02:00" level=debug msg="Listener
> created for HTTP on unix (/var/run/docker.sock)"
> time="2017-10-08T10:21:05.597910790+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 26392"
> time="2017-10-08T10:21:05.606126357+02:00" level=debug msg="containerd:
> read past events" count=0
> time="2017-10-08T10:21:05.606219342+02:00" level=debug msg="containerd:
> supervisor running" cpus=4 memory=7849 runtime=docker-runc
> runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd"
> time="2017-10-08T10:21:05.606245984+02:00" level=debug msg="containerd:
> grpc api on /var/run/docker/libcontainerd/docker-containerd.sock"
> time="2017-10-08T10:21:06.114930279+02:00" level=debug
> msg="libcontainerd: containerd health check returned error: rpc error:
> code = 14 desc = grpc: the connection is unavailable"
> time="2017-10-08T10:21:06.614858989+02:00" level=debug
> msg="libcontainerd: containerd health check returned error: rpc error:
> code = 14 desc = grpc: the connection is unavailable"
> time="2017-10-08T10:21:06.645208467+02:00" level=debug msg="Using
> default logging driver json-file"
> time="2017-10-08T10:21:06.645236784+02:00" level=debug msg="Golang's
> threads limit set to 56430"
> time="2017-10-08T10:21:06.645311511+02:00" level=debug
> msg="[graphdriver] trying provided driver: btrfs"
> time="2017-10-08T10:21:06.645324876+02:00" level=debug msg="Cleaning up
> old mountid : start."
> Error starting daemon: error initializing graphdriver: prerequisites for
> driver not satisfied (wrong filesystem?)
> (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker.log

> time="2017-10-07T14:52:13.178261811+02:00" level=info
> msg="libcontainerd: new containerd process, pid: 32311"
> time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph
> migration to content-addressability took 0.00 seconds"
> time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your
> kernel does not support cgroup blkio weight"

> time="2017-10-07T14:52:14.434423960+02:00" level=warning msg="Your
> kernel does not support cgroup blkio weight_device"
> time="2017-10-07T14:52:14.434759986+02:00" level=info msg="Loading
> containers: start."
> time="2017-10-07T14:52:14.437180876+02:00" level=info msg="Firewalld
> running: false"
> Error starting daemon: Error initializing network controller: list
> bridge addresses failed: no available network

> (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker-out.log
> (tensorflow) pecan@tux ~ $
>
> Docker daemon options:
>
> (tensorflow) pecan@tux ~ $ sudo cat /etc/docker/daemon.json
> {
> "debug": true

> }
>
> On 07/10/17 18:50, Mick wrote:
> >> Docker daemon is crashed. To see the reason, I looked at the logs:
> >> pecan@tux ~ $ cat /var/log/docker.log
> >> time="2017-10-07T14:52:13.178261811+02:00" level=info
> >>
> >> msg="libcontainerd: new containerd process, pid: 32311"
> >>
> >> time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph
> >>
> >> migration to content-addressability took 0.00 seconds"
> >>
> >> time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your
> >>
> >> kernel does not support cgroup blkio weight"
> >
> > OK, start from checking your kernel has all the necessary modules compiled
> > in, rebuild it and reboot.
>

> Look here:
>
> pecan@tux ~ $ sudo ~/check-config.sh
> info: reading kernel config from /proc/config.gz ...
>
> Generally Necessary:
> - cgroup hierarchy: properly mounted [/sys/fs/cgroup]
> - CONFIG_NAMESPACES: enabled
> - CONFIG_NET_NS: enabled
> - CONFIG_PID_NS: enabled
> - CONFIG_IPC_NS: enabled
> - CONFIG_UTS_NS: enabled
> - CONFIG_CGROUPS: enabled
> - CONFIG_CGROUP_CPUACCT: enabled
> - CONFIG_CGROUP_DEVICE: enabled
> - CONFIG_CGROUP_FREEZER: enabled
> - CONFIG_CGROUP_SCHED: enabled
> - CONFIG_CPUSETS: enabled
> - CONFIG_MEMCG: enabled
> - CONFIG_KEYS: enabled
> - CONFIG_VETH: enabled
> - CONFIG_BRIDGE: enabled
> - CONFIG_BRIDGE_NETFILTER: enabled
> - CONFIG_NF_NAT_IPV4: enabled
> - CONFIG_IP_NF_FILTER: enabled
> - CONFIG_IP_NF_TARGET_MASQUERADE: enabled
> - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
> - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
> - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
> - CONFIG_IP_NF_NAT: enabled
> - CONFIG_NF_NAT: enabled
> - CONFIG_NF_NAT_NEEDED: enabled
> - CONFIG_POSIX_MQUEUE: enabled
>
> Optional Features:
> - CONFIG_USER_NS: enabled
> - CONFIG_SECCOMP: enabled
> - CONFIG_CGROUP_PIDS: enabled
> - CONFIG_MEMCG_SWAP: enabled
> - CONFIG_MEMCG_SWAP_ENABLED: enabled
> (cgroup swap accounting is currently enabled)
> - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
> - CONFIG_BLK_CGROUP: enabled
> - CONFIG_BLK_DEV_THROTTLING: enabled
> - CONFIG_IOSCHED_CFQ: enabled (as module)
> - CONFIG_CFQ_GROUP_IOSCHED: missing
> - CONFIG_CGROUP_PERF: enabled
> - CONFIG_CGROUP_HUGETLB: missing
> - CONFIG_NET_CLS_CGROUP: enabled
> - CONFIG_CGROUP_NET_PRIO: enabled
> - CONFIG_CFS_BANDWIDTH: enabled
> - CONFIG_FAIR_GROUP_SCHED: enabled
> - CONFIG_RT_GROUP_SCHED: enabled
> - CONFIG_IP_VS: enabled (as module)
> - CONFIG_IP_VS_NFCT: enabled
> - CONFIG_IP_VS_RR: enabled (as module)
> - CONFIG_EXT3_FS: enabled
> - CONFIG_EXT3_FS_XATTR: missing
> - CONFIG_EXT3_FS_POSIX_ACL: enabled
> - CONFIG_EXT3_FS_SECURITY: enabled
> (enable these ext3 configs if you are using ext3 as backing filesystem)
> - CONFIG_EXT4_FS: enabled
> - CONFIG_EXT4_FS_POSIX_ACL: enabled
> - CONFIG_EXT4_FS_SECURITY: enabled
> - Network Drivers:
> - "overlay":
> - CONFIG_VXLAN: enabled (as module)
> Optional (for encrypted networks):
> - CONFIG_CRYPTO: enabled
> - CONFIG_CRYPTO_AEAD: enabled
> - CONFIG_CRYPTO_GCM: enabled
> - CONFIG_CRYPTO_SEQIV: enabled
> - CONFIG_CRYPTO_GHASH: enabled
> - CONFIG_XFRM: enabled
> - CONFIG_XFRM_USER: enabled (as module)
> - CONFIG_XFRM_ALGO: enabled (as module)
> - CONFIG_INET_ESP: enabled (as module)
> - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
> - "ipvlan":
> - CONFIG_IPVLAN: enabled (as module)
> - "macvlan":
> - CONFIG_MACVLAN: enabled (as module)
> - CONFIG_DUMMY: enabled (as module)
> - "ftp,tftp client in container":
> - CONFIG_NF_NAT_FTP: enabled
> - CONFIG_NF_CONNTRACK_FTP: enabled
> - CONFIG_NF_NAT_TFTP: enabled
> - CONFIG_NF_CONNTRACK_TFTP: enabled
> - Storage Drivers:
> - "aufs":
> - CONFIG_AUFS_FS: missing
> - "btrfs":
> - CONFIG_BTRFS_FS: enabled
> - CONFIG_BTRFS_FS_POSIX_ACL: enabled
> - "devicemapper":
> - CONFIG_BLK_DEV_DM: enabled
> - CONFIG_DM_THIN_PROVISIONING: enabled
> - "overlay":
> - CONFIG_OVERLAY_FS: enabled
> - "zfs":
> - /dev/zfs: missing
> - zfs command: missing
> - zpool command: missing
>
> Limits:
> - /proc/sys/kernel/keys/root_maxkeys: 1000000
>
> My kernel config:
> https://paste.pound-python.org/show/bFHNp4jrHE8kKzPMtNGq/ (in attachment
> file kernel-config.txt, sudo zcat /proc/config.gz).
>
> --
> Best regards,
> Hubert Hauser.


I have not trimmed your info above, in case someone more knowledgeable in
running docker can advise.

From what I see above you are running btrfs. It may be worth compiling in
your kernel this module you have left out, because I've read somewhere it
prevents fs corruption (Rich seems to know a lot about BTRFS, so I leave it to
him to confirm its usefulness):

# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set


Two warnings in /var/log/docker-err.log attracted my attention:

"Your kernel does not support cgroup blkio weight"

Try settings fair queue scheduling for your storage and check it is being
used.

The second thing is caused by the filesystem:

... prerequisites for driver not satisfied (wrong filesystem?)


You could try starting docker with options '-d -g' which apparently works for
booting btrfs as advised here:

https://stackoverflow.com/questions/30669359/docker-with-btrfs-ubuntu

HTH.
--
Regards,
Mick

[Rich Freeman]

On Sun, Oct 8, 2017 at 11:10 AM, Mick <michael...@gmail.com> wrote:
>
> From what I see above you are running btrfs. It may be worth compiling in
> your kernel this module you have left out, because I've read somewhere it
> prevents fs corruption (Rich seems to know a lot about BTRFS, so I leave it to
> him to confirm its usefulness):
>
> # CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
>

I wouldn't look much further than the config docs on this one:

Btrfs with integrity check tool compiled in (DANGEROUS)
...Enabling this functionality is not intended for normal use. In
most cases, unless you are a btrfs developer who needs to verify the
integrity of (super)-block write requests during the run of a
regression test, say N.

It looks like it is intended only for regression testing. I didn't
dig too deep into the docs, but it probably turns on some assertions
and probably doesn't have much in the way of recovery if they are
triggered. Granted, if they trigger you probably have issues anyway,
but they might not be as severe as whatever happens when this is
turned on. Of course, in theory it shouldn't do anything other than
waste RAM/CPU, since assertions aren't supposed to be triggered.

So, no, you don't want this on a real system.

--
Rich

[Neil Bothwick]

On Sun, 08 Oct 2017 19:10:48 +0100, Mick wrote:

> From what I see above you are running btrfs. It may be worth compiling
> in your kernel this module you have left out, because I've read
> somewhere it prevents fs corruption (Rich seems to know a lot about
> BTRFS, so I leave it to him to confirm its usefulness):
>
> # CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set

That module is not intended for normal use, and is marked as DANGEROUS.
At best it will slow your system dramatically...


--
Neil Bothwick

I used to live in the real world, but I got evicted.

[Mick]

On Sunday, 8 October 2017 20:07:09 BST Neil Bothwick wrote:
> On Sun, 08 Oct 2017 19:10:48 +0100, Mick wrote:
> > From what I see above you are running btrfs. It may be worth compiling
> > in your kernel this module you have left out, because I've read
> > somewhere it prevents fs corruption (Rich seems to know a lot about
> > BTRFS, so I leave it to him to confirm its usefulness):
> >
> > # CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
>
> That module is not intended for normal use, and is marked as DANGEROUS.
> At best it will slow your system dramatically...

Thanks guys, and my apologies to the OP for the bum steer! :-)

--
Regards,
Mick