Docker 1.11 issues with storage

[Fabian Faßbender]

Hello,

i have seen some issues with Docker in 1122.0.0 regarding the deletion of containers. Docker itself claims that it can't remove the root filesystem for the container. But when i look it up the root filesystem mount for the container is gone. 

It appears to occur once we restart a container twice using this fleet Unit:

[Unit]
Description=GommeHD PE QSG Server
Requires=docker.service stocker-plugin.service
After=docker.service stocker-plugin.service


[Service]
EnvironmentFile=/etc/environment
Restart=no
ExecStartPre=-/usr/bin/docker kill QSG%i
ExecStartPre=-/usr/bin/docker rm  QSG%i
ExecStartPre=-/usr/bin/docker network disconnect -f gommehdnet-pe QSG%i
ExecStartPre=/usr/bin/docker pull quay.io/gommehdnet/pe-qsg:latest
ExecStart=/usr/bin/docker run --volume-driver=stocker  -v pe_pelib_locales:/nukkit/server/plugins/PELib/locales/ -v pe_qsg_maps:/nukkit/server/plugins/SurvivalGames/maps/ -v pe_qsg_chesttypes:/nukkit/server/plugins/SurvivalGames/chestTypes/ -v pe_qsg_dmaps:/nukkit/server/plugins/SurvivalGames/dmaps/ -v pe_qsg_lobby:/nukkit/worlds/world/ -v pe_qsg_locales:/nukkit/server/plugins/SurvivalGames/locales/ --name QSG%i -h QSG%i --net=gommehdnet-pe quay.io/gommehdnet/pe-qsg:latest
ExecStop=/usr/bin/docker stop QSG%i
ExecStopPost=/usr/bin/docker rm -v QSG%i

Here is a snippet of the journalctl at that time (with the QSG0 container restarted 2 times):

time="2016-08-18T20:33:55.677062471Z" level=error msg="Handler for POST /v1.23/containers/QSG0/kill returned error: Cannot kill container QSG0: No such container: QSG0"

Error response from daemon: Cannot kill container QSG0: No such container: QSG0

time="2016-08-18T20:33:55.802616385Z" level=error msg="Handler for DELETE /v1.23/containers/QSG0 returned error: No such container: QSG0"

Error response from daemon: No such container: QSG0

time="2016-08-18T20:33:55.823764316Z" level=error msg="Handler for POST /v1.23/networks/gommehdnet-pe/disconnect returned error: endpoint QSG0 not found"

Error response from daemon: endpoint QSG0 not found

> ServerAPI - Hostname: QSG0

new log file: 2016_08_18_20_44_QSG0.log, APP:PEQSG, hash:448

time="2016-08-18T20:44:04.187635275Z" level=error msg="Handler for POST /v1.23/containers/QSG0/stop returned error: Container QSG0 is already stopped"

QSG0

QSG0

time="2016-08-18T20:44:15.690252849Z" level=error msg="Handler for POST /v1.23/containers/QSG0/kill returned error: Cannot kill container QSG0: No such container: QSG0"

Error response from daemon: Cannot kill container QSG0: No such container: QSG0

time="2016-08-18T20:44:15.799774636Z" level=error msg="Handler for DELETE /v1.23/containers/QSG0 returned error: No such container: QSG0"

Error response from daemon: No such container: QSG0

time="2016-08-18T20:44:15.820146156Z" level=error msg="Handler for POST /v1.23/networks/gommehdnet-pe/disconnect returned error: endpoint QSG0 not found"

Error response from daemon: endpoint QSG0 not found

> ServerAPI - Hostname: QSG0

new log file: 2016_08_18_20_54_QSG0.log, APP:PEQSG, hash:6026

time="2016-08-18T20:54:25.392488729Z" level=error msg="Handler for POST /v1.23/containers/QSG0/stop returned error: Container QSG0 is already stopped"

QSG0

QSG0

time="2016-08-18T21:25:05.759358176Z" level=error msg="Handler for POST /v1.23/containers/QSG0/kill returned error: Cannot kill container QSG0: No such container: QSG0"

Error response from daemon: Cannot kill container QSG0: No such container: QSG0

time="2016-08-18T21:25:05.770851291Z" level=error msg="Handler for DELETE /v1.23/containers/QSG0 returned error: No such container: QSG0"

Error response from daemon: No such container: QSG0

time="2016-08-18T21:25:05.884301132Z" level=error msg="Handler for POST /v1.23/networks/gommehdnet-pe/disconnect returned error: endpoint QSG0 not found"

Error response from daemon: endpoint QSG0 not found

time="2016-08-18T21:42:15.805068557Z" level=error msg="Handler for POST /v1.23/containers/QSG0/kill returned error: Cannot kill container QSG0: No such container: QSG0"

Error response from daemon: Cannot kill container QSG0: No such container: QSG0

time="2016-08-18T21:42:15.822311913Z" level=error msg="Handler for DELETE /v1.23/containers/QSG0 returned error: No such container: QSG0"

Error response from daemon: No such container: QSG0

time="2016-08-18T21:42:15.938872353Z" level=error msg="Handler for POST /v1.23/networks/gommehdnet-pe/disconnect returned error: endpoint QSG0 not found"

Error response from daemon: endpoint QSG0 not found

time="2016-08-18T21:44:25.737257735Z" level=error msg="Handler for POST /v1.23/containers/QSG0/stop returned error: Container QSG0 is already stopped"

QSG0

time="2016-08-18T21:44:35.801222806Z" level=error msg="Handler for POST /v1.23/containers/QSG0/kill returned error: Cannot kill container QSG0: Container 3a65387b66fccc3141421e1392000f4735eb5c4ca49eb9732fdd17f4be6464d1 is not running"

Error response from daemon: Cannot kill container QSG0: Container 3a65387b66fccc3141421e1392000f4735eb5c4ca49eb9732fdd17f4be6464d1 is not running

time="2016-08-18T21:44:35.919643287Z" level=error msg="Handler for DELETE /v1.23/containers/QSG0 returned error: Driver overlay failed to remove root filesystem 3a65387b66fccc3141421e1392000f4735eb5c4ca49eb9732fdd17f4be6464d1: mount still active"

time="2016-08-18T21:44:37.852981038Z" level=error msg="Handler for POST /v1.23/containers/create returned error: Conflict. The name \"/QSG0\" is already in use by container 3a65387b66fccc3141421e1392000f4735eb5c4ca49eb9732fdd17f4be6464d1. You have to remove (or rename) that container to be able to reuse that name."

I updated to 1122.0.0 to see if this issue goes away. But i also had the issue in stable 1068. Also tried using BRTFS but same result. Do i do something wrong so Docker does not delete the containers correct?

Thanks for help

Fabian Faßbender