Docker with native CRIU

[Zhao Junping]

Hi,

I'm trying to enable Docker with native CRIU (that's docker checkpoint/restore) on running container. however, ckpt failed as "sys/fs/cgroup/devices doesn't have a proper root mount". 
Anybody hit the same issue? any known solution for that? anybody enables native CRIU with Docker 1.7?

I co Docker (1.8?) at: https://github.com/boucher/docker/tree/cr-combined/   sounds it's based on runC/libcontainer

thanks.

here're the "mount" within container.
none on / type aufs (rw,relatime,si=6da60270486a9881,dio,dirperm1)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)

[Vishnu Kannan]

+Saied

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Saied Kazemi]

+Ross

I think Ross Boucher's Docker is currently blocked on a pull request, which may be why it's failing.

If Docker 1.5 is not too old for you, you can try https://github.com/SaiedKazemi/docker/releases in the meantime.

--Saied

[Zhao Junping]

thank you. I'll try it. Saied.

and Boucher replied me and confirmed it's a known issue.
"There's a fix in libcontainer, but that fix has not yet made its way into docker. Once that happens, things should work again. "

在 2015年7月30日星期四 UTC+8上午12:56:04，Saied Kazemi写道：

[Zhao Junping]

Saied, wonderful! everything works well in your branch. thanks!

在 2015年7月31日星期五 UTC+8上午11:22:39，Zhao Junping写道：

[Zhao Junping]

Saied.
A question about cross-host ckpt-restore/resume.

I'm trying to ckpt a contaienr at host1, then resume the state at a diff host2 (running same base image, docker version) by copying the criu_img. but sounds current implementation still can't do that, right? there're some coupling on container IDs etc? I'm wondering how the live mobility demo in DockerConf15 was achieved :(.

Looks to me (may be wrong) currently Docker with CRIU shall issue ckpt-restore in pair, and must perform on the same host ?
I used to working on enterprise FS and data protection, comparing to mature FS/DB, Docker lacks of interfaces that:
1) start a new container from a point-in-time ckpt (i.e, taken at host1). something like DB's WAL log: launch, sync disk state at first(assume some Rep here), then replay mem state until ckpt by CRIU.
2) Restore a running container from a ckpt: internally the container shall be paused at first, then replaced the state from ckpt.  

I tried this way using your branch, hit error, still not check source code yet.

host1 docker1.5: run image, ckpt, -> gzip ckpt files ckpt1
host2 docker1.5: run same image, ckpt at first but replace criu_img/ by host1.ckpt1, then try to restore it, failed

INFO[0511] POST /v1.18/containers/9957caead3c6/restore
INFO[0511] +job restore(9957caead3c6)
INFO[0511] >>> github.com/docker/docker/daemon.(*Container).AllocateNetwork(): restoring from checkpoint IPAddress=172.17.0.2 MacAddress=02:42:ac:11:00:02
INFO[0511] +job allocate_interface(9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45)
INFO[0511] >>> github.com/docker/docker/daemon/networkdriver/ipallocator.(*allocatedMap).checkIP(): using already allocated ip 172.17.0.2
INFO[0511] >>> github.com/docker/docker/daemon/networkdriver/bridge.Allocate(): using requested MAC address: 02:42:ac:11:00:02
INFO[0511] -job allocate_interface(9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45) = OK (0)
INFO[0511] +job log(restore, 9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45, ubuntu:14.04)
INFO[0511] -job log(restore, 9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45, ubuntu:14.04) = OK (0)
INFO[0511] >>> github.com/docker/libcontainer/namespaces.updateState(): dataPath=/var/lib/docker/execdriver/native/9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45
INFO[0511] >>> github.com/docker/docker/daemon/execdriver/native.func·005(): restorePid=0 <<======
INFO[0511] +job log(die, 9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45, ubuntu:14.04)
INFO[0511] -job log(die, 9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45, ubuntu:14.04) = OK (0)
Cannot restore container 9957caead3c6: restore process failed <<=======
INFO[0511] -job restore(9957caead3c6) = ERR (1)
ERRO[0511] Handler for POST /containers/{name:.*}/restore returned error: Cannot restore container 9957caead3c6: restore process failed
ERRO[0511] HTTP Error: statusCode=500 Cannot restore container 9957caead3c6: restore process failed
ERRO[0511] Error restoring container: no child processes, exitCode=0
INFO[0511] >>> github.com/docker/docker/daemon.(*Container).cleanup(): not calling ReleaseNetwork() for checkpointed container 9957caead3c61cd47b3f03bd2b9941d288113c288a18128846c3a736406edf45



在 2015年7月31日星期五 UTC+8下午3:41:14，Zhao Junping写道：

[Saied Kazemi]

Zhao,

We have not started adding container migration support to Docker yet.  That effort will have to wait until native checkpoint restore is fully merged into Docker.

The Quake demo at DockerCon used runC to create a container, checkpoint, move images, and restore on a different machine (http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-dockercon-work.html).  Docker wasn't involved at all.

Finally, please note that restoring a container from its checkpoint image does not imply any changes to the underlying filesystem -- it restores the process tree.  In this sense, it's different from traditional filesystem and database checkpoint and replay.

--Saied