Checkpoint/Restore features with containerd or docker

[黄良]

When I am reading docs of gvisor about checkpoint/restore, it talks about using docker. But as far as I know, the checkpoint implementation of docker is handled by containerd. And the containerd relies on the shim itself to implement the Checkpoint() TaskService interface, and for now the Checkpoint() in containerd is almost only compatible with runc and CRIU.
When I am reading the source code of gvisor, I find that it does not implement Checkpoint() at all in its shim.

1. So my first question is "Does Checkpoint/Restore in gvisor can be used in docker? If it cannot, why the documentation mentioned about docker? I think it is better to tell the users that C/R  of gvisor is not supported  in docker instead of show the shell command in docs?"
2. My second question is what it the right way to use Checkpoint/Restore features in gvisor? Since it is not compatible with containerd, and as far as I know, the only way to use it is executing the runsc binary directly using command like "runsc checkpoint ...", which is inconvenient. So i wonder is there an integration of Checkpoint/Restore in gvisor with existing container management tools like containerd or probably k8s (I am not familiar with k8s)?

[Ayush Ranjan]

Hi,

IIUC, containerd delegates the checkpoint/restore work to the underlying container runtime (like runc, runsc, etc) via commands like `checkpoint` and `restore`. runc implements these commands; see:

1. https://github.com/opencontainers/runc/blob/main/checkpoint.go

2. https://github.com/opencontainers/runc/blob/main/restore.go

gVisor (runsc) similarly implements these commands:

1. https://github.com/google/gvisor/blob/master/runsc/cmd/checkpoint.go

2. https://github.com/google/gvisor/blob/master/runsc/cmd/restore.go

AFAICT, this is the interface we need to implement. And gVisor checkpoint restore does work with Docker; here is a demo:

$ docker run --runtime=runsc --name=demo -d ubuntu sleep infinity

dcba7ead268c6ae1b86144165c39772cfeb82199c18c59c54fb9469564322f46

$ docker exec -it demo bash

root@dcba7ead268c:/# echo "Hello World!" | tee /tmp/foo

Hello World!

root@dcba7ead268c:/# exit

exit

$ docker checkpoint create demo ckpt1

ckpt1

$ docker start --checkpoint ckpt1 demo

$ docker exec -it demo bash

root@dcba7ead268c:/# cat /tmp/foo 

Hello World!

root@dcba7ead268c:/# exit

exit

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/a7db2950-750e-4417-9373-dca381b9658dn%40googlegroups.com.

--

Ayush Ranjan

[Vseslav Kasatskii]

Hey everyone, 

> IIUC, containerd delegates the checkpoint/restore work to the underlying container runtime (like runc, runsc, etc) via commands like `checkpoint` and `restore`

Just to build on the discussion above, when using gVisor, `containerd` actually communicates through gvisor-containerd-shim, which doesn’t appear to implement checkpointing (see this line in the code).

I ran into this when trying to checkpoint a pod in a Kubernetes cluster with gVisor: 
E0612 11:37:47.274108 495270 remote_runtime.go:1214] "CheckpointContainer from runtime service failed" err="rpc error: code = Unimplemented desc = method CheckpointContainer not implemented" containerID="test"

I also opened an issue about it here: https://github.com/google/gvisor/issues/11810.
One thing I’m still unclear about: Docker supports checkpointing with `runsc` as a runtime, but doesn’t it also go through `containerd` and its `shim`? How does that work in Docker’s case?

Thanks!

[Ayush Ranjan]

> One thing I’m still unclear about: Docker supports checkpointing with `runsc` as a runtime, but doesn’t it also go through `containerd` and its `shim`? How does that work in Docker’s case?

It seems like Docker runs the runsc sandbox against a runc shim. You can test this: run a sleep container with gVisor using Docker, you can observe the `containerd-shim-runc-v2` process in `ps aux` output. And runc shim implements the Checkpoint RPC. And it calls into runsc binary for invoking checkpoint.

This is OK because we don't need any of our special shim logic in Docker (single container use case). The runsc shim is mainly used in k8s environments.