Cross-Host Checkpoint Incompatibility Taxonomy

[David Hwang]

Hi, my team recently learned about CPU instruction set differences leading to cross-host checkpoint incompatibility.

We were interested in learning about other categories of incompatibility that can arise when performing a cross-host checkpoint->restore. 

Our use case: we host a platform to execute untrusted code & checkpoint/restore is a core feature that we provide. However, supporting this feature can potentially couple us the machine types that we use that can be quite restrictive.

[Ayush Ranjan]

Hi David,

Yes, checkpoint/restore is currently not possible with runsc across hosts with different CPU features. However, this can be mitigated. We'd need to implement CPU feature leveling, wherein users can specify a base CPU FeatureSet to use (before checkpoint) and the sandbox would only use those even if the underlying CPU supports more features.

And at restore, runsc would check if the host supports this base CPU FeatureSet. Users can define this base CPU FeatureSet as the intersection of CPU features available across their hosts. This is not implemented yet, but contributions towards this are most welcome!

There are other factors that must remain the same between checkpoint restore:

• Parts of the OCI container spec. We have added spec validation logic which reports and discrepancies in the OCI specs used.
• Runsc version.
• Container filesystem. If you are using default runsc flags, (and not overriding the --overlay2 flag), then your container rootfs will have an overlay. Any changes to the rootfs are stored in gVisor sandbox and are part of checkpoint/restore. However, the container base image (rootfs lower layer) and any bind mounts provided must be the same between checkpoint restore. For the base image, make sure you have pulled the same version of the image at the path specified by spec.Root.Path.

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gvisor-users/62225cc7-70fa-48f7-9d53-8256b27a574dn%40googlegroups.com.

--

Ayush Ranjan