WG-Creation-Request: Checkpoint/Restore Working Group

[Adrian Reber]

What is the exact problem this group is trying to solve?

This working group aims to provide a central location for the community
to discuss the integration of Checkpoint/Restore functionality into
Kubernetes. This functionality has been discussed in the context of
Kubernetes for more than five years now. In the past, the main
discussions have happened in the SIG Node meetings, but with
Checkpoint/Restore introducing completely new concepts into Kubernetes,
it was not the right audience.

Especially right now, we feel we are reaching a point where we want to
have more detailed discussions about the architecture and the design of
this functionality, which are not related to the topics typically
discussed at SIG Node. We are already discussing the future of
Checkpoint/Restore in Kubernetes at conferences, in private email
threads, private Slack channels, in GitHub issues and pull requests, and
an increasing number of people propose good ideas on how to move
forward. Several people in the community have suggested creating a
working group, as it is difficult to learn about our plans, the existing
discussions, and contribute to the future of Checkpoint/Restore in
Kubernetes.

Over the past two years, the ability to transparently checkpoint and
restore GPU-accelerated workloads has led to an increased interest in
using this functionality with AI/ML workloads. Now that it is possible
to quickly switch between different inference workloads on a GPU or
migrate a training session to another node without restarting it, has
led to many new users and especially questions.

What is the artifact that this group will deliver, and to whom?

Design and implementation of Checkpoint/Restore functionality in
Kubernetes. In this early stage, we mainly want to offer a well-defined
location for the community to find information, ask questions, and
discuss the next steps of enabling Checkpoint/Restore in Kubernetes.

How does the group know when the problem-solving process is completed,
and it is time for the Working Group to dissolve?

At some point, we would like to enable native support for transparent
container Checkpoint/Restore integrated into Kubernetes that enables
the scheduler to preempt and migrate workloads from one node to
another. This functionality allows for an improved utilization of
resources by preserving the runtime state of preempted workloads. If
that is possible at some point in the future, the Working Group has
achieved the integration of Checkpoint/Restore into Kubernetes and can
be dissolved.

Who are all of the stakeholder SIGs involved in this problem this group
is trying to solve?

It all started with SIG Node, and currently, we have prepared a KEP
targeting SIG API Machinery. At some point in the future, we would also
like to talk with SIG CLI, SIG Scheduling, and SIG Autoscaling. One of
our goals is to be able to present long-term design documents and not
just one KEP after another.

What are the meeting mechanics (frequency, duration, roles)?

Every two weeks, 60 minutes.

Does the goal of the Working Group represent the needs of the project as
a whole, or is it focused on the interests of a narrow set of
contributors or companies?

Considering the questions and feedback we have received from the
community, this functionality is used to solve a wide range of
problems, from fault-tolerance to improved resource utilization and
accelerated start-up time of applications. Thus, the project is not
focused on the interests of a narrow set of contributors or companies.
Instead, contributors interested in the Checkpoint/Restore in
Kubernetes range from all over the industry and different research
groups.

Who will chair the group, and ensure it continues to meet these requirements?

Adrian Reber, Viktória Spišaková, Radostin Stoyanov

Is diversity well-represented in the Working Group?

We encourage inclusive participation and ensure that all members feel
empowered to contribute. We make a conscious effort to include members
from varied backgrounds across gender, race, geography, professional
experience, and continue to look for ways to include more voices from
underrepresented groups.

Initial list of interested persons

Radostin Stoyanov, University of Oxford
Lukas Hejtmanek, Masaryk University
Viktória Spišaková, Masaryk University
Adrian Reber, Red Hat

[Jay Pipes]

On Fri, Jun 27, 2025 at 10:02 AM 'Adrian Reber' via dev <d...@kubernetes.io> wrote:

<snip>

 

Does the goal of the Working Group represent the needs of the project as
a whole, or is it focused on the interests of a narrow set of
contributors or companies?

   Considering the questions and feedback we have received from the
   community, this functionality is used to solve a wide range of
   problems, from fault-tolerance to improved resource utilization and
   accelerated start-up time of applications. Thus, the project is not
   focused on the interests of a narrow set of contributors or companies.
   Instead, contributors interested in the Checkpoint/Restore in
   Kubernetes range from all over the industry and different research
   groups.

Hi Adrian,

When I originally read "Checkpoint/Restore", I associated this proposal to long-standing terminology in the storage and database technology spaces, not GPU or AI/ML-specific technology. How many of the contributors or companies in the working group are working on non-AI/ML, non-GPU/CUDA-specific-technology areas?

Best,

-jay 

[Tim St. Clair]

Any long-running batch process that stands the chance of getting
pre-empted will want to checkpoint to ensure it doesn't need to start
from scratch.
This core idea has been around since the early days of grid systems,
which used custom compilation or DMTCP
(https://dmtcp.sourceforge.io/).

So, anyone using k8s to execute long-running batch jobs will likely
want this feature.

-Tim

> --
> You received this message because you are subscribed to the Google Groups "dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
> To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAAE6tVZ1M9nmrN2%2BGJB04ewJngjioyhHmNqy29WYKZNJbGpgog%40mail.gmail.com.



--
Cheers,
Timothy St. Clair

“Do all the good you can. By all the means you can. In all the ways
you can. In all the places you can. At all the times you can. To all
the people you can. As long as ever you can.”

[Jay Pipes]

Thanks Tim,

That's why I was asking whether contributors or companies come from areas other than GPU/AI/ML technology spaces. For instance, does anyone hail from a VM live migration/memory quiescence background or even non-memory-based checkpoint/restore technology background (say, DRBD or more traditional persistent storage technology)? :)

I'm wondering if the "variety of interests" is really just the variety of interests in the area of enabling GPU-specific technology or whether this working group is broader than that.

Best,

-jay

[Ryan Phillips]

+1 on the Working Group. I would like to see an artifact from the WG with the security boundaries defined and reasoned about in the K8S context.

Regards,

Ryan

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/CAAE6tVYLd5EzdrVSZ6rBiZgiujdOpC1F0mVehoXhyYarMRG_vw%40mail.gmail.com.

[Adrian Reber]

It is definitely broader than GPU/AI/ML.

Your comment, however, was interesting to read. For me, working around
process checkpoint/restore for almost 15 years now, it never has
anything to with storage or database. So it seems to be dependent on
your background.

Since 2022 Kubernetes has support to checkpoint and restore containers.
Either for the official forensic use case (just checkpointing) or also
to migrate containers:

https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/

As mentioned in our proposal it has been a topic discussed with SIG Node
but much has happened in personal conversations. When trying to extend
checkpoint/restore in Kubernetes (see
https://github.com/kubernetes/enhancements/issues/5091 and also
https://github.com/kubernetes/enhancements/pull/5092 for detailed list
of possible use cases) it became clear that we need to talk with a
larger group of people and have more public conversations. That is one
of the reasons we would like to see these public conversations
formalized by having a working group.

The GPU/AI/ML use case has just put more pressure on us to move forward
because it can help with better utilization of expensive resources. From
our point of view the GPU use case is just an additional use case, even
if it is a use case many people are very interested in.

Adrian

[Jay Pipes]

Thanks very much, Adrian, appreciate your response!

[jerry zhuang]

+1 on the Working Group. It's exciting to see Checkpoint/Restore being recognized as a critical capability for improving workload flexibility and efficiency in Kubernetes, especially with the rise of AI/ML and GPU-accelerated applications. I fully support the formation of this working group and the effort to coordinate community discussions around it.

I'd also like to share a project we’ve been working on:  GRIT: GPU workload checkpointing and restoration  GRIT is a prototype designed to automate GPU workload migration within a Kubernetes cluster. It leverages transparent checkpoint/restore and distributes checkpoint data using custom Persistent Volumes (PVs), which provides more flexibility and efficiency compared to OCI-based image checkpoints.

We hope this can be a useful input for the working group's discussions, especially around design choices for workload migration mechanisms. We're also looking forward to any feedback from the community on how we can align better or contribute further to the broader efforts in this space.

Thanks again for pushing this forward!

Regards,
Qinghui Zhuang

[Andrei Vagin]

Hi Jay,

As part of the gVisor team at Google, we're very interested in this
group too. We focus on sandboxing a diverse range of workloads, from
AI/ML applications to general computing tasks. Historically, gVisor's
Checkpoint/Restore (C/R) feature was primarily used for suspending
low-priority tasks, but there's a growing recognition of its value in
significantly improving application startup times.

Thanks,
Andrei

[Niranjan Ravichandra]

Also interested! For extra context, I'm one of the founders at Cedana (cedana.com), where we built a custom Kubernetes solution for checkpoint/restore and live migration. We've also built our homegrown GPU checkpoint/restore, which has a couple extra benefits on top of the existing CRIU NVIDIA plugin.

Hoping to add some more input to make Kubernetes more amenable to checkpoint/restore! We've had to do a lot by hand.

Regards,

[Pengzhan Hao]

+1 for the WG; I am very interested in participating. As part of GKE, we've seen increasing needs for checkpoint/restore capabilities for both GPU and non-GPU use cases.

Pengzhan Hao

[Jinda Lu]

+1 on the WG. I am from Alibaba Cloud and hope to join this WG. We use the checkpoint/restore capability. Currently, the support for checkpoint/restore on the kubernetes is not good. Our implementation is very customized. I hope to work with everyone to improve kubernetes C/R.

[Cheng Wang]

+1 for the WG; I am interested in this area.

Jinda Lu <q888...@gmail.com> 于2025年7月3日周四 22:15写道：

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/a6ff9e42-3b93-4c52-be97-3452d31983e7n%40kubernetes.io.

--

Less is more...

[Roger Xi]

i'd like to join the WG. i work in Microsoft. R/C will be helpful for gpu related workload and i'd like to contribute on it.

cheers,

Roger

[Fury kerry]

+1 for the WG; I am interested in this area.

On Tue, Jul 29, 2025 at 8:18 PM Roger Xi <roger....@gmail.com> wrote:

i'd like to join this group. thanks.

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/88e4a6a7-45a9-422f-9764-b9b697898906n%40kubernetes.io.

--

Zhen Zhang

Alibaba Cloud

[Parthiba Hazra]

+1 for the WG, Hi I'm Parthiba from DevZero (devzero.io). We're building a custom Kubernetes autoscaler with migration caps, and several of our users are asking for CUDA C/R to improve GPU utilization for inference workloads. I'm excited to help shape a native production-ready way to handle CUDA C/R in Kubernetes. 

Regards

Parthiba Hazra

[Dhruv Bindra]

Hello, +1 for WG, I'm Dhruv from Manifold Labs. I'm interested in contributing

[Camilla Montonen]

Hello everyone!

>   We are already discussing the future of
Checkpoint/Restore in Kubernetes at conferences, in private email
threads, private Slack channels, in GitHub issues and pull requests, and
an increasing number of people propose good ideas on how to move
forward. 

I was wondering if there is a single document or maybe documents that give an overview of the current

state of the design/sketches/ideas/discussions of how this functionality would work from the perspective of a user (someone runningworkloads on k8s)?

I have an idea on what I'd like the user-facing design of checkpoint-restore to look like based on my own very narrow experiences of running ML pipelines and model training jobs on k8s, 

but I'd like to expand my context and thinking on this by reading what others have already proposed in this space. 

Thanks in advance!

Camilla

[Adrian Reber]

Hello Camilla,

no there is not. That is one of the reasons why we decided to create
this working group. We have a lot of ideas and plans but that was always
discussed between a small number of people and mainly privately. With
the working group we want to change that. I am currently waiting on some
things concerning setting up the mailing list and once that is done we
will make an announcement how to continue.

Adrian

On Sat, Nov 15, 2025 at 04:48:50AM -0800, Camilla Montonen wrote:
> Hello everyone!
>
> > We are already discussing the future of
> Checkpoint/Restore in Kubernetes at conferences, in private email
> threads, private Slack channels, in GitHub issues and pull requests, and
> an increasing number of people propose good ideas on how to move
> forward.
>
> I was wondering if there is a single document or maybe documents that give
> an overview of the current
> state of the design/sketches/ideas/discussions of how this functionality
> would work from the perspective of a user (someone runningworkloads on k8s)?
>
> I have an idea on what I'd like the user-facing design of
> checkpoint-restore to look like based on my own very narrow experiences of
> running ML pipelines and model training jobs on k8s,
> but I'd like to expand my context and thinking on this by reading what
> others have already proposed in this space.
>
> Thanks in advance!
> Camilla
>
>
> On Wednesday, October 29, 2025 at 8:29:20 PM UTC+1 Dhruv Bindra wrote:
>
> > Hello, +1 for WG, I'm Dhruv from Manifold Labs. I'm interested in
> > contributing
> >
> > On Saturday, August 16, 2025 at 8:59:15 AM UTC-5 Parthiba Hazra wrote:
> >
> >> +1 for the WG, Hi I'm Parthiba from DevZero (devzero.io

> >> <https://www.devzero.io/>). We're building a custom Kubernetes

> >>>> <https://groups.google.com/a/kubernetes.io/d/msgid/dev/88e4a6a7-45a9-422f-9764-b9b697898906n%40kubernetes.io?utm_medium=email&utm_source=footer>
> >>>> .