Container Migration

[Wenqi Cao]

I am wondering why kubernetes does't provide one approach for online migrating containers. Even in the latest proposal, kubernetes talks about it's going to create one rescheduler, which terminates a pod that is managed by a controller, and the controller will create a replacement pod that is then scheduled by the pod's scheduler. This approach works for stateless applications, since it simply shutdowns old ones and creates new ones. However, what is the solution for stateful application? 

Moreover, how do you think the following statement

"Stateful things scale vertically, stateless things scale horizontally", is this the idea that Googlers are using for designing kubernets?

Thanks

[David Aronchick]

What you are referring to is frequently called "improved stateful application support" (codename: petset).

An alpha version will be available in Kubernetes 1.3 (https://github.com/smarterclayton/kubernetes/blob/961f1f94c35d4979ac83bbad482090cb6c22781c/docs/proposals/petset.md), and here are some examples: https://github.com/kubernetes/contrib/tree/master/pets

To be clear, it is a best practice NOT to make any changes and/or use local disk and/or care about the exact size and shape of your container/node. Ideally, you'd mount in any persistent storage from externally (e.g. NFS store, external database, etc). By breaking up your components in this way, you'll likely achieve better utilization and more fault tolerant infrastructure.

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/28fe04c4-06df-49a3-ab68-ab422c183098%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Hockin]

There are 2 answers. PetSet is coming, and that will make clustered
apps like zookeeper more happy. But Simple PersistentVolumes also
work well for stateful apps - it decouples the lifetimes of the state
and the containers.

> https://groups.google.com/d/msgid/kubernetes-dev/CADSfKX%3Dini_Yp2DtT6Txkt%3DAxv2__DGUpj7R2bEXwA2bhrjxsA%40mail.gmail.com.

[David Oppenheimer]

"Container migration" can mean a lot of things -- the app doesn't know it is killed, or the data is transparently migrated (if local) and reattached, or the IP address stays the same when you move to a new node, etc.

AFAIK there is no "true" container migration solution available today, in Kubernetes or any other system, that is a full solution (i.e. totally transparent to the application) analogous to GCE Live Migration. But you can address pieces of it, for example as David and Tim said by using a networked Persistent Volume.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAO_RewYjrFkG4kAAsbOj0mux859Txu7aDMM7U4mKQKJ3GEL8ow%40mail.gmail.com.

[Wenqi Cao]

Actually what I am thinking is how to migrate data stored in memory from one container to another. If everything is stored in disk, just simply migrating container from A place to B is ok, and transparent for application level. However, what if data in memory also needs to migrate. how to solve this?

Thanks

[David Oppenheimer]

On Wed, Jun 8, 2016 at 1:34 PM, Wenqi Cao <wenq...@gmail.com> wrote:

Actually what I am thinking is how to migrate data stored in memory from one container to another. If everything is stored in disk, just simply migrating container from A place to B is ok, and transparent for application level. However, what if data in memory also needs to migrate. how to solve this?

Yeah, that's exactly the kind of thing I was talking about when I said  "there is no "true" container migration solution available today, in Kubernetes or any other system, that is a full solution (i.e. totally transparent to the application) analogous to GCE Live Migration"

If you do a web search for [container live migration] you can find information about some of the experimental work in this space, e.g. CRIU.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/7ad492c9-648c-470b-bc6d-9133c8d206bc%40googlegroups.com.

[Wenqi Cao]

Ok. I got. Thanks for all your answers.

[Tim Hockin]

There's an opportunity for "warm" migration, but it requires careful
application design and orchestration. Specifically, you can use a
tmpfs and store all of your "durable" state there (and maybe mmap() it
into your app). An orchestrator (kube does *not* do this) could
bulk-copy that data to the destination. Your app would bounce when it
moves but you could save the important state.

I don't know anyone who is doing that.

[David Aronchick]

Can you say more about your requirements? Are you trying to mimic live VM migration (e.g. on Google Compute Platform or with VMWare)? 

[Wenqi Cao]

There are some pretty mature technique and algorithm for live migration, but in virtual machine level. For example, pre-copy and post-copy. Are you planning to merge them into Kubernetes?

[Tim Hockin]

On Wed, Jun 8, 2016 at 3:32 PM, Wenqi Cao <wenq...@gmail.com> wrote:
> There are some pretty mature technique and algorithm for live migration, but
> in virtual machine level. For example, pre-copy and post-copy. Are you
> planning to merge them into Kubernetes?

In short, no. At least, not any time soon. Live migration of
arbitrary running code is a *very* hard problem and not something
we're currently aiming at. It would bring with it a lot of changes to
Kubernetes that would (in my opinion) move is the wrong direction. If
someone can eventually demonstrate a 95% success rate for
live-migrating arbitrary user code, we can talk about the requirements
on the core system.

I'd much rather approach it from base requirements and understand what
problems people are really trying to solve.

> https://groups.google.com/d/msgid/kubernetes-dev/17bb4b18-8daf-4194-ba5b-2265daabd86d%40googlegroups.com.

[Jeremy Ong]

You could create a ram disk approach and change the malloc/calloc/free/new/delete implementation to DMA to the ramdisk instead using the mmap as suggested earlier. The ramdisk can persist after the container restarts and you'd effectively have a container hotswap with persistent RAM. This is something I've been considering for our own internal deploys, but of course you need to work out compatibility issues if struct/class definitions change or even if the compiler changes (things may not be memory aligned or padded the same way). Optimization level changes to the build may affect it too. The approach being considered is to use a ramdisk only for things supported by a reflection/serialization library to take care of compatibility issues.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAO_RewaiSUHDqR%3D_F_qNkhYLHdW%2BYYTCybNiJOAA7HUxq%2BZUvg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Jeremy Ong

PlexChat CTO

650.400.6453

[David Aronchick]

On Wed, Jun 8, 2016 at 3:57 PM, Tim Hockin <tho...@google.com> wrote:

I'd much rather approach it from base requirements and understand what
problems people are really trying to solve.

100% agree - Wenqi, can you help us understand what you're trying to do?