How to share memory across different pods?

[hongbo xu]

Hi, all:

      With emptyDir(medium=memory), k8s Containers in same pod can share with  POSIX shared-memory.

      But does k8s support same mechanism to share memory between different pods?

      Following is our use case:

      

      Our applications comprise  multi-services . each service is a state-less computing heavy applications. And they shared the same cache. The biggest problem is we need very fast cache access speed, distributed cache like redis is un-acceptable. So we implement our own  high speed cache framework using POSIX shared memory.

      Now we are try to migrate  application to k8s. we want to deploy each service as a normal pod. and the share memory as a daemon-pod. So we need to shared memories between the service pod with the shared-memory pod.

      By the way ,  other mechanism to share data between  pods is also acceptable if as fast as shared-memory-IPC.

[Bowei Du]

If you have control over the node itself, you can try using mounting the same `hostPath` (http://kubernetes.io/docs/user-guide/volumes/#hostpath) in all of the pods.

This comes with the caveats of of having to manage things yourself.

Bowei

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/c12d25e5-e42b-4beb-ac48-fa76b07333e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Hockin]

It also comes with the caveat of not knowing who will get "charged"
for the memory, so every pod has to be provisioned to be able to
absorb it, depending how it is written. It also might "leak" up to
the root namespace and be charged to nobody but appear as "overhead"

On Wed, Dec 21, 2016 at 10:33 AM, 'Bowei Du' via Kubernetes
developer/contributor discussion <kuberne...@googlegroups.com>
wrote:

>> email to kubernetes-de...@googlegroups.com.
>> To post to this group, send email to kuberne...@googlegroups.com.

>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kubernetes-dev/c12d25e5-e42b-4beb-ac48-fa76b07333e6%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes developer/contributor discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-de...@googlegroups.com.
> To post to this group, send email to kuberne...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-dev/CAGv8vBq76YJ2QQG5zjZ7SdGUgqgfuvaEyMECxGfoagXe87PK-g%40mail.gmail.com.

[Vishnu Kannan]

In general, pods are not guaranteed to land on the same node. Since sharing memory typically requires pods to be on the same node, we do not recommend sharing memory across pods. You can run multiple containers within a pod and they get shared memory automatically.

On Wed, Dec 21, 2016 at 4:05 PM, hongbo xu <xuhongb...@gmail.com> wrote:

Hi, all:

      With emptyDir(medium=memory), k8s Containers in same pod can share with  POSIX shared-memory.

      But does k8s support same mechanism to share memory between different pods?

      Following is our use case:

      

      Our applications comprise  multi-services . each service is a state-less computing heavy applications. And they shared the same cache. The biggest problem is we need very fast cache access speed, distributed cache like redis is un-acceptable. So we implement our own  high speed cache framework using POSIX shared memory.

      Now we are try to migrate  application to k8s. we want to deploy each service as a normal pod. and the share memory as a daemon-pod. So we need to shared memories between the service pod with the shared-memory pod.

Can you provide some more details on the functionalities of this "Daemon-pod" and the overall architecture of your application? 

It might be possible to have the "daemon-pod" setup a tmpfs on the node at a standard location and have pods access that tmpfs mount via HostPath volumes.  

      By the way ,  other mechanism to share data between  pods is also acceptable if as fast as shared-memory-IPC.

--

You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

[hongbo xu]

Thanks for your suggestion. I need to express our application architecture and our demand more clearly. following is the details (some details has been discussed , but I repeated it here)

1) What's the architecture of our applications now ? (not using k8s)

1.1) application architecture: 

Our applications is composed of  to multi processes. One process maintenance a cache implemented by Shared-memory and keep consistence with other machine's cache.  All the other processes  are  http restful service and served as  query service for the other system (each query will do heavily computing over data in cache ).  When business-logical changed, we need add new process or changed the exist process;

1.2) cluster architecture: 

applications'cluster composed of multi physical machines, each machine is deployed same processes; when we want scale up, we must add new physical machine manually.

  

2. What's our purpose when we want to using k8s?

2.1)Each query service can be deployed and evolution independently. 

So  each service should be deployed as one pod, and cannot combine all services in one pod.

2.2) Elastic-Scale:

Business Service elastic scale : The application is composed of multi business-service to do query. The service's cpu load are dynamically and not synchronous fluctuation. Sometime load of service a,b,... will be very heavy , and other times service x,y, will be very heavy. So we want each services can elastically scale up && down according their true cpu-load.

Cache Process  elastic scale:    Business-services can be spread to new nodes without cache, so  the cache process should also be spread to new node correspondingly.

3. What's the solution and difficult problems when we using k8s?

At first,  I want  deploy each query service as a pod , and cache-maintenance-process as a pod(cache pod for simplify). but difficult problem occurs.

3.1) When the query service scale up/down to new node, how to scale up/down the cache pod together; and keep only single cache pod in one node.  Even more difficult thing is to  recycle the cache-maintenance-process.

At first I think deploy cache pod as  Daemon-Set  is enough, but actually it seems not. Because it cannot scale up/down dynamically. 

So I wonder if any share-memory tech between different pods. But I think I have make a mistake, the actually requirement for us is some pod correlation mechanism supported by k8s.

 

在 2016年12月22日星期四 UTC+8上午10:57:56，Vishnu Kannan写道：

In general, pods are not guaranteed to land on the same node. Since sharing memory typically requires pods to be on the same node, we do not recommend sharing memory across pods. You can run multiple containers within a pod and they get shared memory automatically.

On Wed, Dec 21, 2016 at 4:05 PM, hongbo xu <xuhongb...@gmail.com> wrote:

Hi, all:

      With emptyDir(medium=memory), k8s Containers in same pod can share with  POSIX shared-memory.

      But does k8s support same mechanism to share memory between different pods?

      Following is our use case:

      

      Our applications comprise  multi-services . each service is a state-less computing heavy applications. And they shared the same cache. The biggest problem is we need very fast cache access speed, distributed cache like redis is un-acceptable. So we implement our own  high speed cache framework using POSIX shared memory.

      Now we are try to migrate  application to k8s. we want to deploy each service as a normal pod. and the share memory as a daemon-pod. So we need to shared memories between the service pod with the shared-memory pod.

Can you provide some more details on the functionalities of this "Daemon-pod" and the overall architecture of your application? 

It might be possible to have the "daemon-pod" setup a tmpfs on the node at a standard location and have pods access that tmpfs mount via HostPath volumes.  

      By the way ,  other mechanism to share data between  pods is also acceptable if as fast as shared-memory-IPC.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

[David Oppenheimer]

You can use pod affinity to say "pods from collection X can only run on nodes that are running pods from collection Y"

You can use pod anti-affinity to say "run no more than one pod from collection Z on any particular node"

However, this isn't sufficient to ensure that when Y scales auto-scales up, X will also scale up. You'd need to write your own pod autoscaler or take the one that comes up wtih Kubernetes and modify it.

If you're willing to do the scaling manually, the two features I mentioned above should solve your pod placement problem. It won't solve the memory sharing problem, though.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/6030ecd3-219c-4214-bdff-c45164ff1c64%40googlegroups.com.

[Justin Garrison]

I will have a need for the same thing. Without Kubernetes (just running docker on a local machine) I am able to share memory between containers with --ipc container:name [1] but I don't see that available in Kubernetes (or at least not documented). Because the names of pods is autogenerated I had thought using --ipc host may be good enough but that only works when the applications are trusted to have access to the host's memory. I have not tested but had planned to run a DaemonSet per node that was responsible for maintaining the local in memory cache of the host. Other pods that would run on each node would then use the host's memory namespace and hopefully the application will do the right thing to pull cached data when requested.

I have no idea yet if this will actually work in practice, but what Tim pointed out about memory allocation will be something I'll have to figure out still.

1. https://docs.docker.com/engine/reference/run/#/ipc-settings---ipc

[hongbo xu]

Thanks you guys, Thanks your suggestion:

        It seems now I can use  hostPath  to resolve share memory problem, pod affinity && anti-affinity to resolve placement problem,  manual scale to resolve elastic query & cache service's elastic-scale problem. 

        I'll try use these together to solve our problem.

[Dheeraj Singh]

Did the solution for mounting the hostpath /dev/shm to pods /dev/shm worked for you?

In my case I am facing problem when I spawn multiple pods simultaneously(ie: 30)