State of LocalStorage support in StatefulSets

[krma...@gmail.com]

Hi All

I see this https://github.com/kubernetes/kubernetes/pull/30044

I am assuming Local Storage support and StatefulSets are not tied in terms of implementation although local storage support does enable a large number of stateful clustered applications deployment.

We are very much interested in this and would like to know what release is this being scheduled for(in how many months would that release happen) and who is working on it ? Are there any active PR's in this area, which sig is driving this.

Also another related thing is support for stable ips for StatefulSets which dont change across pod deletion. I dont see any issues related to that. Is this also being actively worked on ? What is the relative priority of stable ip support in statefulset vs localstorage support in StatefulSet ?

Thanks for your help

Mayank

[Eric Tune]

You are correct that they are not tied in term of implementation.  A StatefulSet might or might not use local storage; local storage could be used by any sort of Pod, regardless of what controller creates it.

I view local storage as a much higher priority than stable IPs.  Here is why: 

- Relatively few applications need stable IPs.   Many apps could make use of local storage. 

- I think we mostly know how to build Local Storage.  It can be implemented in a way that is relatively independent of what sort of machines you are using (cloud provider vs on-prem, etc).  I don't think we know fully how to build sticky IPs in a way that works well with all different networking layers.

- lack of sticky IPs can in theory be worked around by changing apps that require it.  Lack of Local Storage support can't very easily be worked around without core kubernetes changes.  Hence, the latter is a priority for Kubernetes team;  maybe there is even some progress on the former within other projects.

In terms of what the actual schedule is for developing those two features, I don't know, that is up to the SIGs.  I'll poke people from sig-network and sig-storage and see if they can reply to this thread.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/65d8d963-b0ca-47a3-a4b0-c625e54fb6a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Oppenheimer]

On Mon, Dec 5, 2016 at 9:34 PM, 'Eric Tune' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:

You are correct that they are not tied in term of implementation.  A StatefulSet might or might not use local storage; local storage could be used by any sort of Pod, regardless of what controller creates it.

I view local storage as a much higher priority than stable IPs.  Here is why: 

- Relatively few applications need stable IPs.   Many apps could make use of local storage. 

- I think we mostly know how to build Local Storage.  It can be implemented in a way that is relatively independent of what sort of machines you are using (cloud provider vs on-prem, etc).  I don't think we know fully how to build sticky IPs in a way that works well with all different networking layers.

- lack of sticky IPs can in theory be worked around by changing apps that require it.  Lack of Local Storage support can't very easily be worked around without core kubernetes changes.  Hence, the latter is a priority for Kubernetes team;  maybe there is even some progress on the former within other projects.

+1

 

In terms of what the actual schedule is for developing those two features, I don't know, that is up to the SIGs.  I'll poke people from sig-network and sig-storage and see if they can reply to this thread.

On Sun, Dec 4, 2016 at 10:21 PM, <krma...@gmail.com> wrote:

Hi All

I see this https://github.com/kubernetes/kubernetes/pull/30044

I am assuming Local Storage support and StatefulSets are not tied in terms of implementation although local storage support does enable a large number of stateful clustered applications deployment.

We are very much interested in this and would like to know what release is this being scheduled for(in how many months would that release happen) and who is working on it ? Are there any active PR's in this area, which sig is driving this.

Also another related thing is support for stable ips for StatefulSets which dont change across pod deletion. I dont see any issues related to that. Is this also being actively worked on ? What is the relative priority of stable ip support in statefulset vs localstorage support in StatefulSet ?

Thanks for your help

Mayank

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/65d8d963-b0ca-47a3-a4b0-c625e54fb6a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAASt_VFFxp_p127sicY9%3D5YgiwofdeCqYqA_DBczuyCN5ALL-g%40mail.gmail.com.

[Vishnu Kannan]

FYI: I will be working on improving local storage accounting and management. This issue is the most relevant one that I can think of. I'd recommend following that to track progress. We are hoping to have a design finalized by v1.6.

On Mon, Dec 5, 2016 at 10:17 PM, 'David Oppenheimer' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAASt_VFFxp_p127sicY9%3D5YgiwofdeCqYqA_DBczuyCN5ALL-g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAOU1bzec1pMcJwP_D9ZvpVtq3JOwYYu-GTcD5sSgj3h7B0XgCQ%40mail.gmail.com.

[Prashanth B]

We've discussed sticky ips (https://github.com/kubernetes/kubernetes/issues/28969, https://github.com/kubernetes/kubernetes/issues/28718, https://github.com/kubernetes/kubernetes/issues/28660). 

The immediate solution would be to create a Service per pet, but in addition to being gross, that doesn't solve the problem in all cases (eg if the app is looking up the actual source IP on the packet it receives for whiltelists). In theory one can segment out a "petset-CIDR" from the `--cluster-cidr` range and pipe it down through the IPAM plugin, then allocate the same IP and re-program routing for a given pet. 

I don't think we have a concrete proposal out, let alone a timeline for when/if that'll land. 

On Monday, December 5, 2016 at 10:22:04 PM UTC-8, Vishnu Kannan wrote:

FYI: I will be working on improving local storage accounting and management. This issue is the most relevant one that I can think of. I'd recommend following that to track progress. We are hoping to have a design finalized by v1.6.

On Mon, Dec 5, 2016 at 10:17 PM, 'David Oppenheimer' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:

On Mon, Dec 5, 2016 at 9:34 PM, 'Eric Tune' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:

You are correct that they are not tied in term of implementation.  A StatefulSet might or might not use local storage; local storage could be used by any sort of Pod, regardless of what controller creates it.

I view local storage as a much higher priority than stable IPs.  Here is why: 

- Relatively few applications need stable IPs.   Many apps could make use of local storage. 

- I think we mostly know how to build Local Storage.  It can be implemented in a way that is relatively independent of what sort of machines you are using (cloud provider vs on-prem, etc).  I don't think we know fully how to build sticky IPs in a way that works well with all different networking layers.

- lack of sticky IPs can in theory be worked around by changing apps that require it.  Lack of Local Storage support can't very easily be worked around without core kubernetes changes.  Hence, the latter is a priority for Kubernetes team;  maybe there is even some progress on the former within other projects.

+1

In terms of what the actual schedule is for developing those two features, I don't know, that is up to the SIGs.  I'll poke people from sig-network and sig-storage and see if they can reply to this thread.

On Sun, Dec 4, 2016 at 10:21 PM, <krma...@gmail.com> wrote:

Hi All

I see this https://github.com/kubernetes/kubernetes/pull/30044

I am assuming Local Storage support and StatefulSets are not tied in terms of implementation although local storage support does enable a large number of stateful clustered applications deployment.

We are very much interested in this and would like to know what release is this being scheduled for(in how many months would that release happen) and who is working on it ? Are there any active PR's in this area, which sig is driving this.

Also another related thing is support for stable ips for StatefulSets which dont change across pod deletion. I dont see any issues related to that. Is this also being actively worked on ? What is the relative priority of stable ip support in statefulset vs localstorage support in StatefulSet ?

Thanks for your help

Mayank

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/65d8d963-b0ca-47a3-a4b0-c625e54fb6a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAASt_VFFxp_p127sicY9%3D5YgiwofdeCqYqA_DBczuyCN5ALL-g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.

To post to this group, send email to kuberne...@googlegroups.com.

[krma...@gmail.com]

Thanks all of you for the pointers. While this is being designed, what is the suggested solution for local storage. Are people using hostPath on a pod/daemonset with a service per pod/daemonset ?

[krma...@gmail.com]

Thanks all of you. Is there a recommendation on how to use local storage in a scalable way for doing stateful containers until official support for local storage arrives ? Are there design guides which allow us to make use of hostPath with replicated databases and service per node  ?

[Deyuan Deng]

FYI, we wrote custom controllers on top of kubernetes which knows how to place pods and assign hostPath to each pod (use node selector).  We do this for each stateful application; they all have different requirements.

[krma...@gmail.com]

Hi Deyuan

Interesting. Does hostPath have any limitation that you ran into. Can the same hostPath be mounted by many pods at the same time ?

-Mayank

[Deyuan Deng]

We don't see any problems right now.  If different pods land on the same machine and use the same hostPath, you should make sure the app can handle that (e.g. don't write to the same file concurrently); but of course, you can control how pods are scheduled with ur own controller.