Prior Sidecar KEP's prerequisite has been satisfied and should move forward

[McFly Marty]

Hi,all

The initial KEP Sidecar Container KEP had been proposed about 4 years ago, lots of discussions on that demonstrates its huge popularity,  today its prerequisite described in this KEP, that is Graceful Node Shutdown, has been proposed and implemented.So shall we move it forward to implement the KEP and make it cover more user scenarios?

Here are some previous discussions links：

• https://github.com/kubernetes/enhancements/issues/753
• https://github.com/kubernetes/community/pull/2148
• user case discussions： https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#

Initial KEP just concerns sidecar container startUp/shutsown behaviors, but service mesh may need more specific function like independent/hot upgrde to make business be not aware the upgrade of proxy, therefore, maybe we could extend Sidecar Container functionality,take Envoy as an example, this blog shows us that Envoy can transfer states between new and old instance, so what we should do on k8s side is supporting a mechanism to allow two sidecar containers both exists during upgrading Envoy sidecar container and kill the old one after upgrade complete.

So maybe we can re-discuss API design here to make it cover more  scenarios.It's amazing if we could move this forward because this KEP is pending for much long time and indeed it can solve many pain points we faced tody.  

[Rodrigo Campos]

On 4/21/22 16:07, McFly Marty wrote:
> Hi,all
>
> The initial KEP Sidecar Container KEP

> <https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers> had

> been proposed about 4 years ago, lots of discussions on that
> demonstrates its huge popularity,  today its prerequisite described in
> this KEP, that is Graceful Node Shutdown

> <https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown>,

> has been proposed and implemented.So shall we move it forward to
> implement the KEP and make it cover more user scenarios?

Well, there is no such KEP that covers more scenarios today. That is
kind of a blocker to do that :)

As you linked in the use cases below, there was that use case collection
and we created pre-proposals to further discuss which direction we want
to go. But that discussion to set our north star, didn't happen.

To continue to move, though, this other KEP was created by some
community members and myself (added them on Cc, just in case):

https://github.com/kubernetes/enhancements/pull/2869

The idea was to solve a simpler problem as a way to also collect
feedback and gain more experience, while we decide what a more complex
solution should look like (the north star). This simple KEP is also
extensible to any of the pre-proposals we have so far (extensible in the
sense that we can incrementally add more complexity and features until
we reach what is proposed in a pre-proposal)

This KEP got one SGTM from Tim Hockin, but didn't make the cut for the
past k8s release during the sig-node meetings. I think from that point
onwards (a few months ago), progress was stalled.

> Here are some previous discussions links：
>

> * https://github.com/kubernetes/enhancements/issues/753
> * https://github.com/kubernetes/community/pull/2148
> * user case discussions：

> https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#
>
>
> Initial KEP just concerns sidecar container startUp/shutsown behaviors,
> but service mesh may need more specific function like independent/hot
> upgrde to make business be not aware the upgrade of proxy, therefore,
> maybe we could extend Sidecar Container functionality,take Envoy as an
> example, this blog

> <https://blog.envoyproxy.io/envoy-hot-restart-1d16b14555b5> shows us

> that Envoy can transfer states between new and old instance, so what we
> should do on k8s side is supporting a mechanism to allow two sidecar
> containers both exists during upgrading Envoy sidecar container and kill
> the old one after upgrade complete.

Wow, that seems to go way more than the scope of any of the already
quite complex pre-proposals we have discussed.

My advice would be to not try to solve so many things, but define a
clear problem that has enough users and is extensible for incremental
improvements. But even if you do that, what is hard is to get things
moving. The things to discuss here are _hard_ things, that take time and
will have _lot of consequences_, so is usually hard to make progress
with those discussions with the people that needs to be involved.

> So maybe we can re-discuss API design here to make it cover more
> scenarios.It's amazing if we could move this forward because this KEP is
> pending for much long time and indeed it can solve many pain points we
> faced tody.

Count me in to help in the discussion in any way that I can.



Best,
Rodrigo

[Matthias Bertschy]

Hi, matthyx here,

Indeed, we have a KEP number (2872) and were tracked in the 1.24 enhancements tracking list.

However after a misunderstanding with ehashman we were too short for the PRR approval and it was basically it.

Let's try to be in 1.25 just after code thaw.

Regards,

Matthias

[McFly Marty]

Hi, all

Thanks for your timely reply, the new KEP you mentioned defines only one `type` filed, which doesn't include some upgrade business,

as the sidecar usercase  discussions mentioned, there are some upgrade scenarios we may take into account in this new KEP,

and truly it's a very common use case as sidecar container becomes more popular.

Actually, the Envoy hot restart is a more complicated case, which needs smooth upgrading of old and new envoy instances , so we can begin with a simpler case, 

in some logging/monitoring cases, we may just need this sidecar to restart instead of hot restart because we can tolerate some miss of logs/monitoring data when upgrading, 

this is a simpler case than Envoy, we can call the two cases as hot upgrade and cold upgrade(just tocompare) , so far, when we upgrade sidecar container, all containers in a pod will be killed, which is

a little untolerate for us, so even it's a simpler case, the k8s and this KEP can't deal with it well, so my thought is that if we could define some upgrade policies like hotUpgrade or coldUpgrade 

in a struct to cover this donot killing business container case, which hotUpgrade means starting a new sidecar container first and cooperating with old instance to deal with Envoy like case,

while coldUpgrade means just killing sidecar container as normal container do and start a new one, and both cases here should not kill the business container.

Best,

Mcfly

[Rodrigo Campos]

On 4/22/22 13:12, McFly Marty wrote:
> Hi, all
>
> Thanks for your timely reply, the new KEP you mentioned defines only one
> `type` filed, which doesn't include some upgrade business,
> as the sidecar usercase discussions mentioned, there are some upgrade
> scenarios we may take into account in this new KEP,
> and truly it's a very common use case as sidecar container becomes more
> popular.

Yes, as I mentioned in the previous mail, this was intentional.

Would you like to write a pre-proposal (see the other existing ones in
the link you pasted in the first email) with the idea you have to handle
more use cases?

[Matthias Bertschy]

Count me in, I will help support this initiative... even if I think the future or service mesh is sidecar-less (with eBPF or GRPC xDS).

Please include me in the list, cheers,

Matthias

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/1a172c25-79a9-4fe9-8f13-b46417982266n%40googlegroups.com.

[Marty Mcfly]

Hi,all

I have writen a pre-proposal Sidecars upgrading policy, which is also appended at the shared doc mentioned before, it's my pleasure to get your review and thank you all!

Best, 

Marty