Sidecar: Let's collect use cases for sidecar containers

[Rodrigo Campos]

Hi!

In the meeting we had past Friday, we decided to continue by creating a document of the different use cases we want to properly support in Kubernetes.

This use cases collection is crowdsource and please add yours to help us to understand everyone's needs! :)

Once we have several use cases, different interested people can start to experiment/explore different ideas to support them in Kubernetes. Please do not hesitate to explore yourself! :)

The doc with use cases is here (please check if there isn't an entry for your use case already):

https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#heading=h.eo3yz0vw744k

Thanks again to everyone for your time and help with this :)
--
Rodrigo Campos
---
Kinvolk GmbH | Adalbertstr.6a, 10999 Berlin | tel: +491755589364
Geschäftsführer/Directors: Alban Crequy, Chris Kühl, Iago López Galeiras
Registergericht/Court of registration: Amtsgericht Charlottenburg
Registernummer/Registration number: HRB 171414 B
Ust-ID-Nummer/VAT ID number: DE302207000

[Tim Hockin]

Where should we share proposal docs? I have started on one.

[Rodrigo Campos]

Awesome, thanks!

What about linking it in the same doc? I've just added a section to link them there: https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#heading=h.t08dvudw5b1r

[Tim Hockin]

Sadly, I have to share with $someone since my docs domain won't allow
open shares. How about I share with kubernetes-dev and
kubernetes-sig-node ? I will do another run thru the requirements
today and drop a note here :)

[Rodrigo Campos]

Yes, that sound great. Thanks again! :)

[Tim Hockin]

Hi all,

I have shared a first draft: http://bit.ly/thockin-pod-lifecycle-phases-proposal

Comments are very welcome. Hopefully other ideas will materialize.

[Josh Berkus]

Tim,

Question for this proposal: what's the definitional difference between a
"sidecar" container, and any secondary container that shares the same pod?

Certainly service mesh containers are sidecars. But should, for
example, an automatic application backup container be considered a
sidecar? Why or why not?

Not trying to be pedantic here -- I'm looking over the list of use
cases, and I don't feel like we have consensus on what a sidecar *is*.
Your proposal makes a number of assumptions, like that the sidecar needs
to start before application containers and close after them, which might
be part of that definition, but it's not explicitly stated.

--
--
Josh Berkus
Kubernetes Community
Red Hat OSPO

[Tim Hockin]

On Fri, Nov 6, 2020 at 9:51 AM Josh Berkus <jbe...@redhat.com> wrote:
>
> Tim,
>
> Question for this proposal: what's the definitional difference between a
> "sidecar" container, and any secondary container that shares the same pod?

Only whether it is started during AppRun. The distinction is
arbitrary and artificial today. By earmarking AppRun as the real
workload, everything else is implicitly "scaffolding".

> Certainly service mesh containers are sidecars. But should, for
> example, an automatic application backup container be considered a
> sidecar? Why or why not?

That's up to the app owner. If you start it during AppRun, then the
end-of-life does not start until both the application and the backup
container terminate. If you start it during AppInit, then it will be
automatically stopped once the main app dies.

Which of those is correct for your app is not for me to decide.

> Not trying to be pedantic here -- I'm looking over the list of use
> cases, and I don't feel like we have consensus on what a sidecar *is*.
> Your proposal makes a number of assumptions, like that the sidecar needs
> to start before application containers and close after them, which might
> be part of that definition, but it's not explicitly stated.

Fair. I think "must envelope the app" is a requirement for some
(most?) sidecars and seems unlikely to be harmful to those for whom it
is not a requirement.

[Tim Hockin]

What's next? The process here was supposed to be that we
collect a handful of alternate ideas and then debate/discuss the
relative merits. I clearly took some opinions here which are open to
debate. I can refine this proposal more, but I don't know if that is
the best course - I don't have very much feedback on it, considering
the scope of the proposal :)

I am waiting eagerly for critical feedback and/or alternate ideas.

Tim

[Rodrigo Campos]

(Adding Derek in Cc)

Hi Tim! Sorry for the delay, I was on PTO.

I've created a doc that explores a wide range of different _possible_ options (very high level), one is along the lines of what you propose (but I was waiting to align internally here, so haven't published, sorry :-/). Your proposal goes in way more detail than I did and is better polished, so I don't think it has merit to share mine now :)

As far as I know, there was no one else working on this when I asked on SIG-node.

Derek, did you or Seth want to explore other alternatives too? Would you be able to review Tim's "pre-proposal" so we can discuss afterwards (not sure if in SIG-node or how)?

Here is the use cases doc we collected:
https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#heading=h.eo3yz0vw744k. At the top of that document it is the link to the current pre-proposals, for now only one from Tim.

Thanks a lot,

Rodrigo

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/CAO_RewYtG%3D26Tm5c-sqicqUhkp4SF5PgYUZAROcDqeOf6crp_Q%40mail.gmail.com.

[Rodrigo Campos]

On Thu, Nov 26, 2020 at 6:27 PM Rodrigo Campos <rod...@kinvolk.io> wrote:
>
> (Adding Derek in Cc)
>
> Hi Tim! Sorry for the delay, I was on PTO.
>
> I've created a doc that explores a wide range of different _possible_ options (very high level), one is along the lines of what you propose (but I was waiting to align internally here, so haven't published, sorry :-/). Your proposal goes in way more detail than I did and is better polished, so I don't think it has merit to share mine now :)
>
> As far as I know, there was no one else working on this when I asked on SIG-node.
>
> Derek, did you or Seth want to explore other alternatives too? Would you be able to review Tim's "pre-proposal" so we can discuss afterwards (not sure if in SIG-node or how)?
>
> Here is the use cases doc we collected:
> https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#heading=h.eo3yz0vw744k. At the top of that document it is the link to the current pre-proposals, for now only one from Tim.

Friendly ping, Derek?

[Rodrigo Campos]

(Adding Seth in Cc:)

Hi, we talked about sidecars at yesterday's SIG node meeting. In the
agenda there are some notes about what we talked
(https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit)

Derek will try to finish reading Tim's proposal and we will coordinate
another meeting in early Jan (it would be great if you can come too,
Tim!).

It is not clear to me if Derek/Seth want to create other proposals,
but they are definitely busy now. Derek/Seth, feel free to update that
yourselves :)


Thanks,
Rodrigo

[Tim Hockin]

On Wed, Dec 9, 2020 at 6:32 AM Rodrigo Campos <rod...@kinvolk.io> wrote:
>
> (Adding Seth in Cc:)
>
> Hi, we talked about sidecars at yesterday's SIG node meeting. In the
> agenda there are some notes about what we talked
> (https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit)
>
> Derek will try to finish reading Tim's proposal and we will coordinate
> another meeting in early Jan (it would be great if you can come too,
> Tim!).

Just give me warning and I will come :)

> It is not clear to me if Derek/Seth want to create other proposals,
> but they are definitely busy now. Derek/Seth, feel free to update that
> yourselves :)

We're all busy and this is an enormous and subtle topic. Don't let me
force anyone's hands. It has languished this long, it can languish
some more. It's not like I am personally signing up to implement this
all.

[Brendan Burns]

fwiw, David Oppenheimer and I wrote a paper about this a really long time ago (2016) that may be useful

https://www.usenix.org/conference/hotcloud16/workshop-program/presentation/burns

---

From: 'Tim Hockin' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com>
Sent: Wednesday, December 9, 2020 1:45 PM
To: Rodrigo Campos <rod...@kinvolk.io>
Cc: decarr <dec...@redhat.com>; Seth Jennings <sjen...@redhat.com>; Josh Berkus <jbe...@redhat.com>; kubernetes-sig-node <kubernete...@googlegroups.com>; Kubernetes developer/contributor discussion <kuberne...@googlegroups.com>
Subject: [EXTERNAL] Re: Sidecar: Let's collect use cases for sidecar containers

 

On Wed, Dec 9, 2020 at 6:32 AM Rodrigo Campos <rod...@kinvolk.io> wrote:
>
> (Adding Seth in Cc:)
>
> Hi, we talked about sidecars at yesterday's SIG node meeting. In the
> agenda there are some notes about what we talked

> (https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg%2Fedit&amp;data=04%7C01%7Cbburns%40microsoft.com%7Cefd4b17f76b447c135d108d89c8bc3d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637431471396240530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=51HyJrNig%2BpYoda3BqcCTaYcl%2Bj%2FKDHJAv%2FN79VmK2g%3D&amp;reserved=0)

>
> Derek will try to finish reading Tim's proposal and we will coordinate
> another meeting in early Jan (it would be great if you can come too,
> Tim!).

Just give me warning and I will come :)

> It is not clear to me if Derek/Seth want to create other proposals,
> but they are definitely busy now. Derek/Seth, feel free to update that
> yourselves :)

We're all busy and this is an enormous and subtle topic.  Don't let me
force anyone's hands.  It has languished this long, it can languish
some more.  It's not like I am personally signing up to implement this
all.


> On Fri, Dec 4, 2020 at 8:03 PM Rodrigo Campos <rod...@kinvolk.io> wrote:
> >
> > On Thu, Nov 26, 2020 at 6:27 PM Rodrigo Campos <rod...@kinvolk.io> wrote:
> > >
> > > (Adding Derek in Cc)
> > >
> > > Hi Tim! Sorry for the delay, I was on PTO.
> > >
> > > I've created a doc that explores a wide range of different _possible_ options (very high level), one is along the lines of what you propose (but I was waiting to align internally here, so haven't published, sorry :-/). Your proposal goes in way more detail than I did and is better polished, so I don't think it has merit to share mine now :)
> > >
> > > As far as I know, there was no one else working on this when I asked on SIG-node.
> > >
> > > Derek, did you or Seth want to explore other alternatives too? Would you be able to review Tim's "pre-proposal" so we can discuss afterwards (not sure if in SIG-node or how)?
> > >
> > > Here is the use cases doc we collected:

> > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M%2Fedit%23heading%3Dh.eo3yz0vw744k&amp;data=04%7C01%7Cbburns%40microsoft.com%7Cefd4b17f76b447c135d108d89c8bc3d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637431471396240530%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=nH6n58PaRc1T5zIG7vi7gEGezCWM3PcsxiROlkKvRYI%3D&amp;reserved=0. At the top of that document it is the link to the current pre-proposals, for now only one from Tim.

> >
> > Friendly ping, Derek?
>
>
>
> --
> Rodrigo Campos
> ---
> Kinvolk GmbH | Adalbertstr.6a, 10999 Berlin | tel: +491755589364
> Geschäftsführer/Directors: Alban Crequy, Chris Kühl, Iago López Galeiras
> Registergericht/Court of registration: Amtsgericht Charlottenburg
> Registernummer/Registration number: HRB 171414 B
> Ust-ID-Nummer/VAT ID number: DE302207000

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To view this discussion on the web visit https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkubernetes-dev%2FCAO_RewYw-8QA3Xmyz86Rnds1uxWGJJD-8mvG3P4duG7cjXD7tw%2540mail.gmail.com&amp;data=04%7C01%7Cbburns%40microsoft.com%7Cefd4b17f76b447c135d108d89c8bc3d7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637431471396250528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=HsG8K2YfybRvRTOWET%2FB6Wnm9rKkuUOvz%2F84ZIxU0Ms%3D&amp;reserved=0.

[Rodrigo Campos]

Hi! Sorry for the late follow-up, I was sick for a long time and
couldn’t send this at the start of 2021.

I won’t have the bandwidth to carry this all the way from the
pre-proposals discussions stage we currently are to a KEP and GA. If
someone else is interested to move this forward, please go ahead. We
have collected good use cases, Tim did a pre-proposal and it will be
great if we can finally find a way forward with this.

I can help with reviews and discussions, but I won’t have the time to
champion this effort. I’d love to help someone to move this, though, I
think we made good progress and would like to help in any way that I
can.


Thanks again for your time,
Rodrigo

[Tim Hockin]

Thanks for your help Rodrigo, and for your willingness to let go. I
know that can be hard.

I hope you're feeling better.

Tim

> --
> You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CACaBj2awsOkO3E07%3DCvVSLFWXX5bEEm5LyaGjaNreRsgxwPcPQ%40mail.gmail.com.

[Matthias Bertschy]

Hello,

I have discussed this with Rodrigo, and if nobody else is willing to move his legacy forward, I volunteer for the journey from KEP to GA.

I think Tim's proposal is good as is, but I am open to another round of discussions before drafting the KEP.

I truly believe in the need for a first class citizen status for sidecars, and lifecycle is an important part of it.

You can count on my energy to drive it forward, and of course if anyone wants to help, feel free to let me know.

Best regards,

Matthias

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CACaBj2awsOkO3E07%3DCvVSLFWXX5bEEm5LyaGjaNreRsgxwPcPQ%40mail.gmail.com.

[Rodrigo Campos]

Hi!

I've created a new pre-proposal exploring the idea of using a DAG for
intra-pod container startup dependencies:
https://docs.google.com/document/d/15Ker0cm6n3auAy_lYmvthilhTxF8OVisjl_52nTQJBE/edit#heading=h.g63xga7g2qzb

I've added a link to it also in the Pre-proposals section of the
use-cases doc: https://docs.google.com/document/d/1Drw9C_Ljpcr4X9UPLvms1fn8uMRnTfJLb-xipgX4C1M/edit#heading=h.t08dvudw5b1r

Please let me know what you think and I would be more than happy to
join any meeting to discuss this further :)



Best,
Rodrigo

Geschäftsführer/Directors: Benjamin Owen Orndorff