Considering the retirement of the KPNG project

[Shane Utt]

Hello all,

Due to lack of activity and the general state of the project (being stale for long periods of time) we have been considering that it unfortunately might be time to archive the KPNG project as doing so would be better reflective of the project's true state and it might help incoming community members avoid confusion.

We really appreciate the effort and energy that came out of this project, I will say I myself attended every meeting for almost a year just because I found it so interesting, the group so fun and the technical discussions engaging. I personally made several long-lasting friendships through this project.

But, it seems may be time. So as far as what that would actually mean practically:

1. we would consider the KEP withdrawn (this is solely on the basis that it was never completed)
2. we would archive the repository and generally update community pages and relevant content to reflect its archived state

This does not preclude the notion that we could ever bring the project back, it's more about accurately reflecting current state. We'll leave this post hanging out for a couple weeks for feedback. If you have thoughts, concerns, and general feedback please let us know here (or contact myself or one of the other leads privately). Otherwise if everyone is in agreement and/or there is no serious new proposal to move forward we'll start making the changes mentioned above.

And again, thank you to everyone involved!

[Rajas Kakodkar]

I had a great time working with everyone on KPNG. 

+1 to Shane I haven’t been able to contribute much off late and I think its a good idea to archive the project. 

Thank you to everyone who contributed to KPNG and specially to Mikael for starting this effort and Jay for getting us involved. 

[jay vyas]

+1 shane !  It's not very well maintained now, ... and is mostly an experiment of the past.  

this  project was fun as hell to hack on and we made alot of good friendships along the way !

thx mikael  and everyone else !!!!! 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/6b7d5c98-16d1-48a0-a796-9802d9d3c377n%40googlegroups.com.

--

jay vyas

[Shane Utt]

Appreciate the feedback. The archival process will start soon, thanks again to everyone!

[Shane Utt]

The following issues and PRs have been created for the archival:

* https://github.com/kubernetes-sigs/kpng/pull/534

* https://github.com/kubernetes/org/issues/5054

* https://github.com/kubernetes/community/pull/7969

Let me know if there are other places that need to be updated that I missed.

Additionally, I was thinking there were a couple of productive follow-ups we could do:

* A blog post that talks about KPNG. If anyone is interested in doing such a post please do reach out.
* A retrospective: a sync where we (SIG Network) get together and discuss what went well, what could have gone better, and overall try to take lessons from the experience.

I think the latter in particular could be very useful.

[Douglas Landgraf]

It's sad to see, but Jay and many others put in a great effort. I'm sure, as a community, we gained a lot of technical knowledge from it. KPNG was nice.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/9447b4a3-7dc9-42fe-a65f-fd3ce7d0709bn%40googlegroups.com.

--

Cheers

Douglas

[Andrew Stoycos]

+1 I had a great time experimenting with eBPF in a safe space on this project, and I enjoyed working with everyone involved.  Thanks for all the work to boot-stap it @jay and thanks for taking care of it's archival @Shane :) 

[Evan Jones]

I was never on this project, but I just happened to take a glance at the repo and saw the "How we work section":

The KPNG group is small but dedicated, because this is a hard project. Your first commit might take weeks or months, especially if you're new to the kube-proxy or to K8s networking. However, we will pair with people wanting to join and make an impact. We pair program every friday at our meetings, and also, informally at other times as well.

Our goal is to:

• make the kube proxy fun to work on and easy to understand
• make the kube proxy extensible from a command line and backend perspective
• learn as much as we can about k8s service networking, and build an upstream community around it

And that goes for people working on KPNG especially. So don't hesitate to join us, but just be ready for a lot of work and low level troubleshooting.

I wanted to give major kudos to everyone on the project for working in this spirit. What a phenomenal way to attract and grow an open source community working in a tough problem space. These are very admirable goals even if the project didn't pan out as hoped.

Best

Evan Jones

Website: www.ea-jones.com

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/8d10f541-a6f3-418b-8d76-d6a079586561n%40googlegroups.com.

[Dan Winship]

On 7/10/24 18:08, Evan Jones wrote:
> I was never on this project, but I just happened to take a glance at the
> repo and saw the "How we work section":
>
> The KPNG group is small but dedicated, because this is a hard
> project. Your first commit might take weeks or months, especially if
> you're new to the kube-proxy or to K8s networking. However, we will
> pair with people wanting to join and make an impact. We pair program
> every friday at our meetings, and also, informally at other times as
> well.
>
> Our goal is to:
>

> * make the kube proxy fun to work on and easy to understand
> * make the kube proxy extensible from a command line and backend
> perspective
> * learn as much as we can about k8s service networking, and build

> an upstream community around it
>
> And that goes for people working on KPNG especially. So don't
> hesitate to join us, but just be ready for a lot of work and low
> level troubleshooting.
>
> I wanted to give major kudos to everyone on the project for working in
> this spirit. What a phenomenal way to attract and grow an open source
> community working in a tough problem space. These are very admirable
> goals even if the project didn't pan out as hoped.

Unfortunately, this is part of *why* the project didn't pan out; nobody
ever did the "boring" parts of the work. People came up with lots of
great ideas, but there was never enough discussion of "does SIG Network
actually want this idea?" or "is this the best way to implement this
idea?" or "is there a bigger idea that would solve both this problem and
related ones?". People implemented a whole bunch of features, but there
was not enough code review or testing, so there was no immediate path to
get any of that code into the main product.

KPNG was sort of like the Mirror Universe SIG Network; it was good at
the things SIG Network has historically been bad at (building community
and excitement, avoiding endless debates) but bad at the things SIG
Network is good at (achieving consensus around decisions, ensuring code
quality). Unfortunately, to be part of a Kubernetes-sized open source
project, you have to be good at the things that SIG Network is good at.
The question is, can we figure out a way to be good at both sets of
things at the same time?

-- Dan

>
> Best
>
> Evan Jones
> Website: www.ea-jones.com <http://www.ea-jones.com>

> that it unfortunately /might/ be time to

> archive the KPNG project as doing so would
> be better reflective of the project's true
> state and it might help incoming community
> members avoid confusion.
>
> We really appreciate the effort and energy
> that came out of this project, I will say I
> myself attended every meeting for almost a
> year just because I found it so interesting,
> the group so fun and the technical
> discussions engaging. I personally made
> several long-lasting friendships through
> this project.
>
> But, it seems may be time. So as far as what
> that would actually mean practically:
>

> 1. we would consider the KEP withdrawn

> (this is solely on the basis that it was
> never completed)

> 2. we would archive the repository and

> generally update community pages and
> relevant content to reflect its archived
> state
>
> This does not preclude the notion that we
> could ever bring the project back, it's more
> about accurately reflecting current state.
> We'll leave this post hanging out for a
> couple weeks for feedback. If you have
> thoughts, concerns, and general feedback
> please let us know here (or contact myself
> or one of the other leads privately).
> Otherwise if everyone is in agreement and/or
> there is no serious new proposal to move
> forward we'll start making the changes
> mentioned above.
>
> And again, thank you to everyone involved!
>
> --
> You received this message because you are
> subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop
> receiving emails from it, send an email to
> kubernetes-sig-ne...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-sig-network/6b7d5c98-16d1-48a0-a796-9802d9d3c377n%40googlegroups.com <https://groups.google.com/d/msgid/kubernetes-sig-network/6b7d5c98-16d1-48a0-a796-9802d9d3c377n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
>
>
> --
> jay vyas
>
> --
> You received this message because you are subscribed to the
> Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to kubernetes-sig-ne...@googlegroups.com.
>
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-sig-network/9447b4a3-7dc9-42fe-a65f-fd3ce7d0709bn%40googlegroups.com <https://groups.google.com/d/msgid/kubernetes-sig-network/9447b4a3-7dc9-42fe-a65f-fd3ce7d0709bn%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
>
>
> --
> Cheers
> Douglas
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com

> <mailto:kubernetes-sig-ne...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-sig-network/8d10f541-a6f3-418b-8d76-d6a079586561n%40googlegroups.com <https://groups.google.com/d/msgid/kubernetes-sig-network/8d10f541-a6f3-418b-8d76-d6a079586561n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com

> <mailto:kubernetes-sig-ne...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/kubernetes-sig-network/CAFtMSC6hZW8C1ABGR1k5UegDt5_J7-GS%3DWQ1Fz%3DRa1oK589XiQ%40mail.gmail.com <https://groups.google.com/d/msgid/kubernetes-sig-network/CAFtMSC6hZW8C1ABGR1k5UegDt5_J7-GS%3DWQ1Fz%3DRa1oK589XiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Evan Jones]

I thought it may have been a contributing factor. Nonetheless, there's still a nugget of goodness in there.

[Mikaël Cluseau]

I'm always a bit afraid of commenting those things, but I guess in this case I have too 😅

Le jeu. 11 juil. 2024 à 15:09, Dan Winship <danwi...@redhat.com> a écrit :

Unfortunately, this is part of *why* the project didn't pan out; nobody
ever did the "boring" parts of the work. People came up with lots of
great ideas, but there was never enough discussion of "does SIG Network
actually want this idea?" or "is this the best way to implement this
idea?"

The KEP was written, and comments were addressed, AFAICT. The biggest "controversy" seemed to be the gRPC API, especially the global model and the idea of allowing kpng to work a relay in front of the apiservers; that actually appeared in the KEP process (with the xDS discussion on distributing objects across clusters).

 

or "is there a bigger idea that would solve both this problem and related ones?"

I can hardly think of a wider idea than "we need to separate k8s business from technical backend implementation". I think this issue needs a solution for the k8s project to keep uniformity of UX across implementation technologies. Since it's not 2020 anymore, things may have changed, but the tough question is: how do SDN vendors (hardware or not) currently ship their kube-proxy equivalent nowadays? What do they have to follow to make sure they're at feature parity and correct?

 

People implemented a whole bunch of features, but there
was not enough code review or testing, so there was no immediate path to
get any of that code into the main product.

e2e tests where green (eg https://github.com/kubernetes-sigs/kpng/actions/runs/6002925095), but from what I understood, it wasn't enough to make sure it was good enough. With the separation of concerns, the term "feature" will then depend on what we speak about. On the business side, we had the basics + topology, which is not much. On the backend side, we had many people trying things and implementing that last mile for multiple technologies:

• nft from me,
• iptables to try to get something we could compare with kube-proxy,
• ipvs, multiple times,
• eBPF,
• userspace,
• windows

To me, this kind of diversity explosion was more a proof that the concept worked than a failure of the project.

The today's archival comes from a "paradoxical mix": excitement of people working on it, lack of interest outside.

As I said on slack, that Shane's decision is the right one as it makes things clear and avoids people working on it while it's not going anywhere at least for now.

As you note, it may be the size: on kube-proxy's side, it's preferred to have small code changes in the main tree; on the vendor/tech-specific implementations side, the target is a smaller community with a more optional feature parity with the upstream.

[Dan Winship]

On 7/11/24 11:00, Mikaël Cluseau wrote:
> I'm always a bit afraid of commenting those things, but I guess in this
> case I have too 😅
>
> Le jeu. 11 juil. 2024 à 15:09, Dan Winship <danwi...@redhat.com

> <mailto:danwi...@redhat.com>> a écrit :

>
> Unfortunately, this is part of *why* the project didn't pan out; nobody
> ever did the "boring" parts of the work. People came up with lots of
> great ideas, but there was never enough discussion of "does SIG Network
> actually want this idea?" or "is this the best way to implement this
> idea?"
>
>
> The KEP was written, and comments were addressed, AFAICT.

The KEP was *started*. It was never completed and merged, and so it was
never actually an agreed-upon plan; it was just an idea for a possible
future plan.

And yes, the KEP process can be terrible at times, and there have been
periods (including the early kpng days) when SIG Network has been
especially not great at tracking KEPs and keeping discussion going and
getting KEPs merged. But the fact that we should have done a better job
reaching consensus on kpng doesn't change the fact that we *didn't*
reach consensus on kpng, and so the kpng WG ended up implementing
something that wasn't what the SIG wanted.

(And this sucks and we need to make sure we don't do this again.)

> People implemented a whole bunch of features, but there
> was not enough code review or testing, so there was no immediate path to
> get any of that code into the main product.
>

> e2e tests were green

In a comment in the KEP PR about the test plan, over a year after the
repo was created, Jay described the e2e tests as "mostly passing" [1].
Maybe he meant "it's passing for most of the backends", but my
impression was that at that time, it was not fully passing even for
iptables.

-- Dan

[1]
https://github.com/kubernetes/enhancements/pull/2094#discussion_r870750644

[Mikaël Cluseau]

Le jeu. 11 juil. 2024 à 18:41, Dan Winship <danwi...@redhat.com> a écrit :

On 7/11/24 11:00, Mikaël Cluseau wrote:
> I'm always a bit afraid of commenting those things, but I guess in this
> case I have too 😅
>
> Le jeu. 11 juil. 2024 à 15:09, Dan Winship <danwi...@redhat.com
> <mailto:danwi...@redhat.com>> a écrit :
>
>     Unfortunately, this is part of *why* the project didn't pan out; nobody
>     ever did the "boring" parts of the work. People came up with lots of
>     great ideas, but there was never enough discussion of "does SIG Network
>     actually want this idea?" or "is this the best way to implement this
>     idea?"
>
>
> The KEP was written, and comments were addressed, AFAICT.

The KEP was *started*. It was never completed and merged, and so it was
never actually an agreed-upon plan;

I didn't know what "boring parts" meant here.

 

it was just an idea for a possible future plan.

Yes, an idea with a tentative implementation. I think it's ok to have some code to explore things, even if it ends in archives.

 

(And this sucks and we need to make sure we don't do this again.)

COVID was probably a major contributor here.

 

In a comment in the KEP PR about the test plan, over a year after the
repo was created, Jay described the e2e tests as "mostly passing" [1].
Maybe he meant "it's passing for most of the backends", but my
impression was that at that time, it was not fully passing even for
iptables.

They've been mostly passing before passing yes.

[Antonio Ojea]

On Thu, 11 Jul 2024 at 19:03, Mikaël Cluseau <mikael....@gmail.com> wrote:
>
>
>
> Le jeu. 11 juil. 2024 à 18:41, Dan Winship <danwi...@redhat.com> a écrit :
>>
>> On 7/11/24 11:00, Mikaël Cluseau wrote:
>> > I'm always a bit afraid of commenting those things, but I guess in this
>> > case I have too 😅
>> >
>> > Le jeu. 11 juil. 2024 à 15:09, Dan Winship <danwi...@redhat.com
>> > <mailto:danwi...@redhat.com>> a écrit :
>> >
>> > Unfortunately, this is part of *why* the project didn't pan out; nobody
>> > ever did the "boring" parts of the work. People came up with lots of
>> > great ideas, but there was never enough discussion of "does SIG Network
>> > actually want this idea?" or "is this the best way to implement this
>> > idea?"
>> >
>> >
>> > The KEP was written, and comments were addressed, AFAICT.
>>
>> The KEP was *started*. It was never completed and merged, and so it was
>> never actually an agreed-upon plan;
>
>
> I didn't know what "boring parts" meant here.
>

There is a common misunderstanding, specially caused by how our
companies are organized, that adding a new feature is just adding the
code.
Adding a new feature means adding the CODE + TESTS + JOBs/INFRA, there
are no separate teams, usually named as engprod or QA or testing,
people that want to add code to the Core need to take this into
account and be prepared to implement all those things and maintain
them.

In addition, there is a team under SIG release that monitors the CI
signal on the blocking and informing jobs and opens an issue
immediately https://testgrid.k8s.io/sig-release because those errors
can block the release, people adding the features need to sign up to
be as contact point and commit to fix the problem before the release
is cut in case is blocking it.

>>
>> it was just an idea for a possible future plan.
>
>
> Yes, an idea with a tentative implementation. I think it's ok to have some code to explore things, even if it ends in archives.
>
>>
>> (And this sucks and we need to make sure we don't do this again.)
>
>
> COVID was probably a major contributor here.
>
>>
>> In a comment in the KEP PR about the test plan, over a year after the
>> repo was created, Jay described the e2e tests as "mostly passing" [1].
>> Maybe he meant "it's passing for most of the backends", but my
>> impression was that at that time, it was not fully passing even for
>> iptables.
>
>
> They've been mostly passing before passing yes.

When we review a KEP we do a thorough review, especially if the impact
on the project is big in terms of complexity and surface area.
I went to verify the tests jobs during this KEP review and I found
that there were a lot of false positives, some tests were failing but
the jobs were green
https://github.com/kubernetes-sigs/kpng/issues/400, that removes
confidence on the proposal.

Independently, to be clear, "tests passing" in Kubernetes means it
MUST BE 10/10 GREEN, NO FLAKES, the jobs in kpng are flake
https://github.com/kubernetes-sigs/kpng/actions/workflows/e2e_tests.yaml

>
> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CAFJQUxjRLHfzi-MWgQiFPDes3mGVA_rpLDTB8W588TEdRen5GA%40mail.gmail.com.

[Mikaël Cluseau]

Then, we needed the testgrid's test conditions.

[Shane Utt]

Hey Team,

It's obvious from reading through this thread that there are a lot of different perspectives on this. The fact that we're at this point, but not really on the same page I think speaks to things that could have gone better not just within the project, but across the SIG during the time this project was active. I really do think a retrospective where we discuss this and try to determine how we as a SIG can better support sub-projects in the future and set them up better for success really could be very important and productive!

If you're all on board, I'm happy to organize such a retrospective so we can have some coffee together, and discuss all the things that went well and what could have gone better.

Cheers,

Shane

[Michael Zappa]

Hello,

I like the idea of the retrospective as I agree that a lot of different perspectives are present. Ideally the retrospective can result in useful action items to ensure we have a healthy SIG. 

[Mikaël Cluseau]

Hi Shane, yes, to me it's 100% post mortem analysis. It will be important to have Jay in too, because he handled a lot.

I hope no one thought I was seeking anything but telling the story from my point of view.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/5b171506-da5f-4e3d-9993-7a944ec0e122n%40googlegroups.com.

[Shane Utt]

Absolutely! Very much appreciate you sharing your experience. This kind of feedback from contributors is invaluable.

[jay vyas]

Hi folks just catching up on this!  Am happy to contrib to a post mortem..... !

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/0c0197d2-f09b-4a1a-803c-1d5e0a788a86n%40googlegroups.com.

[jay vyas]

Hi folks ! 

So i as read through this whole thread.  A few things that come to mind.... Somewhat unrelated - 

- The history of the KPNG project is documented in the notes.... here https://docs.google.com/document/d/1yW3AUp5rYDLYCAtZc6e4zeLbP5HPLXdvuEFeVESOTic/edit....   Theres alot of interesting history there and its interesting to read and look back on.   Like the early debates on diffstore, grpc, and so on.  Or the time mikael figured out that we needed t proto.MarshalOptions{Deterministic: true} to get the hashing of the endpoints fixed ...  If anyone wants to know what its like to try to make a battle hardened kube proxy , thats a good starting point :) 

-  Me and Amim shot nerf darts at ricardo during  KPNGCON https://youtu.be/GT_p2mkbn2E?t=252 .  That slowed the project down by at least a few months while ricardo recovered (the wounds were emotional, as well as physical) 

- I think Shane's recent post on K8s certification - if that was around at the time we were doing KPNG, there could have been a good alternative reality where we could have ended up as a "Certifiable" Kube proxy - that passed all 250 or so k8s sig-net tests + Conformance.    Then the idea of in tree or out of tree would be irrelevant.  Of course, the effort of making such a certiffication might not be worth it given that ... well... how many people are actually trying to rewrite the kube-proxy ?  Im assuming its handfulls, but its not like theres going to be 100s of companies in that business.  Most folks are happy with the stock in tree proxy. 

- The idea of making it "in-tree" to me was always confusing. I think the goal was always (in my mind) to have (like dan said) kinda our  parallel mirror universe of sig-network and see how far we could go. The reality is - not having a large corporate sponsor made it virtually impossible for that universe to continue to co-exist forever.  People came and went.  Nobody was paid to work on KPNG.  It was destined ultimately to lack in the consistency and polish that other initiatives would have in this area.