[KCE] Kubernetes Certified Extensions (Tentative Name) - Proposal

[Shane Utt]

Hello my fellow SIG Network folks!

As many of you know, with the Gateway API project, we pioneered a path for building effectively "official" extension APIs using CRDs that are hosted and supported upstream, but remain optional and not part of core.

The Gateway API has demonstrated the benefits of this approach, and since then, other projects like Network Policy have followed suit, using it as a model for their own APIs. Recently, the Multi-Network project has also decided to move out-of-tree and adopt a similar strategy.

While it's possible that APIs developed this way could eventually be integrated back into core, it's clear that a trend is emerging. Given this trend, I believe it is crucial to consider the impact on users: at a minimum, we should create a standardized method for developing, delivering, and using these kinds of APIs to ensure a consistent user experience.

Therefore, I propose that we develop either an official standard or a guide, drawing from the work already established by the Gateway API and Network Policy projects. This could potentially include a template repository with "start here" documentation for new projects. However, I think it's more important to align on the "what" and "why" before diving into the "how."

If you agree with the following:

• What: Creating a standard or guide for developing APIs in a similar way to what we have done organically with Gateway API and Network Policy.
• Why: Because a trend for this has emerged, and a consistent user experience is important.
  

Please show your support! If you have concerns or do not agree, please share your thoughts.

I recognize that this initiative may extend beyond just SIG Network, but I am starting here. If there is general support, my intention is to draft a KEP which covers the "how" and share it broadly across the organization for further feedback.

Looking forward to your feedback!

Shane

[Mike Morris]

+1

I don't think we've figured out a _perfect_ pattern for doing this in Gateway API yet, but aiming for a consistent approach across similarly-situated out-of-tree projects (thinking of SIG-Multicluster's MCS API as a potential next step beyond SIG-Network) could be a big benefit for end users.

-Mike

[Costin Manolache]

While the Gateway API does define this pattern - I am not sure it is at all clear how it impacts users and if it is something that can be generalized.

Even for Gateway, there are big differences between implementations and I don't think we've seen a lot of feedback from users who attempt

to move from one implementation or another, or write helm or manifests that include optional CRDs/features and attempt to use them

on different clusters with different implementations and feature sets. 

So it would be great to be very very clear about the operational and security risks before marketing this as a 

best practice, in particular the core/optional fields and features in a CRD that may be used for security or critical operations, with 

a model of accepting the resource and expecting users to check the status to find which fields worked and which didn't. 

Costin

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/8bfe76a7-44a3-4dcc-a593-2ba76dbd1aa4n%40googlegroups.com.

[Shane Utt]

Thank you for highlighting the importance of considering the downsides and risks associated with this approach, Costin. These will definitely need to be detailed in our larger proposal. The community is aware, and has been sharing problems and issues with the CRD approach, particularly since Gateway API went GA. Somewhat ironically, I myself have several reservations about the CRD approach and still have some lingering desire to see more in-tree development in the future. That said, and in spite of my own issues and reservations, I am committed to ensuring that what we have here and now works as well as is feasible. Hopefully the high level motivation itself sounds good, even if there's more details to iron out. We'd appreciate having you participate in a larger proposal and help us enumerate the downsides and risks.

[Flynn -]

I definitely agree that exploring the idea is a great plan, including taking a good hard look at the downsides. [ ;) ]

​

​ -- Flynn

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/4f76540b-a103-46e6-a4aa-50df04407569n%40googlegroups.com.

[Benjamin Elder]

I think if you're going to call it anything like "Kubernetes Certified" this really should be going through SIG Arch as this will have overlap with approved APIs and the conformance program.

I do think this is a topic we should be talking about, just in broader scope than sig-net

+kubernetes-sig-architecture 

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/4f76540b-a103-46e6-a4aa-50df04407569n%40googlegroups.com.

[Costin Manolache]

Very good point - how to test conformance of all permutations of optional fields, and general usability and complexity if never knowing what you'll get 

are some of the serious problems. 

I think Gateway is solving this by only testing the 'core' APIs - which are required, and vendors can implement any mix of optional APIs they want

and document it somewhere.  Some core APIs also have optional fields - and it is not unreasonable - just the magnitude is different. I mostly 

gave up on Gateway and starting to appreciate Istio APIs far more...

Costin

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CAOZRXm-V0qBmVVXQ-jkLj6g4Jq-%3DZkgwWa1SaUtMgU1-cUazhQ%40mail.gmail.com.

[Nick Young]

I definitely agree that folks are starting to use Gateway API patterns (or at least derivations of them) in other spaces, so it's a good time to stop and think about what's worked and what hasn't. Definitely also agree that SIG-Arch should be deeply involved.

Costin said:

> I think Gateway is solving this by only testing the 'core' APIs - which are required, and vendors can implement any mix of optional APIs they want

> and document it somewhere.  Some core APIs also have optional fields - and it is not unreasonable - just the magnitude is different.

This is incorrect. Gateway API has three levels of API or field - Core, Extended, and Implementation-Specific. 

Core is required of all implementations, with behavior enforced with conformance tests.

Extended is optional, but has behavior enforced with conformance tests.

Implementation Specific is an acknowledgement that some behaviors will not be able to be common enough to be Extended (or Core), but still need to be mentioned in the spec, so that users of the API can make informed decisions.

This is exactly the sort of thing that we need to make clearer, evaluate, and educate people about - especially since someone as familiar with the project as Costin is clearly doesn't understand something so fundamental to how Gateway API works. Which is why I strongly support talking about what we have done in Gateway API more widely.

Rob, Shane and I have been attempting to talk about this in venues like the Gteway API meetings and Contrib Summit for some time, but it would be very useful to be able to do this in an environment that's more accessible to more people.

Nick

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CANbFp%3DazSMSjbms8Mmz%2BQ9LdC48b96bWSV3%3D2JN8XNh%2BWBNgnA%40mail.gmail.com.

[Evan Jones]

+1 here. I'm particularly interested in lessons learned and pitfalls, though that level of detail is probably impractical for a "how to" guide.

Over in the Serving WG there's a nascent proposal to build a "cloud native AI gateway" to route, secure, and monitor traffic for self-hosted and 3rd party LLMs. Some folks have proposed building on top of Envoy with the help of plugins. Others, namely Benjamin who I see here, has raised questions about whether such a proposal is even in scope for the group. Regardless appropriateness or route taken, this sort of guide and the broader discussion invoked by its creation would be very helpful for steering that proposal in the right direction before it gets too far along.

[Mattia Lavacca]

Big +1. In the Gateway API project we learned many valuable lessons on how to develop a CRD-based out-of-tree API, and creating a standard model to centralize such information has great value in my opinion. As others have already pointed out, I think a cross-SIG discussion is worth it. 

Mattia

[Adrian Moisey]

+1 to this idea.

As an end-user the thought of having more "extra things I need to
manually install and maintain" is a scary one, but I totally see the
value that out-of-tree extensions bring to Kubernetes.

I'll definitely get involved with this KEP to see if I can help ensure
that we strike the right balance between being able to iterate and
move quickly, but also provide the stability and quality that users
expect.

> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/2dd3ee90-c2f4-4a1f-96ef-d813f9787654n%40googlegroups.com.

[Surya Seetharaman]

Huge +1.

Having a certain degree of consistency for out of tree APIs to follow a pattern for releasing the APIs, adding new features, graduating the features etc would be awesome.

Thanks Shane for starting this discussion.

Best Regards,

-SS

--