WG-Creation-Request: WG AI Gateway

[Shane Utt]

Dear Kubernetes Community,

After several rounds of conversation in SIG Network, including a SIG Network mailing list thread on the topic, we are proposing the formation of a new Working Group focused on the intersection of AI and networking. The goal of this group is to pull in experts from the field and further explore the networking and API management aspects of AI. We want to focus mainly on the routing, filters and policies for managing AI traffic that we think will be common for implementations.

Below is our definition of the working group for your considerations:

https://docs.google.com/document/d/10WTdHYW5x2rw6BTgDzW7X-5QNesAh205MuoaUe5-IQg

However we will answer the key questions directly:

1. What is the exact problem this group is trying to solve?

We aim to standardize terminology like "AI Gateway" in Kubernetes and identify common implementation patterns and user requirements, proposing appropriate standards/APIs.

2. What is the artifact that this group will deliver, and to whom?

We'll make proposals to SIG Network, specifically to sub-projects like Gateway API and the Inference Extension where appropriate, or propose new sub-projects. We anticipate potential outreach to other SIGs as our exploration develops as well.

3. How does the group know when the problem solving process is completed, and it is time for the Working Group to dissolve?

In the above document we defined our exit. Our exit criteria include defining key terms and establishing a comprehensive plan for user-needed features.

4. Who are all of the stakeholder SIGs involved in this problem this group is trying to solve?

SIG Network: We're proposing it, as it's an area with a strong networking focus.

But we anticipate interest from other areas, and would like sponsorship from:

SIG Storage: Some features with "semantic" in the name use vector databases for similarity search.

SIG Security: Some of the features (prompt guards) are about enforcing boundaries.

And anyone else interested!

5. What are the meeting mechanics (frequency, duration, roles)?

Weekly or bi-weekly 60m meetings. The WG will have chair(s)/lead(s) responsible for direction and working towards completion. Notes and recordings will be made available. We'll also create a new #wg-ai-gateway Slack channel and Github Discussions.

6. Does the goal of the Working Group represent the needs of the project as a whole, or is it focused on the interests of a narrow set of contributors or companies?

The entire project, as AI systems need networking just like anything else. We actively reached out to multiple people from varied organizations to help ensure we have a broad appeal.

7. Who will chair the group, and ensure it continues to meet these requirements?

We have quite a number of people who have stepped forward, ready to committing to help lead the effort to meet our requirements and to push for completion:

Kellen Swain (https://github.com/kfswain)

Dan Sun (https://github.com/yuzisun)

Keith Mattix (https://github.com/keithmattix)

David Martin (https://github.com/david-martin)

Huamin Chen (https://github.com/rootfs)

Flynn (https://github.com/kflynn)

As a lead of SIG Network, I too intend to help lead the effort to move it forward.

8. Is diversity well-represented in the Working Group?

Our Working Group aims to work through diverse, collaborative participation. We welcome contributors from varied backgrounds, companies, and expertise levels to ensure comprehensive and inclusive development. One of the reasons we want to make this working group is that we want to see Kubernetes become and remain the leading platform for AI/ML workloads, and we believe that collaboration and inclusion with a diverse range of participants is critical to achieve that.

---

Please let us know if there are any questions/comments/concerns or if there's any need for further clarifications!

Thank you,

Shane Utt (https://github.com/shaneutt) (on behalf of the folks in SIG Network interested in AI Gateway)

[Keith Mattix]

+1 to the creation of this WG. Outside of serving LLMs, consuming AI from Kubernetes workloads is going to be a defining pattern for this new era. I believe a WG is an appropriate vehicle for representing the many Kubernetes users (current and future) who need to solve this use-case.

[Rob Scott]

This is exciting, thanks for sharing the proposal Shane! One thing I'd appreciate some clarity on though. The name of the working group "AI Gateway" and the linked doc suggest a fairly tight scope, but this email proposes a much broader scope ("the intersection of AI and networking"). Is the doc a better guide for scope here, or were you hoping for the broader scope suggested in this email?

Thanks!

Rob

--
You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.
To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/34a1838a-bcdf-4686-b78c-1e5b646b445en%40kubernetes.io.

[Shane Utt]

Great question! That language at the top was meant to signal early for the reader that this working group has a strong network focus. The document continues to reflect our working group objectives.

[Xunzhuo]

+1 for creating WG, thanks for sharing it Shane! It is exciting to see the AI Gateway WG, and looking forward to its creation. With the AI Gateway WG creation, I believe that we will bring the next generation of Ingress/Egress to Kubernetes in LLM era.

And as a maintainer of Envoy Gateway, Envoy AI Gateway and vLLM AIBrix Inference GW, I think I can help the AI GW WG move forward as well, to put AI GW into the production at large scale in Kubernetes. I would definitely love to step forward to this direction,  as a co-chair to contribute and lead this WG to grow and work well with other ecosystem! Always ready to committing to help put the effort to meet its requirements and to push for completion.

It is very exciting to see its creation!

Thanks, Xunzhuo!

[Ricardo Katz]

+1, looking forward to it!

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/caeda978-ebe5-42ee-9c5f-dcb82a339ebfn%40kubernetes.io.

[Michael Zappa]

+1 to the formation of this working group. With the increased consumption of AI services from pods themselves this WG will be great! 

[David Martin]

+1 Looking forward to exploring the problems in this space beyond running and serving AI workloads.

[Jintao Zhang]

+1 Thanks for sharing. I'm looking forward to this.

[Jimmy Song]

Thanks for driving this important initiative. I fully support the creation of WG AI Gateway. It’s timely and much needed to standardize and advance how AI traffic is handled in Kubernetes. I’m looking forward to contributing and collaborating with the community on this.

Best,

Jimmy Song

[Cheng Wang]

+1 for the WG; I am interested in this area.

Jimmy Song <roots...@gmail.com> 于2025年7月11日周五 14:26写道：

--

You received this message because you are subscribed to the Google Groups "dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@kubernetes.io.

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/30bddf14-d3f0-4274-b082-71a617ea45e0n%40kubernetes.io.

--

Less is more...

[yingqi ge]

+1  Looking forward to explore more storage possiblities in this scenario

--

[Shane Utt]

It's really awesome to see all this support!

I wanted to mention that I accidentally missed:

* Nir Rozenbaum - https://github.com/nirrozenbaum

* Xunzhuo - https://github.com/Xunzhuo

Both of whom also have offered their support to help lead, run regular meetings, and help build consensus for proposals.

At this point we have more than a full roster of folks ready to be active shepherds of the effort (and that is super appreciated).

[Rayo Wang]

+1,  I’ve been looking forward to it!

[yxx hero]

+1 Thanks for sharing. I'm looking forward to this.

在2025年7月11日星期五 UTC+8 08:05:24<Jintao Zhang> 写道：

[Rayo Wang]

+1, I’ve been looking forward to it.

在2025年7月11日星期五 UTC+8 08:05:24<Jintao Zhang> 写道：

[Amit Kumar]

+1, highly interested. Since I am also currently contributing in AGNTCY. So it’s going to be fun. Please add me to the list.

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/48ee4d35-153d-4843-b8a6-c70bedb62000n%40kubernetes.io.

[Sanggu Han]

+1, Would love to participate in!

2025년 7월 15일 화요일 오후 10시 6분 54초 UTC+9에 Rayo Wang님이 작성:

[Pandiyaraja Ramamoorthy]

+1 for formation of WG.

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/8640d1d2-f45a-4006-8262-79530a24e561n%40kubernetes.io.

[Mikhail Pleshakov]

Looking forward to this group and would like to get involved!

[Ala Solsvig]

+1, please add me! Aens...@gmail.com

Cheers!

Ala

On Jul 17, 2025, at 1:30 PM, 'Mikhail Pleshakov' via dev <d...@kubernetes.io> wrote:

Looking forward to this group and would like to get involved!

You received this message because you are subscribed to a topic in the Google Groups "dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/kubernetes.io/d/topic/dev/XC_8qAyk8W0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dev+uns...@kubernetes.io.
To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/896975a1-a4a3-4921-864b-664d50b06bcen%40kubernetes.io.

[Eric Ji]

+1, fully support the creation of this WG and look forward to contributing to the discussions and development.

[Byonggon Chun]

+1 for wg creation

On Sat, Jul 26, 2025 at 2:11 PM Eric Ji <zhao...@gmail.com> wrote:

+1

On Thursday, July 17, 2025 at 1:00:53 PM UTC-7 Mikhail Pleshakov wrote:

To view this discussion visit https://groups.google.com/a/kubernetes.io/d/msgid/dev/7718d9ed-1f48-4b26-9ea7-1a56374388a9n%40kubernetes.io.