Request for repository llm-instance-gateway

357 views

Skip to first unread message

Sergey Kanzhelev

unread,

Aug 20, 2024, 12:13:38 PM8/20/24

to kubernete...@googlegroups.com, wg-se...@kubernetes.io, sig-...@kubernetes.io, kubernetes-...@googlegroups.com, sig-n...@kubernetes.io

Hi,

This is a request to create a repository https://github.com/kubernetes-sigs/llm-instance-gateway

As per https://www.kubernetes.dev/resources/services/#repo-requests seeking approvership from SIG Apps and SIG Network leads. Issue is created: https://github.com/kubernetes/org/issues/5106

The sub project was proposed at WG Serving meeting at Aug 6, 2024: WG-Serving Agenda and Notes:

Proposal: [PUBLIC] Dense LLM Serving (+LoRA) for Inference Platform Teams
PoC design: [PUBLIC] Kubernetes LLM Instance Gateway PoC Design
Slides: [PUBLIC] Inference Gateway for Multiplexing models

Project sparked a lot of interest and the need for a space to collaborate. We believe that this subproject will have its own lifecycle and will not be a good fit to be developed in https://github.com/kubernetes-sigs/wg-serving.