Should the CRI include networking? Should it be the CNI?

[Casey Callendrello]

In light of the ongoing multi-network efforts, I want to chat about something that’s been on my mind: making network management part of the CRI.

Aside: I see there is some discussion taking place around the CDI and the CRI. I would argue this takes basically the same form.

For the purposes of this discussion, a simple strawman:

0. The types and protocol of the CNI remain the API used by the runtime.
1. Networks, like sandboxes, are owned by the container runtime.
2. The runtime exposes this via the CRI. PodSandboxes are attached to one or more networks.
3. There is some kind of fallback to existing behavior.

Separately, there are two possible consumers of this API:

1. The kubelet manages networks
2. An external daemon manages networks

I have written up a more detailed elaboration of this design to pick apart.

Making networks a first-class part of Kubernetes and the CRI enables most or all of the user stories described in the multi-network kep. It is also a logical next step towards deprecating Multus.

It also means an end to the awkwardness around deploying a provider to clusters right now. Along with some proposed enhancements to the CNI STATUS verb, we can finally get rid of all the meaning that is attached to the presence-or-absence of a CNI configuration file on disk.

On the other hand, the situation we have now is tolerable enough for everyone, and we all only have so much time to spend on this.

---

A separate but related discussion is whether or not the CNI should be merged in to the CRI. As it stands right now, CNI is predominantly used by Kubernetes. The set of non-k8s CNI users is small, and it might not be worth it for the standard to be so separate. Perhaps an arms-length situation, not unlike the CRI, would serve the community better?

So, thoughts? Am I off my rocker? Would this be worth the effort?

[lars.g...@est.tech]

Hi,

What is the reason to merge CNI into CRI?

CNI as it is now is an elegant and simple interface that allow simple extensions. From a user point of view to merge CNI is not an improvement. Not at all. From a maintainer point of view it might. But IMO the user perspective is more important.

Regards,

Lars Ekman

[jay vyas]

Im really glad you posted this, because i was super confused about all this after the removal of --cni-bin-dir !  .... 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/1a6d6d43-95ad-4991-9367-d32d557f8d5an%40googlegroups.com.

--

jay vyas

[Ed Warnicke]

As a general rule (and note: all rules have exceptions) it's been my experience that loose coupling and simplicity is good, and when I find myself trending towards creating ever more complex 'glom ball' monoliths, often it's because of architectural mistakes I've made in other areas.  YMMV, but it may be worthwhile to take a good hard look at the *solution* that is pushing things in a direction of less-modularity and ask if it's the right approach to the problems for which it's being formulated.

This is doubly true in the more dynamic world we live in.

Ed

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CACRAZPLOBo1L2-cu_hUBSLef1eaPULtkUpUTXFQh9Hb3P_XS%2BA%40mail.gmail.com.

[Rajat Chopra]

Monoliths - no, by default. Agree.

This proposal however falls in the bucket of API consolidation. CRI triggers the CNI but only has a second hand view of the protocol(through status). In the  imperative context of multiple networks the API consolidation has clear benefits (e.g. declarative intent in pod specs, rather than work through implementation specific annotations).

2c in support.

Cheers.

Rajat

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CAFVSqg3%3DadHkafGW_yUEo8dssC7q05b4h0f1FASz91zLU2WRBA%40mail.gmail.com.

[Antonio Ojea]

The problem that I see with "making network management part of the CRI." is that it means that the Kubelet will have to handle "networks" and pods.

From the multiple discussions that we had in the WG, it seems that there are some use cases that will require more "advanced networks" than linux bridges or vxlans tunnels.

I find it kind of difficult and scalable to introduce all this complexity in the kubelet.

In addition, these will require the container runtimes to be onboard from the beginning, which I find complex, but even more complicated that the multi network features are backported to the versions that our users are using today.

I think that if we think about Kubernetes "multi network" as "adding additional interfaces to current pods attached to new networks" (oversimplifying):

- any existing CNI plugin or networking project can implement multi network easily, watch the APIs and go and create networks and add interfaces to the pods

- it is backwards compatible and easier to implement in the core than having to touch all the core components

What I really think is important are the ergonomics, as a user, if I create a Pod with 2 interfaces, and there is no implementation of the multi-network APIs that Pod should fail or indicate clearly that is degraded ...

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-network/CAPnESopOR9QEQzYXaK0ZKOr9ELG0u0U-dXfTLvTgW5b9RQ7ghg%40mail.gmail.com.

[Ed Warnicke]

My concern is that component consolidation (and API consolidation) intrinsically produces pressure to monolith-ism.    Clean separation of concerns is an important part of loose coupling.

The larger the community of participants, and the more the individuals in that community shift over time, the more difficult it is to maintain the discipline required to not have the pressure of component and API consolidation lead to monolithism.  

I do not understand how the question of declarative intent in pod specs vs annotations intrinsically implies combining CRI and CNI, unless there is an underlying presumption of a  particular 'solution' that requires strong coupling to be viable... in which case my tendency is even more strongly towards to caution re-examination of assumptions.

Clean separation of concerns, keeping low level APIs low level and only talking to their neighboring components, etc is part and parcel of loose coupling.  If we are considering a solution that pushes for large changes like this one, we may want to ask ourselves "Have we taken a wrong turn along the way?"

Ed

[vsk...@gmail.com]

To me at first glance, the idea of having networking become a first-class citizen to K8s seems right (as opposed to "being bolted to the side of containerd/cri-o"). But I had the same questions as Antonio.. "why should the kubelet care to take on this added complexity?"

It might perhaps help to enumerate the benefits, if any, of introducing a network subsystem as part of kubelet. Please bear with me while I float some dumb, uninformed ideas... I wonder if any of these these present legit use cases:

- Policy-based network readiness, say "NetworkReady = (Net-A-Ready and Net-B-Ready)" or NetworkReady = (Net-A-Ready or Net-B-Ready or Net-C-Ready)

- Do we see a strong use case for RemovePodNetwork + SetPodNetwork, perhaps to bounce a malfunctioning interface to working order?

  ** consolidating code in kubelet might make sense in this case as opposed to replicating complicated business logic in multiple runtimes or CNI drivers.

- Can exposing network OperStatus (connectivity) to users help monitoring decisions such as "evict pod to reschedule a new pod with fully functional networking"

  ** Changing pod api must meet a high bar with strong justifications.

- Do we envision the need for users to specify network bandwidth or QoS-level 'requests', and tuning them as the needs grow & shrink, assuming underlying drivers can support such a feature?

- Suppose we had the concept of pod NetworkOper status, can it be a useful tool for kubelet to manipulate during termination help ease some of the pains around traffic getting to pods being terminated?

And, no doubt, bring this in would be some task. I hope we find compelling arguments that would justify changes to kubelet.

My $0.02 in favor ... this just feels like the right next step.

Thanks,

Vinay