virtual-kubelet v. sandbox v. multi-tenancy

[Jessie Frazelle]

Hey sig-architecture,

Sup! You don't call, you don't text... just kidding...

I caught up on the discussion from the last sig-architecture meeting
and I wanted to make a few points that I think were missed or
unresolved from the discussion of virtual-kubelet v. sandbox v.
multi-tenancy.

I worked on the initial code for virtual-kubelet and I agree that
there are some familiar aspects but it is still very different.

If sandboxing were the same as virtual-kubelet it would have meant
that we should have just made a CRI implementation instead. But
currently, CRI is limited to one per node. Then a node would be
exclusively reserved for just running "pods" elsewhere, which is a
waste of resources.

CRI implementations also assume all Mounts[1] are just a
container_path and host_path and we could not have the same Volume and
Secret interface controls like we do with the kubelet implementation.
So suffice to say that it just would not work the same as a solution
if we had done it as a CRI implementation.

With regard to multi-tenancy I think you are focusing purely on the
similarities. But when you really dive into the details of a
multi-tenant cluster there is a lot more to consider. All providers of
the virtual-kubelet interface are not equal. This is very similar to
what I pointed out as a problem with the Sandbox API in that all
runtimes are not equal and there is not a way currently to measure and
guarantee that a runtime is actually sandboxing in the way it
promises, but I digress.

Not all providers for virtual-kubelet or that interface will have the
same security guarantees. It is also not focused on security, whereas
multi-tenancy is. It is true that perhaps in this nodeless world a
cluster of that sort _might_ be multi-tenant but that seems like a
side-effect of its creation and not the actual goal of the project.
Whereas the multi-tenancy working-group is hard focused on isolation
and security. Plus this is only one half of the problem with
multi-tenancy. The other half is ensuring the API server is properly
isolated from tenants.

Anyways, just wanted to throw out my thoughts on the differences and
pass it over to Ria to explain more about why the nodeless working
group is important.

To be clear, I don't think this should be lumped into multi-tenancy
since they have different goals but it could be cool to work together
on things if they overlap...

I do think it is cool that the "pods" like ACI/Fargate have
bi-directional isolation in that they keep things in (like a sandbox)
and also keep things from the main kubelet/nodes out... which is very
different from containers today... just saying.

Best,
Jess

[1] https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto#L171

--


Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu

[Jessie Frazelle]

Also the whole bi-directional isolation is just a side-effect of its
creation as well...

[Jessie Frazelle]

Does it have to be called nodeless...

[Ria Bhatia]

Thanks Jess, for the explanation on the differences between our efforts with Virtual Kubelet and multi-tenancy.

Here's a fun graphic to also explain the difference, between the goals, of the two projects visually:

As promised the proposal for the nodeless working group is below:

https://docs.google.com/document/d/1Y1GEKOIB1u5P06YeQJYl9WVaUqxrq3fO8GZ7K6MUGms/edit?usp=sharing

We would love to have SIG-Architecture support our effort with nodeless and sponsor us. Let us know if that's a feasible goal or not. We can also come into office hours next week to discuss the proposal! 

We really appreciate the time y'all have taken to work with us so far and we look forward to working with you further to create a nodeless architecture and design within Kubernetes. Specifically we will work to create a nodeless design for platforms to scale up with agents and Containers-as-a-Service services, alike. 

Thanks,

Ria  

---

From: Jessie Frazelle <m...@jessfraz.com>
Sent: Wednesday, May 30, 2018 7:04:17 PM
To: kubernetes-si...@googlegroups.com
Cc: Brendan Burns; Ria Bhatia
Subject: Re: virtual-kubelet v. sandbox v. multi-tenancy