Cluster API

440 views
Skip to first unread message

Kris Nova

unread,
Jul 31, 2017, 5:55:33 PM7/31/17
to kubernetes-sig-cluster-lifecycle
Hi,

So myself and a handful of others are interested in proposing, designing, and maybe even implementing a Kubernetes infrastructure API that we are calling (working name) the cluster API:
  • Weston Hutchins 
  • Robert Bailey
  • Justin Santa Barbara
  • Michael Danese
  • Jacob Beacham
  • Alexis Richardson
  • Ilya Dmitrichenko
  • Luke Mars
(Note: these are just the names on the original thread, and I'm trying not to put words in anyone's mouth. Some folks on this list haven't even responded to the first thread)

The original thread was to gauge interest on this idea that has grown from an impromptu lunch at the Google office into this larger effort. 

The Problem:

As it stands the infrastructure community in Kubernetes in fragmented. There are a number of projects and efforts that help deploy, manage, and abuse infrastructure for Kubernetes to run on top of. 

This causes a steep learning curve as every project has to independently be evaluated, and creates confusion to a user hoping to onboard onto Kubernetes (as infrastructure is commonly the first thing that needs to be solved)

Furthermore (as experienced first hand in sig-cluster-lifecycle) the vernacular around each of these projects is highly dependent on the project, and creates a bit of confusion while trying to discuss basic infrastructure ideas that have multiple pseudonym's for the resources.

The Proposal:

Tools like kops, kubicorn, terraform, etc all have a concept of defining infrastructure and then realizing the infrastructure. This effort would be an attempt to bridge those infrastructure representations into one (un)official Kubernetes object. 

We have seen success with community standards (like CNI) that have helped bring a fragmented community together. I would like to do this with infrastructure. Period.

Keeping in the spirit of Kubernetes, we would hope to start an effort to begin coming together as a community via a process we can all feel comfortable working on.

We (well mostly me) started a repository https://github.com/kris-nova/cluster-api that is serving as our starting point for proposing infrastructure APIs.. we hope this repository can be an open learning/sharing space to start gathering data so we can make educated decisions based on data.

Action Items:

1. Can we steal some time at a sig cluster lifecycle meeting to go over this in a high level and answer any questions?
2. Can we/should we start with a repeating meeting to discuss this further?
3. Can we start working on getting proposals into the repo so we can all look at them on one of our calls?
4. We need to figure out how this cluster API will interface with a potential API coming out of Kubeadm

Please let me know how this sits with everyone! Would love to hear thoughts both for/against this idea.

Cheers
Kris

TLDR; There are some of us, who want to entertain the idea of coming together as a community and defining an infrastructure API. They are no bounds to what this API can/can't contain.. so we are asking for data to start looking at this in a more concrete way. We plan on figuring out organizational details on the fly over the next week or so.



-- 
Kris Nova

dan...@platform9.com

unread,
Jul 31, 2017, 7:04:40 PM7/31/17
to kubernetes-sig-cluster-lifecycle
I hope I'm not getting ahead of the conversation, but I am curious about the goals of this effort. It sounds like one goal is designing a set of objects/schemas that cluster management tools can standardize on. Is maintaining a reference implementation another goal?

In any case, I look forward to participating.

Daniel

Jimmy Zelinskie

unread,
Jul 31, 2017, 7:18:11 PM7/31/17
to kubernetes-sig-cluster-lifecycle
Hey,

After briefly looking at some of the examples in the linked GitHub repository, I feel like this might have a slight overlap for Service Catalog. SIG-Service Catalog is essentially trying to provide a native way for Kubernetes to interact with Open Service Brokers which are often APIs for creating resources on a cloud provider. If left unattended, it's quite possible that Kubernetes would get two different representations of the same cloud objects, one for the infrastructure-level (Kubernetes) and one for the application-level (my apps running on top of Kubernetes). If this happened, I think it would be very confusing and would feel as fragmented as the current status quo. We should make sure there's an effort to see if it's possible unify these two SIGs' concepts of resources outside of Kubernetes.

Please let me know if I'm missing something or if this sounds crazy.

Thanks,
Jimmy

Kris Nova

unread,
Jul 31, 2017, 7:34:45 PM7/31/17
to Jimmy Zelinskie, kubernetes-sig-cluster-lifecycle
Jimmy,

It was my understanding that sig service catalog focused directly on application layer APIs? This proposal is for the underlying infrastructure layer of different types of Kubernetes clusters. 

Does sig service catalog deal with infrastructure? We certainly don't want to impede or fragment the community more, that is the opposite of our goal here.

Kris

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/e15e0842-6c93-456b-816e-33e05c18c02a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Kris Nova

Jimmy Zelinskie

unread,
Jul 31, 2017, 8:45:31 PM7/31/17
to kubernetes-sig-cluster-lifecycle
Kris,

I think may have jumped in without fully understanding your proposal. As a part of describing your types of deployments, there's going to be a description of external resources. I'm proposing that we should experiment in determining if Kubernetes can have some kind of concept for what is "EC2" or "RDS" that is consistent across multiple use-cases. If such a centralized interface existed, it could become trivial to implement features such as labeling cloud resources (using their APIs) such that they could be tracked back into the Kubernetes cluster regardless of where they were created.

>Does sig service catalog deal with infrastructure?
There are definitely resources that are used by both applications and Kubernetes itself: consider a hosted etcd service or blob store service. Don't you think that some users will get jealous of the fact that Kubernetes knows how to provision an etcd cluster for itself, but not for their application?

I may have jumped on the gun on mentioning Service Catalog as it operates as a proxy for mapping an arbitrary Open Service Brokers' resources into Kubernetes objects, but at least having a conversation with this SIG might be interesting since they primarily work on mapping external resources into Kubernetes.

David Oppenheimer

unread,
Aug 1, 2017, 1:54:21 AM8/1/17
to Jimmy Zelinskie, kubernetes-sig-cluster-lifecycle
It seems like there are three layers
- applications
- Kubernetes components (etcd, apiserver, controller manager, kubelet, kube-proxy, fluentd, etc. etc.)
- what I envisioned when I first heard the term "cluster API": provisioning and deprovisioning VMs (on a public or private cloud), load balancers, network disks, etc.


To post to this group, send email to kubernetes-sig-cluster-lifecycl...@googlegroups.com.



--
Kris Nova

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/7cdf14ee-47a4-4c5e-82e7-bbf8b59f074c%40googlegroups.com.

luis....@coreos.com

unread,
Aug 1, 2017, 1:50:13 PM8/1/17
to kubernetes-sig-cluster-lifecycle
Hi Kris,
  As you mention below, like CNI, a "CII" (Common Infrastructure Interface) would be interesting. I look forward to reading your proposal to understand better how this works.

- Luis


On Monday, July 31, 2017 at 5:55:33 PM UTC-4, Kris Nova wrote:

Joseph Jacks

unread,
Aug 1, 2017, 4:38:01 PM8/1/17
to kubernetes-sig-cluster-lifecycle
@David - I roughly agree with this layering.

@Kris - This is an interesting and timely proposal that highlights clearly excessive complexity inherent in K8s' dependency on underlying cloud/infra services. For e.g., I have a sheet (0) with nearly 70 systems which also contain nearly 70 different representations of "cluster APIs" in their own ways. Cluster/node-level primitives that a given cloud/infra provider exposes via APIs or other means (let's just say APIs, for now) vary quite drastically. If the control and unification/normalization/canonicalization of these underlying primitives is to be achieved in a maintainable and durable way inside an overarching bridge/umbrella system like cluster API, it could be useful to learn from different yet similar efforts like jclouds (1) and libcloud (2).  Both of these projects were originally focused on API abstractions atop core IaaS primitives (VM call lifecycle - start/stop/snapshot/pause - LB provisioning/config - DNS - CDN - object/block storage etc.). The problem statement targeted by these efforts is quite similar: abstract away/unify the interface/representation of many different underlying primitives and provide a single interface to simplify management and speed development. One of the challenges I've observed over the years in using and following these "multi-cloud API" approaches is that the abstractions underneath are constantly changing, making it very hard to really normalize to a common set of higher level "stable" representations. This might just be due to the fact that these projects were created well before it was clear who the major/dominant cloud providers would be, i.e. ~4-6 years ago there were probably ~10 public clouds (not the "dominant" three, like today) and maybe 4-6 private cloud frameworks (Eucalyptus, OpenStack, CloudStack, VMware vCD/vFlavorOfTheMonth...) each with core IaaS services all implemented differently exposing their compute products with non-standardized and unique APIs. It might be easier this time around and with K8s, there is a much narrower surface area than needing to abstract the entire suite of IaaS services per cloud. Just some thoughts, HTH! Looking forward to learning more.

To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.



--
Kris Nova

Brian Grant

unread,
Aug 1, 2017, 9:46:08 PM8/1/17
to Joseph Jacks, kubernetes-sig-cluster-lifecycle
I don't think a Kube-like cluster API solves the problem of everyone wanting to use their favorite infrastructure orchestrator / config-management solution, such as CloudFormation, Terraform, Ansible, Juju, or Infrakit, especially in the case that it is a supported product of the provider.

https://xkcd.com/927/

It could fill a gap for tools/systems such as kubicorn and kops that need/want a shared/reusable, portable, K8s-compatible infrastructure API layer.

There are indeed interesting questions, such as how/whether to expose provider-specific features.

BTW, in case you missed it:
https://github.com/kubeup/archon


To post to this group, send email to kubernetes-sig-cluster-lifecycl...@googlegroups.com.



--
Kris Nova

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-lifecycl...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.

Kris Nova

unread,
Aug 1, 2017, 10:18:44 PM8/1/17
to Brian Grant, Joseph Jacks, kubernetes-sig-cluster-lifecycle
@JJ,

Thanks for sharing the data, and the support here. I know this endeavour might end up to be a fools errand, but I am equally set on proving this attempt *wrong* as I would be on proving this attempt *right*. At this point I just want to discover meaningful technical reasons for either doing this or not doing this. At least having direction for these 70 projects would be a huge win. Right now if everyone is continuing to solve the same problem on their own, we might not be working effectively as a community.

@Brian,

I don't think this proposal is meant to replace config management solutions like CF, Terraform, Ansible etc.
I think the idea here is to offer an infrastructure layer abstraction that is inclusive enough to cross many cloud providers, and still expressive enough to give users a sense of mastery over their infrastructure. 
The API attempts to solve the problem of getting infrastructure management tools speaking the same language. 

A user could declare a Kuberenetes infrastructure-like API and use a controller pattern (similar to the patterns in the archon project you linked) to reconcile the infrastructure. This proposal is forward thinking and is catered to next-gen deployment tools like kubicorn, kops, archon, etc... and not necessarily aimed at traditional config management tools.

Historically infrastructure has always been a few evolutionary steps behind the application layer in tech. This proposal is intended to open the door to to declarative API driven infrastructure, that is cloud agnostic. We have seen success like this before in CNI, and in core itself (storage, cloud providers, etc) where defining agnostic resources was a powerful win.

Just to drive the point home here.. This isn't proposing a replacement for working config management tools like CloudFormation, Terraform, Ansible, Juju, or Infrakit.
This proposal is to start guiding a new direction in infrastructure management in Kubernetes, as new tooling inevitably arrises in the future for tools like kops, kubicorn, archon etc..

I know I am a bit crazy, but I still think this might be a meaningful endeavor to consider, despite the risk of potentially becoming the 15th standard. Declarative infrastructure is here, whether we like it or not. I was thinking it it might be a big win if we all start declaring our infrastructure the same way :)

Cheers

Kris


To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsu...@googlegroups.com.

To post to this group, send email to kubernetes-sig-cluster-lifecycl...@googlegroups.com.



--
Kris Nova

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsu...@googlegroups.com.

To post to this group, send email to kubernetes-sig-cluster-lifecycl...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-lifecycl...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/CAKCBhs4kYZ%2BXNrLL9NhqDjt_RujnVrnDveuhpukPXUh54gEaOg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Kris Nova

Justin Garrison

unread,
Aug 2, 2017, 2:39:02 AM8/2/17
to kubernetes-sig-cluster-lifecycle
I like the idea but here are some of my concerns.

Abstracting away VMs may not actually be the goal of the infrastructure. We don't really want 100 VMs, what we really want is 200 cores to do work. How that gets split up depends greatly on instance size.
Bare metal also throws a wrench in what we are abstracting. I'm guessing this API wouldn't provision hypervisors (which then provisions VMs), but trying to abstract a running instance of the kernel also gets blurred when containers share a kernel yet can be the basis for our cluster too.

Common abstraction libraries I've found have mostly failed because 1) common abstractions remove implementation benefits 2) abstractions often try to create standards and standards usually slow the rate of innovation
With 1, if I abstract a load balancer and the GCP load balancer has a feature/setting the AWS load balancer doesn't have how do I abstract that feature?
With 2, Just like Docker was hesitant to drive the OCI forward they didn't want to limit the innovation of their product (along with other reasons) while the rest of the industry wanted something stable to build with. Standards are good, but only if the innovators are willing to join in.

Common APIs such as CNI were successful because they were very narrow in scope and specific in implementation. The CNI was very clear about what it did and it only interacted with schedulers (the platforms had to implement CNI and CNI implementations didn't have to fit the platforms).
If the cluster API had a very small set of goals (VM lifecycle) with few integration points (kubernetes) I could see this being successful and able to expand to other resources that eventually would make up "a cluster". It would have to be very opinionated on when and how it is called and not try to be everything to everyone.

I'll be interested to follow this but may not have a broad enough knowledge to help abstract all the necessary components.

As a side question, how is this different than CLOUD_PROVIDER and the influence that has on kubernetes components? I'm not sure how the work on splitting those out from core kubernetes has been going but I was interested to see what needed to change drastically as an external pluggable system.


On Tuesday, August 1, 2017 at 7:18:44 PM UTC-7, Kris Nova wrote:
@JJ,

Thanks for sharing the data, and the support here. I know this endeavour might end up to be a fools errand, but I am equally set on proving this attempt *wrong* as I would be on proving this attempt *right*. At this point I just want to discover meaningful technical reasons for either doing this or not doing this. At least having direction for these 70 projects would be a huge win. Right now if everyone is continuing to solve the same problem on their own, we might not be working effectively as a community.

@Brian,

I don't think this proposal is meant to replace config management solutions like CF, Terraform, Ansible etc.
I think the idea here is to offer an infrastructure layer abstraction that is inclusive enough to cross many cloud providers, and still expressive enough to give users a sense of mastery over their infrastructure. 
The API attempts to solve the problem of getting infrastructure management tools speaking the same language. 

A user could declare a Kuberenetes infrastructure-like API and use a controller pattern (similar to the patterns in the archon project you linked) to reconcile the infrastructure. This proposal is forward thinking and is catered to next-gen deployment tools like kubicorn, kops, archon, etc... and not necessarily aimed at traditional config management tools.

Historically infrastructure has always been a few evolutionary steps behind the application layer in tech. This proposal is intended to open the door to to declarative API driven infrastructure, that is cloud agnostic. We have seen success like this before in CNI, and in core itself (storage, cloud providers, etc) where defining agnostic resources was a powerful win.

Just to drive the point home here.. This isn't proposing a replacement for working config management tools like CloudFormation, Terraform, Ansible, Juju, or Infrakit.
This proposal is to start guiding a new direction in infrastructure management in Kubernetes, as new tooling inevitably arrises in the future for tools like kops, kubicorn, archon etc..

I know I am a bit crazy, but I still think this might be a meaningful endeavor to consider, despite the risk of potentially becoming the 15th standard. Declarative infrastructure is here, whether we like it or not. I was thinking it it might be a big win if we all start declaring our infrastructure the same way :)

Cheers

Kris

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.



--
Kris Nova

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/CAKCBhs4kYZ%2BXNrLL9NhqDjt_RujnVrnDveuhpukPXUh54gEaOg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Kris Nova

Jesse Brown

unread,
Aug 2, 2017, 12:47:54 PM8/2/17
to kubernetes-sig-cluster-lifecycle
FYI, on the federation sig we've been having some difficulty resolving customer use cases with the existing Federation API design, and have been investigating alternatives.  In particular, my team is working on special purpose tooling for things like geo based load balancing configuration, jurisdiction and policy, etc.  We're still pretty early on with this work. 

As we've been working on designing this tooling, it is becoming apparent to us that we need some kind of cluster registry (a thing that has a list of clusters with associated metadata). Other folks on the SIG have noticed a similar need. 

I don't think it is a catalog per-se (there doesn't seem to be a need for deep inspection of the cluster), really more of a filterable index/central list for things like cluster API server IPs, authn/authz hooks, labels and tags (is this cluster pii safe, is this cluster in a certain country, is this cluster 'staging', or 'prod') with extensible meta data. 

It would probably be helpful in building a catalog/schema (it could extend it?), and could probably use a common cluster API for some things, or maybe provide a common cluster API?

Not sure, just figured I would throw this out there. Thoughts?
Reply all
Reply to author
Forward
0 new messages