Kubernetes AMI discussion

[Stuart Williams]

Hi all,

Apologies for the delay in moving this forwards.

I've started a discussion document with the intention that we'll nail down the content of the AMI and steps required to assemble and test it, here: https://docs.google.com/document/d/1l95wotZnQMLkkaXC46mtyfguepCYVBCv_m-_WiqJyUk/edit#heading=h.heekxxwovp25

Please comment and/or add questions.

I'll follow up again early next week.

s

[Stuart Williams]

Update:

I'll put this on the agenda for the next SIG-AWS call.

s

[Brandon Philips]

Thanks for putting this proposal together.

However, I have several concerns from multiple perspectives:

0. If the goal is to get statistics then we should help make Spartakus more awesome. We use Spartakus for reporting of statistics in Tectonic and it works well.

1. The AWS Marketplace is very tedious to navigate and use as there are no APIs to push updates. So much so that CoreOS stopped pushing an AMI there.

2. Why an AMI? Kubernetes is trivially easy to install with a few lines of user-data on all Linux distros which can easily be done via a cloud formation template button.

3. Why export "discovery" of Kubernetes to the AWS marketplace instead of kubernetes.io docs? We already have a landing page that hits kubernetes.io in the #1 spot if you google "AWS Kubernetes".

4. Who owns the security process? Inevitably there is going to be some issue with the AMI and we have to communicate with the users on upgrading. Does this put burden on the security committee? And why not use the official AMI of $distro that we do choose? What makes Kubernetes special here?

5. How do we avoid the inevitable bike-shedding and brand confusion that will emerge from choosing some distro? Users do care about their distro of choice and will get confused seeing Kubernetes's "official community" AMI isn't their $favoritedistro. As much as I think the distro doesn't matter in a containers and Kubernetes world it is on us as a community to make it easy to install across different places.

I commented on the doc too but sometimes I feel doc comments get lost in the noise. 

Brandon

[Kris Nova]

Brandon,

Your points are well taken. I can't answer a lot of these on a technical level as I'm just helping to shepherd the project. I think these are valid concerns and already see some of the bike shedding happening. I think the original idea was based around standardizing the AMI for the k8s on AWS story on a community level in order to create a sense of uniformity. I don't think the intent was to prescribe this to every k8s+AWS story, although I am now seeing that offering this from a community (CNCF) level, inherently creates such a prescription.

Something that came out of the leadership summit when we side-barred this (myself, Joe, Luke) was the idea to let the project move forward while remaining cautious of red flags along the way. I am wondering if we are starting to see our first red flag...

More concrete responses from my end

0. Yes. I think we all will agree that taking advantage of existing work in an incubator project and joining that effort is meaningful. My understanding was that the AMI metrics gave us something "for free" and that was merely attractive, and not necessarily a design goal from the original discussions. Regardless, if it's metrics we want, I vote we go with Spartakus if it fits nicely into this space.

1. We have 2-3 members of AWS internal helping with the effort, so I imagine we are going to get a bit of the "white-glove treatment" with the entire process, wondering if that would help ease any concerns? 

2. I think the pattern that was observed that lead to the desire to have a standardized AMI was that there are a lot of conflicting tools, that install on many different versions/flavors of Linux. Some might say this is good, others might push for standardization. Opinions.

3. I don't think anyone is saying we exclusively need to have discovery in the AMI. I was imagining the AWS documentation effort growing independently of the AMI, and docs on this process also making it into that effort eventually (E.G. We would have kubernetes.io docs about the AMI)

4. We haven't officially named anyone the security stakeholder. Although we do have 2 things that make this effort meaningful. 1) We have battle tested AMI build logic for the kops AMI that we were planning on importing as a starting place. and 2) The reason we even need a custom AMI to begin with all falls back to kops and how we need systemd to watch the kubelet and needed to resolve some kernel panics we were seeing on various instance sizes in AWS. There wasn't anything "off the shelf" that met our criteria, hence we started baking these into an AMI. 

5. See my note above, bike shedding is already well underway. 

I tried my best to give *my* answers here, but would also like to hear back from others pushing for the project! Also Brandon, if you want to jump on a call and go through this more granularly we certainly can schedule something.

Thoughts everyone? 

Kris

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-aws" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-aws+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-aws@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-aws/6a8b912a-5b6d-48fb-b43b-22792f435d03%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kris Nova

nivenly.com

[Alexis Richardson]

As I understand it the principal novelty here is to enable non vendor owned distributions to exist in a form that fits with wider AWS usage.  Namely AMIs and associated tooling.

To date, the AWS marketplace has only allowed vendor owned AMIs. That's now changed - the CNCF can act as an AMI owner so that eg kops can distribute AMIs in places where AWS folks (solution architects, customers, etc) are comfortable finding them. 

The benefit to customers is choice and access to community projects

None of this should in any way constrain vendor backed distribution projects with their own opinions

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-aws/6a8b912a-5b6d-48fb-b43b-22792f435d03%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kris Nova

nivenly.com

[Brandon Philips]

On Sat, Jun 17, 2017 at 9:36 AM Alexis Richardson <ale...@weave.works> wrote:

As I understand it the principal novelty here is to enable non vendor owned distributions to exist in a form that fits with wider AWS usage.  Namely AMIs and associated tooling.

To date, the AWS marketplace has only allowed vendor owned AMIs. That's now changed - the CNCF can act as an AMI owner so that eg kops can distribute AMIs in places where AWS folks (solution architects, customers, etc) are comfortable finding them. 

The benefit to customers is choice and access to community projects

None of this should in any way constrain vendor backed distribution projects with their own opinions

Sure, I understand this is novel. But, I don't understand why Kubernetes needs to maintain an entire Linux distro AMI. It sounds like kops hit some bugs and that is the motivator?

Brandon

[Brandon Philips]

Hey Kris-

Thanks for the detailed reply. I will reply in-line to a few.

On Sat, Jun 17, 2017 at 9:20 AM Kris Nova <kr...@nivenly.com> wrote:

Something that came out of the leadership summit when we side-barred this (myself, Joe, Luke) was the idea to let the project move forward while remaining cautious of red flags along the way. I am wondering if we are starting to see our first red flag...

More concrete responses from my end

0. Regardless, if it's metrics we want, I vote we go with Spartakus if it fits nicely into this space.

+1

 

1. We have 2-3 members of AWS internal helping with the effort, so I imagine we are going to get a bit of the "white-glove treatment" with the entire process, wondering if that would help ease any concerns? 

Not really. The problem we encountered was the manual process to patch stuff like the Stack Clash stuff that just got disclosed in the Linux Kernel. It was all manual for the AWS Marketplace vs self-service for regular AMIs.

 

2. I think the pattern that was observed that lead to the desire to have a standardized AMI was that there are a lot of conflicting tools, that install on many different versions/flavors of Linux. Some might say this is good, others might push for standardization. Opinions.

Sure, but why an AMI? I feel using or pinning "vanilla" AMIs and having a user-data script which installs kube is equivalent and easier to audit.

 

4.  The reason we even need a custom AMI to begin with all falls back to kops and how we need systemd to watch the kubelet and needed to resolve some kernel panics we were seeing on various instance sizes in AWS. There wasn't anything "off the shelf" that met our criteria, hence we started baking these into an AMI. 

Were the kernel panics not fixed in the upstream distro? Why not?

Is the kubelet work done via /etc/systemd/system? If so why isn't a user-data/cloud-init sufficient?

IMHO, I wouldn't want CoreOS Container Linux to be selected for this AMI as it will really confuse users. We have many many many people using Kubernetes ontop of CoreOS Container Linux and have _zero_ need to bake a custom AMI.

Largely I just don't understand the motivations that lead to this proposal. It isn't metrics, it doesn't seem technical, and it isn't discoverability. What is it?

Brandon

[Brandon Philips]

Hey Y'all-

Kris and I had a brief discussion about this and want to close a few more thoughts:

1. CoreOS Engineers were not super happy with the Marketplace process for both the amount of paper work and speed. Maybe someone can try and take the current kops AMI and run through the process one or two times and report back?

2. If the product goal is "one-click Kubernetes" and also "base image for projects like kops" I think someone needs to try that out and see how it will work. For example the Marketplace builders guide specifically says: " The AMI cannot require custom or user data at instance creation in order to function correctly.  Master/Slave (Head/Worker), Multi-instance, clustered formation or Cloud Formation launches are not currently supported or allowed as part of usage instructions."

Overall, I think we would be better served with a Cloud Formation or something else for "one-click try it out" experiences. But, I think the above two experiments could be ran to disprove that.

Cheers,

Brandon

[Henning Jacobs]

I'm following the whole AMI discussion for a while and I agree with Brandon: I don't really see what we want to achieve with a custom AMI - there are IMHO too many hurdles and unnecessary discussions (OS? Everybody has his favorite one..), let's better invest in high level bundles like CloudFormation or streamlining kops "one-click" experience :-)

I also still think that we should converge to kube2iam+Ingress+ExternalDNS as a full default "one click" experience. IMHO users on AWS most definitely want to use AWS IAM (i.e. kube2iam), but they won't care about AMI as long as the deploy/install experience is streamlined.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-aws" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-aws+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-aws@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-aws/CAD2oYtM4GOQwRtJ7QunL5gp413tOAsH3%2B_6qyZtQnQf-1NgjfQ%40mail.gmail.com.

[greg....@reddit.com]

Hate to echo, but I too wish we'd spend the time on something other than an AMI. I think you care very deeply about the contents of the instances you are operating once you reach a certain stage your company's lifecycle. You may even want to try to use consistent tooling and distributions for the benefit of your expanded/rapidly expanding engineers.

In contrast to an AMI, CloudFormation and Terraform scripts are valuable in that even if I don't use them, I can easily fire them up, poke around, and pull out what I need. Heptio's CF quickstart was incredibly valuable for us, even though we use Terraform. Towards the end of our work, CoreOS's Terraform installer was useful as well despite its usage of Ignition (something we weren't using).

I wish I had had more guidance on stuff like IAM scoping strategies (we still don't have good recommendations for this!), pod-to-IAM stuff, EC2 metadata API's signature stuff and how it can be used to good effect with Kubernetes, easy-to-miss details like disabling source/dest checking, the tradeoffs of VPC routing vs overlays in the context of AWS, the routing table limits and the ways to get around them (with their respective tradeoffs), and etc. You can find bits and pieces of all of these things scattered around, but I'd love to have a thorough "AWS on Kubernetes Operators manual" that we maintained. Kind of like Zalando's: https://kubernetes-on-aws.readthedocs.io/en/latest/

tldr; I feel like the value is in the knowledge and best practices that we would otherwise bake into an AMI. The AMI itself wouldn't be a big win. As an additional note, having a great AWS guide would make AMI creation far more easy. That might even be a good chapter/section/whatever to add.

To post to this group, send email to kubernete...@googlegroups.com.

[duansh...@gmail.com]

As a user of kubernetes on AWS (using kube-aws to provision kubernetes cluster), I feel baking Kubernetes into an AMI is quite strange and probably won't work out. Using tools like kops and kube-aws is so easy and feels to be the right way to go.

[Stuart Williams]

On 22 June 2017 at 05:18, <duansh...@gmail.com> wrote:

As a user of kubernetes on AWS (using kube-aws to provision kubernetes cluster), I feel baking Kubernetes into an AMI is quite strange and probably won't work out. Using tools like kops and kube-aws is so easy and feels to be the right way to go.

The discussion doc[1] attempts to explain the goal, which is not to produce an AMI - it's to put a community-owned Kubernetes entry in the AWS Marketplace. The idea received general support in the SIG-AWS call, which is why I wrote it up.

One way to do this is via an AMI, but a CloudFormation variant is also possible, (I understand the Marketplace supports CloudFormation now).

I note, there is already an AMI in kube-deploy[2]. 

I differentiate in the doc between an ephemeral deployment using CloudFormation and kubeadm (of which there are also existing examples) and one based on kops - which could also be submitted to the Marketplace and matches a pattern recommended for AWS users (EC2 instance as resource/cluster controller).

The Marketplace opportunity is to reach more users and help them get Kubernetes up and running on AWS.

s

1. https://docs.google.com/document/d/1l95wotZnQMLkkaXC46mtyfguepCYVBCv_m-_WiqJyUk/edit#heading=h.heekxxwovp25
2. https://github.com/kubernetes/kube-deploy/tree/master/imagebuilder

 

On Friday, June 2, 2017 at 7:22:58 PM UTC+8, Stuart Williams wrote:

Hi all,

Apologies for the delay in moving this forwards.

I've started a discussion document with the intention that we'll nail down the content of the AMI and steps required to assemble and test it, here: https://docs.google.com/document/d/1l95wotZnQMLkkaXC46mtyfguepCYVBCv_m-_WiqJyUk/edit#heading=h.heekxxwovp25

Please comment and/or add questions.

I'll follow up again early next week.

s

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-aws" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-aws+unsub...@googlegroups.com.

To post to this group, send email to kubernetes-sig-aws@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-aws/af0027a4-eb8d-463a-96e1-90cab6719682%40googlegroups.com.

[Chavis, Brandon]

Hey folks. 

I can appreciate how an AWS Marketplace listing might feel foreign or unusual, but I would encourage you to think about the perspective of companies or individuals who may not be as progressive in terms of open source tool adoption. 

As open source enthusiasts, we can forget that we tend to live in a bit of a bubble, and there are many AWS customers in traditional enterprises, the public sector, and other similar verticals that perhaps wouldn't consider trying a tool like KOPS. 

A Marketplace listing can help validate the AMI in the eyes of those who may otherwise be hesitant to try a tool openly available on Github, and it also provides a more clear path for AWS to apply our marketing power to such an effort. 

We currently have a KOPS blog post in the works, and we'd like to be able to reference the Marketplace listing as well.

Because the Marketplace provides metrics about usage and consumption, we should be easily able to measure this test and understand if the existence of this AMI leads to KOPS adoption or not. 

Just my two cents!

Best,

Brandon Chavis​

Amazon Web Services

---

From: cncf-imag...@lists.cncf.io <cncf-imag...@lists.cncf.io> on behalf of duansh...@gmail.com <duansh...@gmail.com>
Sent: Wednesday, June 21, 2017 9:18 PM
To: kubernetes-sig-aws
Cc: cncf-...@lists.cncf.io
Subject: Re: [cncf-images] Kubernetes AMI discussion

 

[Brandon Philips]

Hey Brandon-

You are advocating for an AWS Marketplace entry that is a single node, like minikube for AWS, right?

That vision doesn't seem in-line with what the rest of the team was thinking here AFAICT. Anyone?

Also, do you have an example of the AWS Marketplace metrics? Might be useful comparing those against what we are collecting for Spartakus.

Cheers,

Brandon

[luis....@coreos.com]

Hi Brandon,

  I'm voicing my concerns I mentioned during our sig-aws meeting last Friday. My concern is that as a community, Kubernetes should be careful on what they mention is the AMI/Distribution to use. This could confuse those users you mentioned since they could think that the AMI is the "approved" model.

  There needs to be a very clear message saying that, just like minikube, this is only for prototyping and trying it out, and that it is highly discouraged that it should be used in production.

  Maybe minikube could be enhanced to deploy on AWS, and provide that model to the user. I think that message is clear in that it is for temporary use only.

- Luis

[Joe Beda]

I had to miss the meeting where we talked about this so I apologize.  I've been avoiding chiming in because of that.

To be honest I'm skeptical of a community AMI also.  Mostly about experience and who is signing up to support this.  An AMI isn't a one time thing but an ongoing commitment.  I'm not sure we have those issues figured out.

If we want the AMI to be useful by itself here is a proposal:

• With no other input, the AMI comes up and creates a single node cluster (using kubeadm?)
• Via a single parameter in userdata (or tag on the instance?) we can disable that behavior and it now becomes a zygot instance that can be used for a kops cluster or multi-node kubeadm cluster or whatever.

Joe

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-aws" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-aws/5d8a2591-451a-4f31-84d1-8c52bbed28c2%40googlegroups.com.