Looking for Kubernetes on AWS users to connect (preferably Berlin/Europe)

[Henning Jacobs]

Hi all,

we (Zalando Tech in Berlin) are currently in the middle of setting up Kubernetes on AWS for production workloads. We are looking for existing experiences and users to connect to (preferably in Berlin or Europe, we can travel ;-)).

Some background:

• We currently have hundreds of AWS accounts with thousands of production EC2 instances (all running Docker, but one container per EC2 instance)

• We are planning to migrate to a more consolidated setup with less AWS accounts, running Kubernetes

• We currently deploy test clusters with Cloud Formation (adapted from kube-aws)

Some open issues we are facing right now:

• Cluster/network setup, multi-AZ or not, etcd

• Kubernetes federation

• Operating a production cluster incl. upgrades

• Autoscaling for nodes

• AWS integration (e.g. Application Load Balancer)

For many of these we have potential solutions, but we would like to hear what you are doing and exchange ideas.

See you around :-)

PS: we have cake

[Brandon Philips]

On Thu, Oct 13, 2016 at 9:28 AM Henning Jacobs <henning...@zalando.de> wrote:

• We currently deploy test clusters with Cloud Formation (adapted from kube-aws)

Why couldn't you use kube-aws directly? What would your ideal tool do differently?

Thanks,

Brandon 

[Henning Jacobs]

Brief answer why we are not using kube-aws directly right now:

• It uses a single master EC2 instance --- we want to have an ASG for the master nodes (probably running with size 1 usually, but having the option for more, e.g. during updates/migrations etc)
• It runs etcd on the master --- we want to run etcd separately (currently we use our own 3 node etcd appliance with DNS discovery (SRV records))
• It does not configure an ELB for the API server --- we want to terminate SSL at ELB in order to leverage existing SSL infrastructure (including ACM)
• It uses a single CF template --- we want to split into at least 3 CF templates to facilitate future upgrades and extra node pools (one for etcd cluster, one for master and one for worker nodes)

We therefore adapted the generated Cloud Formation to YAML and are using our own Senza Cloud Formation tool for deployment (it's not doing any magic, but e.g. makes ELB+DNS config easy).

I'll put our current (hacked) config into some public repo tomorrow...

- Henning

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/WRsVjG7vc9Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Yusuke KUOKA]

Hi,

FYI, we have most of what you've mentioned in our roadmap and some of them already have working PRs:

• Discrete etcd cluster (3+ dedicated etcd nodes)
• HA control plane (multiple EC2 instances and an ELB load balancer, a separate ASG for the master=controller)
• Cluster upgrades

If you'd like to apply the cluster-autoscaler from kubernetes/contrib for node auto-scaling, you would be interested in the issue I've reported.

Regarding the single CF template thing, I suppose splitting templates would be an implication of that issue.

Also, if you'd like to apply AWS native node auto-scaling based on CloudWatch alarms, especially when automatic down-scaling, you would be interested in this PR to avoid small downtime while terminating nodes.

Yusuke

2016年10月14日金曜日 6時58分34秒 UTC+9 Henning Jacobs:

Brief answer why we are not using kube-aws directly right now:

• It uses a single master EC2 instance --- we want to have an ASG for the master nodes (probably running with size 1 usually, but having the option for more, e.g. during updates/migrations etc)
• It runs etcd on the master --- we want to run etcd separately (currently we use our own 3 node etcd appliance with DNS discovery (SRV records))
• It does not configure an ELB for the API server --- we want to terminate SSL at ELB in order to leverage existing SSL infrastructure (including ACM)
• It uses a single CF template --- we want to split into at least 3 CF templates to facilitate future upgrades and extra node pools (one for etcd cluster, one for master and one for worker nodes)

We therefore adapted the generated Cloud Formation to YAML and are using our own Senza Cloud Formation tool for deployment (it's not doing any magic, but e.g. makes ELB+DNS config easy).

I'll put our current (hacked) config into some public repo tomorrow...

- Henning

2016-10-13 21:23 GMT+02:00 Brandon Philips <brandon...@coreos.com>:

On Thu, Oct 13, 2016 at 9:28 AM Henning Jacobs <henning...@zalando.de> wrote:

• We currently deploy test clusters with Cloud Formation (adapted from kube-aws)

Why couldn't you use kube-aws directly? What would your ideal tool do differently?

Thanks,

Brandon 

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/WRsVjG7vc9Y/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[Henning Jacobs]

Hi,

great to hear that kube-aws matures --- we will probably reconsider (we are not sure whether we will stick with our homegrown "hack"), but right now we just want to go forward quickly (and learn along the way).

FYI: I made our current hacked setup available on https://github.com/zalando-incubator/kubernetes-on-aws

- Henning

[anton ivanov]

Hi Henning,

If you're interested, you may want to take a look into Terraform based k8s setup and management, much cleaner than Cloudformation (even yaml based) for me. There is one very nice project here https://github.com/kz8s/tack 

Also of course there is Berlin Kubernetes Meetup and CoreOS meetup, though so far not that many interesting war stories.

Cheers,

A

--

You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Henning Jacobs]

Hi Anton,

I did not know the linked Terraform-based projects, thanks for sharing.

We decided to not use Terraform (let's not discuss pros/cons here, has nothing to do with Kubernetes), but the linked repos definitely provide inspiration.

- Henning

[Henning Jacobs]

To answer my own mail: I now created a repo and Google form to compile a list of companies/organizations using Kubernetes on AWS: https://github.com/hjacobs/kubernetes-on-aws-users

I hope that helps others who are in the same situation to network and share best practices around Kubernetes on AWS..