Announcement: kubeadm-dind-cluster (multi-node replacement for hack/local-up-cluster.sh)

[ishve...@mirantis.com]

Hi,

I'd like to announce kubeadm-dind-cluster:

https://github.com/Mirantis/kubeadm-dind-cluster

It was initially based on kubernetes-dind-cluster project [1] by

Dr. Stefan Schimanski, but as of now it diverged too much from it

so I decided to give it a new name.

kubeadm-dind-cluster is a Kubernetes multi-node cluster for developer

_of_ Kubernetes based on kubeadm and DIND (Docker in Docker).

The project has following features:

* Building from local Kubernetes source. There are quite a few

  projects for starting local development k8s clusters, but they

  often consume prebuilt packages which is inconvenient when

  making changes in k8s itself.

* Multi-node clusters. hack/local-up-cluster.sh is often used for

  k8s development but unfortunately it can't start more than one node,

  which means that many e2e tests can't run and there may be problems

  when working on issues that require multi-node setup,

  e.g. scheduler-related.

* Realistic cluster setup. Although based on DIND (Docker in Docker),

  kubeadm-dind-cluster uses kubeadm to launch the cluster which makes

  it closer to real clusters than e.g. hack/local-up-cluster.sh

* Running e2e tests. kubeadm-dind-cluster enables developers to run

  e2e tests locally and provides some convenience tools to do so.

* Fast cluster startup. Initial base image preparation takes some

  time but after that cluster startup / restarts / updates are

  quite fast.

* Both local and remote workflows (more on this below)

* Mac OS X support

See also "Motivation" section in README.md [2].

An important feature of kubeadm-dind-cluster is support for remote

workflows. Some developers prefer to use not very powerful laptops

for their work offloading the heavy lifting such as builds, testing

etc. to remote machines, for example cloud instances or powerful boxes

under the desk. Thanks to work by Joe Beda [3] it's now possible

to employ such workflows for everything that can be done inside the

build container, e.g. building k8s binaries or running unit tests

(for more info, see [4]). The only requirement for the remote machine

is Docker.

With kubeadm-dind-cluster, it's now possible to start DIND cluster on

the same remote Docker. The script takes binaries needed for the

cluster from the build data container directly without even copying

them back to the developer's machine (with exception of kubectl

which is still copied on Linux/built locally on Mac OS X).

The plans for future include:

 * Start some of the addons out of the box, e.g. Dashboard

 * Support bazel builds

 * Make publicly available prebuilt base images for faster

   initial startup

 * Support CNI. At the moment, a docker0 bridge trick inherited

   from kubernetes-dind-cluster is used for container networking,

   but later there will be an option to use CNI to make clusters

   more realistic.

 * Try to find more common grounds with other kubeadm+dind

   projects, such as kubeadm-ci-* projects [5] (kubeadm-dind-cluster

   already utilized kubeadm-ci-dind image but probably

   more close integration/merger is possible) and

   nkube [6] (Nested Kubernetes).

I'd like to thank Dr. Stefan Schimanski (sttts) for his work on

kubernetes-dind-cluster, Ilya Dmitrichenko (errordeveloper)

and Maru Newby (marun) for their work on other kubeadm+dind projects

from where I borrowed some ideas and for their help on

#sig-cluster-lifecycle slack channel. Also I'd like to thank Joe Beda

(jbeda) for the improvements in k8s dockerized builds.

Finally, I'd like to ask whether it makes sense to move this project

to Kubernetes Incubator. In my opinion this maybe useful because

kubeadm-dind-cluster may serve as a good replacement for

hack/local-up-cluster.sh and this would give the project

more visibility.

[1] https://github.com/sttts/kubernetes-dind-cluster

[2] https://github.com/Mirantis/kubeadm-dind-cluster#motivation

[3] https://github.com/kubernetes/kubernetes/pull/30787

[4] https://github.com/kubernetes/kubernetes/tree/master/build-tools#really-remote-docker-engine

[5] https://github.com/errordeveloper/kubeadm-ci-dind

    https://github.com/errordeveloper/kubeadm-ci-packager

    https://github.com/errordeveloper/kubeadm-ci-tester

[6] https://github.com/marun/nkube

[Luke Marsden]

Very cool!

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster...@googlegroups.com.
To post to this group, send email to kubernetes-sig-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/ca05b6b8-2d6f-4240-8025-a9aaa6c52092%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tomasz 'Zen' Napierala]

CC: ContribX

It would be good to collect some initial feedback and than present it on ContribX meeting

Regards,

> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster...@googlegroups.com.
> To post to this group, send email to kubernetes-sig-c...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/ca05b6b8-2d6f-4240-8025-a9aaa6c52092%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Tomasz 'Zen' Napierala
Kubernetes Engineering - Poland

[Devan Goodwin]

This is extremely useful for development, thanks!

[Pires]

Great job!

Pires

[Sandeep Srinivasa]

this is awesome!

On Thursday, November 3, 2016 at 2:01:46 PM UTC+5:30, ishve...@mirantis.com wrote:

* Building from local Kubernetes source. There are quite a few

  projects for starting local development k8s clusters, but they

  often consume prebuilt packages which is inconvenient when

  making changes in k8s itself.

 can you contribute this code to core kubeadm as well ? we have this open feature request https://github.com/kubernetes/kubeadm/issues/3 

[ishve...@mirantis.com]

Well... It's a bit hard to separate from cluster creation code.

It basically creates 'hypokube' image (like hyperkube, but without

the parts kubeadm doesn't need) from the binaries it takes from

the build container, puts it on the nodes and points kubeadm to it.

I'm not sure this should be added to kubeadm, maybe it should be better

kept as an external script?

[Sandeep Srinivasa]

On Nov 23, 2016 4:56 PM, <ishve...@mirantis.com> wrote:

I'm not sure this should be added to kubeadm, maybe it should be better

kept as an external script

well YMMV and I respect your right to decide. But kubeadm being pulled into its own repo is basically because of the ability for it to bootstrap and manage different versions of k8s.  

It stands to reason that building (which is effectively just launching a build container) should be in scope... and something that a lot of us are requesting for in the bug. 

IMHO your code will have a massive impact from inside kubeadm. 

Thanks! 

[Luke Marsden]

It makes sense to me to add this code to a directory in the kubeadm repo, now that kubeadm is becoming a separate repo. What do other people think?

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster...@googlegroups.com.
To post to this group, send email to kubernetes-sig-c...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-cluster-lifecycle/CAOAQpbkm7nQ-jdyj1osteQMoCgRh%3DGsAge4Fm2Mu9Vj-ay5f3Q%40mail.gmail.com.

[Jedrzej Nowak]

I think it should stay as separate project. What benefits do you see of adding it to kubeadm repo? Kubeadm is one tool, dind + kubeadm is another tool with different purpose. When kubeadm + dind will land in incubator there will be no problem to link them more tightly. But I see no benefits of merging it into single project.

On Wednesday, November 23, 2016 at 1:01:38 PM UTC+1, Luke Marsden wrote:

It makes sense to me to add this code to a directory in the kubeadm repo, now that kubeadm is becoming a separate repo. What do other people think?

On Wed, 23 Nov 2016 at 11:57 Sandeep Srinivasa <s...@redcarpetup.com> wrote:

On Nov 23, 2016 4:56 PM, <ishve...@mirantis.com> wrote:

I'm not sure this should be added to kubeadm, maybe it should be better

kept as an external script

well YMMV and I respect your right to decide. But kubeadm being pulled into its own repo is basically because of the ability for it to bootstrap and manage different versions of k8s.  

It stands to reason that building (which is effectively just launching a build container) should be in scope... and something that a lot of us are requesting for in the bug. 

IMHO your code will have a massive impact from inside kubeadm. 

Thanks! 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-cluster-lifecycle" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-cluster-lifecycle+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-sig-cluster-life...@googlegroups.com.

[ishve...@mirantis.com]

As of extracting parts that do k8s build, here's the "hypokube" image

https://github.com/Mirantis/kubeadm-dind-cluster/tree/master/image/hypokube

but actual bits that build k8s / extract binaries from the container

are somewhat tied to DIND case, they're in this script

https://github.com/Mirantis/kubeadm-dind-cluster/blob/master/dind-cluster.sh

but it may be hard to 'generalize' them.