Best Method to Customize Image on GKE

[ch...@oakleon.com]

On GCE one can use a startup script to modify the VM environment, but on GKE I can only see the possibility to use a custom source image. I would like to, for example, modify security limits (nofile) for docker processes. Is using a custom source image the best way to accomplish this type of OS-customizing task?

Thanks,
Chris

[Samuel Cozannet]

+1 on the question. 

The current method is very misleading. If you add a startup script as a project metadata, and start instances in GCE, and, in the same project, a GKE cluster, the cluster fails to deploy. I assume this is because the mechanic of GKE is actually using startup scripts to install k8s. 

But this is a big problem for one willing to customize the deployment. 

Any official answer much appreciated. 

[George]

Hello Chris,

As you suggested, the best approach is using a custom source image.

Samuel, what is the key that you are using for the startup script? Make sure it doesn't conflict with other scripts as documented here.

I hope this helps.

Sincerely,

George

[Ju Go]

Hi,

I stumbled upon this topic when desparately searching for ways to run a custom VM image (or customize one of the two existing images you can choose fromwhen creating your cluster) with GKE.

In the GKE cluster creation wizard, I can only choose two images (cos and ubuntu).

Motivation is that I need to set a certain host-level parameter to make Elasticsearch work on the cluster. Basically it is just one command

sysctl -w vm.max_map_count=262144

(see https://www.elastic.co/guide/en/elasticsearch/reference/5.2/vm-max-map-count.html if you want details).

Without this setting, my Elasticsearch pods will crash on startup.

I got it to run by using a NON-autoscaling cluster, logging in to each worker node, and change the setting manually (by also adding it to /etc/sysctl.d/00-sysctl.conf file).

Alas, this gets annoying really fast and will obviously not work for autoscaling clusters.

Next I tried to create a regular (non-GKE) instance from the cos image manually, then change the setting, and then use this instance's disk to create a custome image.

That worked, but  the resulting image would not show up in the "create cluster" wizard.

Alas, I also noticed that an overlayfs is used for /etc in the cos image, so my change wouldn't be persisted anyway - which also means that the manual change I made on my GKE nodes would also not survive a reboot.

So is there a way to create and select one's own VM image for GKE?

Regards

JuGo

[Tomasz W]

A possible workaround is to use a DaemonSet:

https://stackoverflow.com/questions/37163549/correct-way-to-define-k8s-user-startup-script

https://cloud.google.com/kubernetes-engine/docs/concepts/daemonset