Adding swap to Kubernetes nodes on the new GCI image

[June Rhodes]

I used to automatically create and mount a swapfile on Kubernetes nodes at startup, however, this doesn't appear to be working anymore on the newer GCI images.

The startup script used to do:

fallocate -l 2G /swapfile

chmod 600 /swapfile

mkswap /swapfile

swapon /swapfile

This obviously no longer works because the root partition is now read-only, but it doesn't even work if I try and place the swapfile under the stateful partition:

fallocate -l 2G /mnt/stateful_partition/swapfile

chmod 600 /mnt/stateful_partition/swapfile

mkswap /mnt/stateful_partition/swapfile

swapon /mnt/stateful_partition/swapfile

Whenever I try and mount a swap file, I just get:

# swapon /mnt/stateful_partition/swapfile

swapon: /mnt/stateful_partition/swapfile: swapon failed: Invalid argument

Is there a recommended way of mounting a swapfile on Kubernetes nodes at startup now?

[Matthias Rampke]

This is aside from the actual question (I'm not very familiar with GCE) but:

Be very careful with swap and containers. We found that having swap available changes the behaviour when a container hits its memory limit – container memory was being paged out even though the host had free memory; we actually expected the container to be killed. This does depend on details of your kernel configuration (both build and boot), and we thought we had set things up so swap is accounted for in memory limits, but apparently we were wrong.

If this swapping is what you want, go right ahead, but don't add swap just because you've always added swap (we did, and it came back to bite us).

/MR

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[June Rhodes]

Yeah, we don't mind swapping here. This is for a non-production cluster (basically our staging/dev cluster), and we just want to keep costs low by using smaller instance sizes.

Kind regards,
June

You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/uhl5R24EOlY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-use...@googlegroups.com.

[Matthias Rampke]

The danger is that a pod spec may "work" in staging but get OOM-killed in production. Just be aware that this may happen (but again, it depends on the configuration of the image, test it out).

[Aditya Kali]

Swap is disabled by default in gci images. To enable it, run:

$ sudo sysctl vm.disk_based_swap=1

OR 

$ echo 1 > /proc/sys/vm/disk_based_swap

After that, your instructions below should work.

Moreover, instead of using /mnt/stateful_partition/ path, you can create your swapfile under /var. Its the same partition as /mnt/stateful_partition, but the path then won't be distro specific.

[Vishnu Kannan]

K8s does not recommend running with swap. 

--

You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Jimmi Dyson]

On Fri, 7 Oct 2016 at 17:11 'Vishnu Kannan' via Kubernetes user discussion and Q&A <kubernet...@googlegroups.com> wrote:

K8s does not recommend running with swap. 

This makes it sound like K8s has become sentient - I'm sure that's not too far off :-b

 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.
To post to this group, send email to kubernet...@googlegroups.com.

[ravi prasad l r]

Matthias, how are disabling swap for the containers ?  Kubelet seems to set MemorySwap (libcontainer config) to -1 by default here:
https://github.com/kubernetes/kubernetes/blob/v1.4.0/pkg/kubelet/dockertools/docker_manager.go#L662

which means  /sys/fs/cgroup/memory.memsw.limit_in_bytes is set to unlimited (mem + swap) inside the container. 

Vish, isn't the Memory+swap should also be set to container.Resources.Limits.Memory().Value() above. Are am i missing something.

-Ravi

[Matthias Rampke]

On Sat, Oct 8, 2016, 00:15 ravi prasad l r <ravipr...@gmail.com> wrote:

Matthias, how are disabling swap for the containers ? 

We disabled and removed all swap on the nodes, and changed new installs not to include any.

Kubelet seems to set MemorySwap (libcontainer config) to -1 by default here:
https://github.com/kubernetes/kubernetes/blob/v1.4.0/pkg/kubelet/dockertools/docker_manager.go#L662

which means  /sys/fs/cgroup/memory.memsw.limit_in_bytes is set to unlimited (mem + swap) inside the container. 

Ah, that would explain why it's behaving different from our older container system.

Vish, isn't the Memory+swap should also be set to container.Resources.Limits.Memory().Value() above. Are am i missing something.

That would match my expectation in the absence of a separate swap limit.

/MR

[Vishnu Kannan]

Ideally, kubelet should be disabling swap by default since it cannot manage swap.  

/MR

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.