Boot loops with Container Linux 1688.4.0

[Benjamin Gilbert]

Hi all,

We've become aware of an intermittent boot loop issue with Container Linux stable version 1688.4.0, released earlier today.  Instead of booting the kernel, affected machines display a message reading “free magic is broken”, reboot, and repeat the process.  We’ve paused the rollout of upgrades to 1688.4.0 while we investigate this problem, but any machines newly launched with that release may be affected.  If your environment is sensitive to machines failing to boot, we recommend that you continue to launch machines using the previous stable release, 1632.3.0.

We’ll update this thread with further news as it becomes available.  For more information, see https://github.com/coreos/bugs/issues/2284.

--Benjamin Gilbert

[Christopher Snell]

I don't know if this is the same bug but a number of us are encountering issues with 1688.4.0 and kops.  For me, Container Linux 1688.4.0 does boot but etcd does not start.

https://github.com/kubernetes/kops/issues/4837

I'm happy to help debug if someone knows where to look.  I've been away from Container Linux for a while and etcd starts differently than it did back in the day.

Chris

[Stephen Lowrie]

Chris,

Is kops attempting to use etcd2?

1688.4.0 is the first stable version since etcd2 was removed (first version was alpha 1675.0.1). More information @ https://coreos.com/blog/toward-etcd-v3-in-container-linux.html & https://groups.google.com/forum/#!topic/coreos-user/x89Az5blhFw/discussion

Thanks,

--Stephen

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christopher Snell]

Not sure what version it chooses by default but etcd still does not start even when I specify v3.2.15.

[Benjamin Gilbert]

Hi,

We do not yet have a fix for the “free magic is broken” boot loop issue in Container Linux stable 1688.4.0.  While we continue to work to resolve the problem, we are mitigating it in the following ways:

First, we have updated http://stable.release.core-os.net/amd64-usr/current/ to point to the previous Container Linux stable release, 1632.3.0.  This will allow tools that use this directory for provisioning, including Tectonic, to launch machines using a reliable release of Container Linux.

Second, we’ve configured CoreUpdate to resume delivering Container Linux 1632.3.0.  This will allow older versions of Container Linux to upgrade to a more recent version, and will not affect machines already running Container Linux 1688.4.0.

We know that OS instability is disruptive to our users’ environments, and we’re aiming to fully address this issue as soon as we can.  We’ll update this thread again when a fixed stable version has been released, or as additional information becomes available.

--Benjamin Gilbert

[Benjamin Gilbert]

Hi,

We've released Container Linux stable 1688.5.3 to address the boot loop issue in 1688.4.0.  http://stable.release.core-os.net/amd64-usr/current/ has been updated to point to the new release, and CoreUpdate is now rolling out updates to 1688.5.3.

Thanks for your patience as we worked to correct this issue.

--Benjamin Gilbert