Upgrading using a custom CoreOS build

[Charles Allen]

We have a custom coreos build (specific portage overlay) which we use to build our images.

Right now we just build the image and call image_to_vm to create the cloud volume snapshot binary blob. I would like to be able to move to something that is compatible with the coreos partition swapping but am having trouble finding a way to either:

1. Have a non-official update channel local to our organization
2. Have a way to really-really-manually update a coreos node.

Looking at https://coreos.com/os/docs/latest/manual-rollbacks.html it seems it might be possible to simply update the not-in-use USR-? partition from the "new" image and continue from there.

Is there a way to either use a non-official update channel, or to manually push an update into the unused USR partition?

Thank you,

Charles Allen

[Charles Allen]

Found https://gist.github.com/kayrus/b47c466d8bbeb46822b1 from the user forums, but can't really tell if it will work in general going forward.

[Charles Allen]

Also found https://github.com/coreroller/coreroller/issues/5#issuecomment-223045029 

[Alex Crawford]

On 06/09, Charles Allen wrote:
> We have a custom coreos build (specific portage overlay) which we use to
> build our images.
>
> Right now we just build the image and call image_to_vm to create the cloud
> volume snapshot binary blob. I would like to be able to move to something
> that is compatible with the coreos partition swapping but am having trouble
> finding a way to either:
>

> 1. Have a non-official update channel local to our organization
> 2. Have a way to really-really-manually update a coreos node.

>
> Looking at https://coreos.com/os/docs/latest/manual-rollbacks.html it seems
> it might be possible to simply update the not-in-use USR-? partition from
> the "new" image and continue from there.
>
> Is there a way to either use a non-official update channel, or to manually
> push an update into the unused USR partition?

You'll need to either purchase CoreUpdate [1] or use an open source
alternative (later in the thread, you mentioned CoreRoller which appears
to be a nice alternative). Once you run the omaha server, you'll need to
maintain your own CA and regularly publish images to your channels.

Can I ask why you are unable to use the official images?

-Alex

[1]: https://coreos.com/products/coreupdate

[Charles Allen]

Thanks, Alex!

> Can I ask why you are unable to use the official images?

We use Mesos instead of Kubernetes. I had to change a few things to get it to work : https://github.com/metamx/coreos-overlay/commits/build-1353-withMesos 

[Alex Crawford]

On 06/13, Charles Allen wrote:
> > Can I ask why you are unable to use the official images?
>
> We use Mesos instead of Kubernetes. I had to change a few things to get it
> to work :
> https://github.com/metamx/coreos-overlay/commits/build-1353-withMesos

Very cool. Looks like you know your way around portage. We'd happily
accept patches you have that may be generally useful. Adding Mesos to
Container Linux by default would be a harder sell. ;)

-Alex

[Charles Allen]

Thanks! Its shocking how useful "hacking around with gentoo in college" has become.

Regarding mesos by default, I couldn't quite unravel the coreos build process enough to have a great way to install a custom ebuild. If I did emerge-amd64-usr it seemed to un-emerge whenever I ran the final build image command. Though admittedly I may have fat fingered a command somewhere since I was doing it manually at the time. If I want to have a custom ebuild added to the image, is that the intended way to do it?

Thank you,

Charles Allen

[Benjamin Gilbert]

On Wed, Jun 14, 2017 at 7:39 AM, Charles Allen <charle...@metamarkets.com> wrote:

Regarding mesos by default, I couldn't quite unravel the coreos build process enough to have a great way to install a custom ebuild. If I did emerge-amd64-usr it seemed to un-emerge whenever I ran the final build image command. Though admittedly I may have fat fingered a command somewhere since I was doing it manually at the time. If I want to have a custom ebuild added to the image, is that the intended way to do it?

emerge-amd64-usr only builds the binpkg; it doesn't add it to the image.  To do that, add the package as a dependency of coreos-base/coreos in the coreos-overlay repo.

By the by, we've enabled threaded asynchronous DNS in curl, which should fix the longjmp crash you were seeing.

--Benjamin Gilbert

[Charles Allen]

Thanks Benjamin!

So a mesos ebuild in the coreos overlay might be ok? but getting it as a stock dependency of coreos-base/coreos sounds like it will be near impossible :-P

To stop the long jump error we enabled c-ares (adns) for curl : https://github.com/coreos/coreos-overlay/compare/f0b0f9a34711dbfeffd6bf45a78add209e3607ae...drcrallen:dc295ffb776d8dae3603cc87400cc0c9dede883e 

[Alex Crawford]

On 06/14, Charles Allen wrote:
> So a mesos ebuild in the coreos overlay might be ok? but getting it as a
> stock dependency of coreos-base/coreos sounds like it will be near
> impossible :-P

For what it's worth, we have several known users who are running Mesos
on top of Container Linux without modifying the underlying image. I
believe they were able to either run it in a container or run the
installer on the base OS.

-Alex