Creating CoreOS AMI with modifications

[Peter Kahn]

Hi all,

I'm trying to create a CoreOS AMI using the official images as a base and am running into problems.  We create AMIs containing a specific set of tools at the OS level for kernel driver testing and the base image is so locked down that I'm having difficulties.

Basically, I want to take 2191.5.0 CoreOS and add python3 + a set of pip'd modules.  Because I want to poll the OS kernel, I think I cannot run my testing daemon in a container.

I'm using Hashicorp packer to attempt the following:

- take base AMI

- mount writeable disk as /usr/local with the stock AMI's /usr/local/ content

- install pyenv and setup my python 3.6.1 with its standard binary and python module dependencies

- generate a new AMI

I tried to do this manually initially and it sure seems like /etc/mnt.tab is off limits for modification.  It seems to me that a key design goal of coreOS is its immutability.

Can someone let me know which docs might be the right ones to look at and if this entire approach seems an anathema given the overall design goals of coreOS?

Thanks

PEter

[Andrew Jeddeloh]

You can't change /usr. It's read only and enforced by verity. If you
change /usr the initramfs will refuse to mount it. You could use the
SDK to build new images with the changes, but that would be a lot of
work and break updates. What problems are you hitting running your
workload in a container?

- Andrew

> --
> You received this message because you are subscribed to the Google Groups "CoreOS User" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/8f60c408-3e7b-4209-b725-9cff09b8efc4%40googlegroups.com.

[Peter Kahn]

So, we test a kernel module and run an rpyc daemon to allow the test controller to connect with the test instance. We use pyenv to build a specific version of python with a many dependencies.  Normally, we want to verify both interaction on the OS level and within a container.  Given that coreOS is basically a container running OS without any modifications it might be OK for us run the daemon directly from a container.

I got hashicorp packer to work with a ct to make an image with an ignition that adds in our test user so I think I might be set.  I just need to get the cert for our test certs added and then I'll try our daemon from a container.  I found some of the docs a little confusing in that some steps but as they said most examples could be pulled from the unit tests.

Thanks for the help

Peter