Building CoreOS kernel

[kernel_newbie]

Hi,

This is a question from an absolute newbie to building kernels but could someone point out how to download/build/install a new kernel on and for a coreos host? 

Any pointers/guides would be most welcome!

Thanks in advance!

[Alex Crawford]

You won't be able to change out the kernel of an existing installation, but you can build a custom image altogether. Take a look at https://coreos.com/docs/sdk-distributors/sdk/modifying-coreos/.

-Alex

[kernel_newbie]

This was a very useful pointer, I've had some luck with getting this going, but had a follow up question. Many thanks for this.

Is there some place that I can set coreos.autologin before I do my VM creation so that I automatically drop into the coreos shell?

Thanks in advance!

[kernel_newbie]

This cant be such a hard question! The issue I am facing is that I need to keep respinning the coreos VM to fix a kernel vxlan issue. So setting the autologin will help speed things up a bit for me!

warm regards

[Brian Harrington]

When you run build_image supply the arguments with --boot_args

i.e. ./build_image --boot_args='coreos.autologin'

--Brian 'redbeard' Harrington

[kernel_newbie]

Many thanks! will give this a try.

Regards

[kernel_newbie]

Could someone point me to how a newly built linux kernel can be packaged into a coreos VM?

We need a fix in the vxlan module of linux for which we need to rebuild the kernel. Using the SDK, I was able to get to the linux code under third party, modify it and build it. However, I haven't found a way to get my Coreos VM to pick up the new kernel. Where do I place the new kernel so that the build_image tool can pick it up?

Any help would be greatly appreciated!

Thanks in Advance!

[Brian Harrington]

CoreOS updates are packaged as ebuilds (think Gentoo) inside of our SDK.

As a prerequisite, make sure you have setup an SDK as per here:

https://coreos.com/docs/sdk-distributors/sdk/modifying-coreos/

To make a modification to the kernel perform the following steps (inside of the SDK):

1) Change to the directory:

  ~/trunk/src/third_party/linux

2) Make any needed changes to the _source_ of your kernel, commit them to git and note the hash of the commit.

3) Place your new kernel config in the path:

  ~/trunk/src/third_party/coreos-overlay/sys-kernel/coreos-kernel/files/

4) Change to the directory:

  ~/trunk/src/third_party/coreos-overlay/sys-kernel/coreos-kernel

5) Make a copy of the ebuild file (as of this email it's coreos-kernel-3.16.2-r8.ebuild) and make sure to increment the release value (i.e. cp coreos-kernel-3.16.2-r8.ebuild coreos-kernel-3.16.2-r9.ebuild)

6) Modify your new ebuild file and note the hash of your commit.  This value should be placed in the variable "CROS_WORKON_COMMIT"

7) From here you should be able to follow all of the steps in the SDK documentation in regards to building packages, then the image, etc.

--Brian 'redbeard' Harrington
--CoreOS

[kernel_newbie]

These steps seem to work only if the modified kernel code is checked into github. I was not able to make this work with a private kernel.

I could modify the linux code under ~/trunk/src/third_party, compile it  and get a vmlinux image. However, modifying the ebuild file does not seem to make any difference and the build scripts stubbornly pick up whatever they were picking up earlier.

I'm sure I'm missing something, but if someone can give me instructions on how I could package the coreos VM with my custom kernel, that would be very helpful. This code used to build this kernel will not be available anywhere other than on the local machine.

Thanks in advance!

[Brandon Philips]

On Fri, Oct 31, 2014 at 11:22 AM, kernel_newbie <uday.n...@gmail.com> wrote:
> These steps seem to work only if the modified kernel code is checked into
> github. I was not able to make this work with a private kernel.
>
> I could modify the linux code under ~/trunk/src/third_party, compile it and
> get a vmlinux image. However, modifying the ebuild file does not seem to
> make any difference and the build scripts stubbornly pick up whatever they
> were picking up earlier.
>
> I'm sure I'm missing something, but if someone can give me instructions on
> how I could package the coreos VM with my custom kernel, that would be very
> helpful. This code used to build this kernel will not be available anywhere
> other than on the local machine.

Generally what we do is add a patch to the ebuild. You can also use
the cros_workon tool to tell the SDK to use your local branch of the
kernel tree.

Cheers,

Brandon

[Christopher Armstrong]

The next CoreOS alpha has kernel 3.17.2: https://github.com/coreos/coreos-overlay/pull/939

Not sure on the timing, but it should be just a few days. That's probably a lot easier than trying to build it yourself.

Chris Armstrong
Head of Services
OpDemand / Deis.io

GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/

On Mon, Nov 3, 2014 at 12:06 PM, kernel_newbie <uday.n...@gmail.com> wrote:

Thanks much for this info, I found that the latest linux kernel in coreos (3.17.2) has the vxlan bug fix that I needed, so there is no need to test a private kernel.

However, the latest stable version of coreos does not have this kernel. So I was trying to build a dev coreos system. 

Now I am unable to enter the SDK chroot, error being:

$ ./chromite/bin/cros_sdk

cros_sdk: Unhandled exception:

Traceback (most recent call last):

  File "./chromite/bin/cros_sdk", line 91, in <module>

    commandline.ScriptWrapperMain(FindTarget)

  File "/home/uday/coreos/chromite/lib/commandline.py", line 412, in ScriptWrapperMain

    ret = target(argv[1:])

  File "/home/uday/coreos/chromite/scripts/cros_sdk.py", line 288, in main

    _ReExecuteIfNeeded([sys.argv[0]] + argv)

  File "/home/uday/coreos/chromite/scripts/cros_sdk.py", line 196, in _ReExecuteIfNeeded

    os.execvp(cmd[0], cmd)

  File "/usr/lib64/python2.7/os.py", line 344, in execvp

    _execvpe(file, args)

  File "/usr/lib64/python2.7/os.py", line 380, in _execvpe

    func(fullname, *argrest)

OSError: [Errno 2] No such file or directory

Any help would be greatly appreciated!

Thanks in advance!

[Gabriele Ricciardi]

Hi,

I'm new in using the CoreOS SDK, and I'm trying to adapt the CoreOS kernel in order to patch it with the RT patch.

Once I setup the SDK and cros_sdk -enter, I cannot find the ~/trunk/src/third_party/linux folder. So I did the following:

- Checkout in ~/trunk/src/third_party from https://github.com/coreos/linux/ the needed kernel branch (3.14.6)
- Patch the kernel with the RT patch
- make menuconfig to enable the RT schedulers and disable unnecessary stuffs
- Commit and take note of the commit hash via git log

(These 4 steps should be equivalent to your point #2)

Then I went on following your instructions, but during the ./build_package step I get the following message:

Determining the location of the kernel source code
 * Unable to find kernel sources at /build/amd64-usr/usr/src/linux
 * Please ensure that the KERNEL_DIR environment variable points at full Linux sources of the kernel you wish to compile against.
 * Unable to calculate Linux Kernel version for build, attempting to use running version
 * Unable to check for the following kernel config options due
 * to absence of any configured kernel sources or compiled
 * config:
 *  - AUTOFS4_FS
 *  - BLK_DEV_BSG
 *  - CGROUPS
 *  - DEVPTS_MULTIPLE_INSTANCES
 *  - DEVTMPFS
 *  - DMIID
 *  - EPOLL
 *  - FANOTIFY
 *  - FHANDLE
 *  - INOTIFY_USER
 *  - IPV6
 *  - NET
 *  - NET_NS
 *  - PROC_FS
 *  - SECCOMP
 *  - SIGNALFD
 *  - SYSFS
 *  - TIMERFD
 *  - TMPFS_XATTR
 *  - !FW_LOADER_USER_HELPER
 *  - !GRKERNSEC_PROC
 *  - !IDE
 *  - !SYSFS_DEPRECATED
 *  - !SYSFS_DEPRECATED_V2
 * You're on your own to make sure they are set if needed.

That suggests me that my kernel configuration is not taken in account. In fact, after the build_image step, I get an image without RT support (checked through /proc/config.gz, the kernel config is the same as an image without any kernel change).

I tried to export KERNEL_DIR=~/src/third_party/linux, but nothing changed.

My questions are:

1) Am I right cloning directly in ~/src/third_party the desired kernel or should I use some command from the SDK to obtain the CoreOS kernel?
2) Am I wrong in some of the performed steps?
3) Am I missing some intermediate steps (like exporting some environment variable, or copying some files)?

Thanks in advance for your help!

[Matt Mathis]

I am trying to inject custom kernel code into CoreOS, to add experimental instrumentation to TCP.  This can not be done in a module: it requires changes to the resident part of TCP.

I am having difficulty connecting the dots from the upstream kernel, through the gentoo build and release process, as wrapped in the CoreOS build containers.

I can inject patches at coreos-overlay/sys-kernel/coreos-sources/coreos-sources-*.ebuild, but I don't see the kernel build and the changes do not propagate into the target.  What am I missing?

Alternatively can I substitute my own local kernel sources?

Thanks in advance!

  

[David Michael]

On Tue, Feb 14, 2017 at 1:46 PM, 'Matt Mathis' via CoreOS Dev
<coreo...@googlegroups.com> wrote:
> I am trying to inject custom kernel code into CoreOS, to add experimental
> instrumentation to TCP. This can not be done in a module: it requires
> changes to the resident part of TCP.
>
> I am having difficulty connecting the dots from the upstream kernel, through
> the gentoo build and release process, as wrapped in the CoreOS build
> containers.
>
> I can inject patches at
> coreos-overlay/sys-kernel/coreos-sources/coreos-sources-*.ebuild, but I
> don't see the kernel build and the changes do not propagate into the target.
> What am I missing?

You have to bump the ebuild revision to declare that there is a new
version of coreos-sources to be built. Also bump coreos-modules and
coreos-kernel, substituting the new coreos-sources ebuild revision in
their ebuild files. Check the Git history for examples.

> Alternatively can I substitute my own local kernel sources?

I wrote some notes on manually building a kernel a while ago:
https://gist.github.com/dm0-/1f656b68491cd22e65ae0f33d4f1dd25 . It's
a bit dated now, but the idea might help. It is also possible to
define other sys-kernel/*-sources projects and build on those, but
that isn't documented by CoreOS and is a bit of a chore.

Thanks.

David

[Matt Mathis]

Thank you!

Does COREOS_SOURCE_REVISION need to be monotonic (within the kernel version), or just not match any recently cached versions?  If I never plan to build a stock kernel, is it risky to stomp on future versions from coreos?

[David Michael]

On Tue, Feb 14, 2017 at 2:42 PM, 'Matt Mathis' via CoreOS Dev
<coreo...@googlegroups.com> wrote:
> Thank you!
>
> Does COREOS_SOURCE_REVISION need to be monotonic (within the kernel
> version), or just not match any recently cached versions? If I never plan
> to build a stock kernel, is it risky to stomp on future versions from
> coreos?

The value of COREOS_SOURCE_REVISION doesn't really matter as long as
it matches a coreos-sources ebuild file. When your coreos-kernel and
coreos-modules ebuilds have newer versions than the stock CoreOS
ebuilds, they will be built and pull in whatever coreos-sources
revision you specified as a dependency.

Thanks.

David

[Geoff Levand]

On 02/14/2017 01:46 PM, 'Matt Mathis' via CoreOS Dev wrote:
> I can inject patches at coreos-overlay/sys-kernel/coreos-sources/coreos-sources-*.ebuild, but I don't see the kernel build and the changes do not propagate into the target. What am I missing?

You need to force a rebuild, so do something like

'emerge-amd64-usr --deep --verbose sys-kernel/coreos-sources'

or

'ECLASS_DEBUG_OUTPUT='on' FEATURES='noclean' emerge-amd64-usr --deep --verbose --debug sys-kernel/coreos-sources'

or increment the ebuild revision like David mentioned.

-Geoff

[Matt Mathis]

I still have not connected the dots all the way from the coreos ebuild files to the resulting coreos image.  

One problem that I have is the final image contains modules and includes under /usr/lib64/modules/4.7.3-coreos-r1 instead of something like /usr/lib64/modules/4.10.1-coreos*   I would guess that some dependency was not updated....   but I don't even see where the modules get generated.   Is the kernel being built twice?

(I do seem to be running the kernel I expect).

Thanks,

--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our services to speak in defiance of unjust governments.   We treat privacy and security as matters of life and death, because for some users, they are.

[Geoff Levand]

On 03/03/2017 03:07 PM, 'Matt Mathis' via CoreOS Dev wrote:
> One problem that I have is the final image contains modules and includes under /usr/lib64/modules/4.7.3-coreos-r1 instead of something like /usr/lib64/modules/4.10.1-coreos* I would guess that some dependency was not updated.... but I don't even see where the modules get generated. Is the kernel being built twice?

Something like this should rebuild all kernel related stuff:

'ECLASS_DEBUG_OUTPUT='on' emerge-amd64-usr --deep --verbose sys-kernel/coreos-sources sys-kernel/coreos-modules sys-kernel/coreos-kernel'

-Geoff