[gentoo-user] Kernel modules: initramfs vs. /lib/modules

[marco restelli]

Hi all,
I am trying to understand a bit initramfs and genkernel and I have
few (basic) questions.

I understand that one must have in the initramfs those modules which
are required to boot the system, for instance to access /dev . Now:

- can a module be present both in the initramfs and as kernel module
in /lib/modules ?

- how does genkernel decide which modules to put in the initramfs ?

- can modules included in the initramfs be unloaded once the system is
running, as modprobe -r

- can modprobe load modules from the initramfs ?


Well, clearly I am a bit confused about the topic - I hope somebody
can help me a bit!

Thank you,
Marco

[Rich Freeman]

On Fri, Feb 10, 2017 at 6:58 AM, marco restelli <mres...@gmail.com> wrote:
> Hi all,
> I am trying to understand a bit initramfs and genkernel and I have
> few (basic) questions.
>
> I understand that one must have in the initramfs those modules which
> are required to boot the system, for instance to access /dev . Now:

Sort-of. You need an initramfs if the kernel cannot otherwise mount
/, and /usr (if it isn't on the same filesystem as /). Being able to
mount / is an absolute requirement, there are other ways to go about
mounting /usr.

An initramfs has some benefits even if the kernel could mount /, such
as making it easier or more reliable to identify the correct root
filesystem.

>
> - can a module be present both in the initramfs and as kernel module
> in /lib/modules ?
>

Yes, and typically all of the initramfs modules are present in both.

> - how does genkernel decide which modules to put in the initramfs ?

I can't speak to genkernel specifically, but most initramfs generators
include all modules. Other than space and miniscule load time there
isn't much reason not to.

>
> - can modules included in the initramfs be unloaded once the system is
> running, as modprobe -r

Yes, assuming it isn't in use. Most of the stuff loaded by the
initramfs is probably going to be in use until you shut down (such as
the module for the root filesystem).

>
> - can modprobe load modules from the initramfs ?
>

Only if it is run from within the initramfs. Otherwise this is like
asking whether a binary in a chroot can run something outside the
chroot. Of course, typically all the initramfs modules are also
present in /lib so modprobe will just load the module from there.

>
> Well, clearly I am a bit confused about the topic - I hope somebody
> can help me a bit!
>

An initramfs is really just a chroot in some sense. Though, it would
be more accurate to say that the system you're using after you've
booted is the chroot, and the initramfs was the original host. The
initramfs is the root filesystem when the kernel boots, and it
basically does whatever it needs to to find the real root filesystem,
mount it, and then it deletes its stuff to free up ram, chroots to the
real root, and execs the real init. At that point very little of the
initramfs is left, other than any kernel modules it might have loaded
(which are no different from kernel modules loaded at a later point in
time).

It is just a way to do userspace bootstrapping. Coreboot/libreboot
take this to yet another level. Rather than try to build the smarts
into the kernel to handle every conceivable system configuration, the
kernel provides the driver and some basic logic, and if you want to do
something fancier you use an initramfs and the initramfs can do
anything you can do in linux userspace to find and mount root. It
could download root from a webserver, or launch postfix and wait for
somebody to send the root filesystem as an attachment, or whatever
your imagination can come up with.

Usually, though, it ends up just mounting /dev/sda2 or whatever. Most
distros use an initramfs by default because it is more robust and can
handle things like UUIDs and labels. That way if you plug in a new
drive and your existing drives get renumbered the correct filesystem
gets mounted. That, and it lets them use highly modular kernels
without having to know what kind of filesystem you'll use for /, since
it can just be modprobed at run time. This lets them build all the
drivers as modules, which costs some disk space and a lot of one-time
compile time, but gives the end-user more flexibility without any need
to custom-build a kernel. Gentoo is a bit unusual in encouraging
users to build their own kernels, but of course once you do that then
there is no need to build all the drivers, or use an initramfs for
modules needed to mount /.

Otherwise, there is nothing special about modules loaded from the
initramfs. They're just kernel modules.

--
Rich

[marco restelli]

2017-02-10 13:35 GMT+01:00, Rich Freeman <ri...@gentoo.org>:
> On Fri, Feb 10, 2017 at 6:58 AM, marco restelli <mres...@gmail.com>
> wrote:
>> Hi all,
>> I am trying to understand a bit initramfs and genkernel and I have
>> few (basic) questions.
>

>> - how does genkernel decide which modules to put in the initramfs ?
>
> I can't speak to genkernel specifically, but most initramfs generators
> include all modules. Other than space and miniscule load time there
> isn't much reason not to.

After checking, genkernel copies only some modules into the initramfs,
unless the --all-ramdisk-modules flag is used.

Concerning the rest of your reply: thank you so much, it really helped
me a lot!

Could you suggest any reference about how an initramfs can help making
it easier to identify the correct root filesystem? Does this
functionality overlap with what grub can do, or is something
different?

Thank you,
Marco

[Rich Freeman]

On Mon, Feb 13, 2017 at 5:53 AM, marco restelli <mres...@gmail.com> wrote:
>
> Could you suggest any reference about how an initramfs can help making
> it easier to identify the correct root filesystem? Does this
> functionality overlap with what grub can do, or is something
> different?
>

The dracut references are fairly extensive, but they probably assume
that you already know about something like this. Keep in mind that on
virtually all other distros end-users aren't expected to set up their
own kernels or initramfs so there isn't a lot of general documentation
out there. And even within Gentoo a lot of people seem to avoid an
initramfs, so our own docs may not be as extensive as they could be.

The short version is that the kernel is very limited in what it can
take in the root= option on the command line, and grub and other
bootloaders don't do anything to ID the root filesystem other than
passing whatever root= parameter you specify (I'd be interested in any
info to the contrary).

The kernel itself can't handle UUIDs or filesystem labels. It
actually can handle some situations like lvm on top of mdadm, but that
is about as complex as it gets as far as I'm aware, and even in that
situation there are some limitations.

An initramfs is typically implemented in a bunch of shell scripts
(though it is really a full userspace so in theory you can really have
it do anything). That gives it a lot more flexibility. Typically
they run udev which gives you all the /dev/disk/by-id symlinks and so
on. They can also do things like fsck root before mounting it (though
mounting it read-only and having it fsck itself isn't a terrible
alternative, which is how things work without an initramfs).

Basically an initramfs should be viewed as an extended bootloader.
For more exotic setups they're essentially required (such as
network-based root filesystems). The trend has also been to not add
new root-finding capabilities to the kernel as the initramfs is the
preferred way of doing things. If lvm+mdadm were being built today,
I'm not sure they would make the kernel capable of directly mounting
them as root.

Anytime you see something like root=UUID=* that is being handled by an
initramfs. And of course a UUID is more reliable than a device name,
since the latter can change if you add/remove a device, or maybe even
if your firmware is having a bad day. Identifying devices by UUID
ensures the right one gets found, assuming it is available. If you're
using something like mdadm/lvm there are alternatives to UUID, but the
point is the same, you're using a logical identifier that is based on
what is stored on the disks and not just what port it is connected to.

--
Rich

[Johannes Rosenberger]

On 13.02.2017 12:34, Rich Freeman wrote:
> On Mon, Feb 13, 2017 at 5:53 AM, marco restelli <mres...@gmail.com> wrote:
>> Could you suggest any reference about how an initramfs can help making
>> it easier to identify the correct root filesystem? Does this
>> functionality overlap with what grub can do, or is something
>> different?
>>
> The dracut references are fairly extensive, but they probably assume
> that you already know about something like this. Keep in mind that on
> virtually all other distros end-users aren't expected to set up their
> own kernels or initramfs so there isn't a lot of general documentation
> out there. And even within Gentoo a lot of people seem to avoid an
> initramfs, so our own docs may not be as extensive as they could be.
>

> [...]

There is some very good documentation about crafting your own initramfs
it the Gentoo wiki:
https://wiki.gentoo.org/wiki/Custom_Initramfs
https://wiki.gentoo.org/wiki/Early_Userspace_Mounting

[Daniel Frey]

On 02/13/2017 03:34 AM, Rich Freeman wrote:
> Anytime you see something like root=UUID=* that is being handled by an
> initramfs. And of course a UUID is more reliable than a device name,
> since the latter can change if you add/remove a device, or maybe even
> if your firmware is having a bad day. Identifying devices by UUID
> ensures the right one gets found, assuming it is available. If you're
> using something like mdadm/lvm there are alternatives to UUID, but the
> point is the same, you're using a logical identifier that is based on
> what is stored on the disks and not just what port it is connected to.
>

Are you sure? When I set up my EFI stub kernel on my Surface tablet, I
did not use an initramfs and I use PARTUUID= in the kernel built in init
line and it boots.

I thought I was going to have to use an initramfs but I tried without it
and it boots with no issues.

Dan

[Remy Blank]

Daniel Frey wrote on 2017-02-13 17:34:
> On 02/13/2017 03:34 AM, Rich Freeman wrote:
>> Anytime you see something like root=UUID=* that is being handled by an
>> initramfs. And of course a UUID is more reliable than a device name,
>> since the latter can change if you add/remove a device, or maybe even
>> if your firmware is having a bad day. Identifying devices by UUID
>> ensures the right one gets found, assuming it is available. If you're
>> using something like mdadm/lvm there are alternatives to UUID, but the
>> point is the same, you're using a logical identifier that is based on
>> what is stored on the disks and not just what port it is connected to.
>>
>
> Are you sure? When I set up my EFI stub kernel on my Surface tablet, I
> did not use an initramfs and I use PARTUUID= in the kernel built in init
> line and it boots.

Note that Rich wrote "UUID=", but you used "PARTUUID=". The former
requires an initramfs, the latter doesn't. The details why escape me: if
the filesystem code is built into the kernel (as opposed to a module), I
see no practical reason why the FS UUID couldn't be determined by the
kernel directly.

[Mike Gilbert]

Determining the FS UUID would require scanning all partitions of all
attached disks, and invoking filesystem-specific code to parse out the
UUID.

Determining the PARTUUID only requires scanning the partition table of
each drive, and is only supported for GPT and MBR partition tables.
It's a much simpler task.

[Daniel Frey]

Yes, but I [incorrectly, apparently] assumed that UUID and PARTUUID
would work the same way. Now I'm curious as to why it doesn't, I'm going
to look it up later.

Dan

[marco restelli]

2017-02-13 12:34 GMT+01:00, Rich Freeman <ri...@gentoo.org>:
> On Mon, Feb 13, 2017 at 5:53 AM, marco restelli <mres...@gmail.com>
> wrote:
>>
>> Could you suggest any reference about how an initramfs can help making
>> it easier to identify the correct root filesystem? Does this
>> functionality overlap with what grub can do, or is something
>> different?
>>
>
> The dracut references are fairly extensive, but they probably assume
> that you already know about something like this. Keep in mind that on
> virtually all other distros end-users aren't expected to set up their
> own kernels or initramfs so there isn't a lot of general documentation
> out there. And even within Gentoo a lot of people seem to avoid an
> initramfs, so our own docs may not be as extensive as they could be.

Yes, indeed I have always used gentoo without an initramfs and I am
now looking into it because I want to make a bootable USB.

> The short version is that the kernel is very limited in what it can
> take in the root= option on the command line, and grub and other
> bootloaders don't do anything to ID the root filesystem other than
> passing whatever root= parameter you specify (I'd be interested in any
> info to the contrary).

I have always generated grub.cfg files with grub-mkconfig. In some
cases I see here

search --no-floppy --fs-uuid --set=root <disk-UUID>
linux /kernel-XYZ root=/dev/sda4 ro

As far as I understand it, the first line searches the partition where
the kernel is located identifying it through the UUID. Then the second
line loads the kernel passing /dev/sda4 as the system root.

On the bootable USB stick, with an initramfs, however I have

search --no-floppy --fs-uuid --set=root <disk-UUID>
linux /kernel-XYZ root=UUID=<another-disk-UUID> ro

so now also the root filesystem is identified by its UUID.

Based on your comment that:

> Anytime you see something like root=UUID=* that is being handled by an
> initramfs.

I understand that this parameter is passed by the kernel to the init
script inside the initramfs which then uses "busybox findfs" to
translate the UUID into a device name. Is this correct?

> Basically an initramfs should be viewed as an extended bootloader.
> For more exotic setups they're essentially required (such as
> network-based root filesystems). The trend has also been to not add
> new root-finding capabilities to the kernel as the initramfs is the
> preferred way of doing things. If lvm+mdadm were being built today,
> I'm not sure they would make the kernel capable of directly mounting
> them as root.

OK.

Thank you again,
Marco

[marco restelli]

2017-02-13 12:47 GMT+01:00, Johannes Rosenberger <gen...@jorsn.eu>:
> On 13.02.2017 12:34, Rich Freeman wrote:
>> On Mon, Feb 13, 2017 at 5:53 AM, marco restelli <mres...@gmail.com>
>> wrote:
>>> Could you suggest any reference about how an initramfs can help making
>>> it easier to identify the correct root filesystem? Does this
>>> functionality overlap with what grub can do, or is something
>>> different?
>>>
>> The dracut references are fairly extensive, but they probably assume
>> that you already know about something like this.
>>

>> [...]
>
> There is some very good documentation about crafting your own initramfs
> it the Gentoo wiki:
> https://wiki.gentoo.org/wiki/Custom_Initramfs
> https://wiki.gentoo.org/wiki/Early_Userspace_Mounting

Johannes, these documents are indeed very informative, thank you for
pointing them out.

Marco

[Rich Freeman]

On Wed, Feb 15, 2017 at 5:58 AM, marco restelli <mres...@gmail.com> wrote:
>
>> The short version is that the kernel is very limited in what it can
>> take in the root= option on the command line, and grub and other
>> bootloaders don't do anything to ID the root filesystem other than
>> passing whatever root= parameter you specify (I'd be interested in any
>> info to the contrary).
>
> I have always generated grub.cfg files with grub-mkconfig. In some
> cases I see here
>
> search --no-floppy --fs-uuid --set=root <disk-UUID>
> linux /kernel-XYZ root=/dev/sda4 ro
>
> As far as I understand it, the first line searches the partition where
> the kernel is located identifying it through the UUID. Then the second
> line loads the kernel passing /dev/sda4 as the system root.
>
> On the bootable USB stick, with an initramfs, however I have
>
> search --no-floppy --fs-uuid --set=root <disk-UUID>
> linux /kernel-XYZ root=UUID=<another-disk-UUID> ro
>
> so now also the root filesystem is identified by its UUID.

Correct. Just note that "root" in GRUB lingo is the filesystem that
contains all the grub binaries, the kernels, and so on. That is
typically /boot on a linux system. Unless they happen to be the same
filesystem it isn't the same root you pass to the kernel. If it were
then you would have the same UUID in the second example.

>
>> Anytime you see something like root=UUID=* that is being handled by an
>> initramfs.
>
> I understand that this parameter is passed by the kernel to the init
> script inside the initramfs which then uses "busybox findfs" to
> translate the UUID into a device name. Is this correct?
>

I suppose that is one way it could be done, but of course it could be
implemented in other ways. As far as I can tell Dracut does not use
busybox findfs. I didn't do a careful read but a quick look at the
source suggests that it is actually using udev and referencing
/dev/disk/by-uuid and so on. I suspect the logic is that if udev
could find the root device on the host when the initramfs was being
built, then it should be able to find it in the initramfs if the same
software is used to do it.

--
Rich

[Ian Zimmerman]

On 2017-02-15 10:19, Rich Freeman wrote:

> I suppose that is one way it could be done, but of course it could be
> implemented in other ways. As far as I can tell Dracut does not use
> busybox findfs. I didn't do a careful read but a quick look at the
> source suggests that it is actually using udev and referencing
> /dev/disk/by-uuid and so on. I suspect the logic is that if udev
> could find the root device on the host when the initramfs was being
> built, then it should be able to find it in the initramfs if the same
> software is used to do it.

There's something unsettling in running udev from the initramfs though,
or any daemon really.

This actually came up at one of my past jobs ... I was responsible for
the plumbing in our appliance and for some reason (possibly even this
one) it was convenient to run udev early but I kept saying "... but this
is ODD!"

Didn't end well. Anyway, do you happen to know how findfs is
implemented? Does it actually read the UUID and label off the
partitions itself?

--
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign: http://cr.yp.to/smtp/8bitmime.html

[Neil Bothwick]

On Wed, 15 Feb 2017 23:19:21 -0800, Ian Zimmerman wrote:

> > I didn't do a careful read but a quick look at the
> > source suggests that it is actually using udev and referencing
> > /dev/disk/by-uuid and so on.

The second paragragh of the dracut home page would tell you the same ;-)

> There's something unsettling in running udev from the initramfs though,
> or any daemon really.
>
> This actually came up at one of my past jobs ... I was responsible for
> the plumbing in our appliance and for some reason (possibly even this
> one) it was convenient to run udev early but I kept saying "... but this
> is ODD!"
>
> Didn't end well. Anyway, do you happen to know how findfs is
> implemented? Does it actually read the UUID and label off the
> partitions itself?

The idea seems to be to get away from custom scripts written purely for
initramfs use to try and find the information needed. udev already does
what is needed and is already installed and well tested. It also helps
make udev iniramfses more portable without adding tons of "what if" code.


--
Neil Bothwick

Some people are born mediocre, some people achieve mediocrity, and some
people have mediocrity thrust upon them. - Joseph Heller, "Catch-22"

[marco restelli]

2017-02-15 16:19 GMT+01:00, Rich Freeman <ri...@gentoo.org>:
>>
>>> Anytime you see something like root=UUID=* that is being handled by an
>>> initramfs.
>>
>> I understand that this parameter is passed by the kernel to the init
>> script inside the initramfs which then uses "busybox findfs" to
>> translate the UUID into a device name. Is this correct?
>>
>
> I suppose that is one way it could be done, but of course it could be
> implemented in other ways. As far as I can tell Dracut does not use
> busybox findfs.

OK, yes. Indeed I was looking at the init used by genkernel.

Marco