[gentoo-user] systemd and initramfs

[Helmut Jarausch]

Hi,

what binaries and libraries have to be put into an initramfs for a
system
booting with init=/usr/lib/systemd/systemd ?
(I am building the initramsfs myself)

Thanks for some hints,
Helmut

[Mike Gilbert]

On Mon, Aug 19, 2013 at 5:58 AM, Helmut Jarausch
<jara...@igpm.rwth-aachen.de> wrote:
> Hi,
>
> what binaries and libraries have to be put into an initramfs for a system
> booting with init=/usr/lib/systemd/systemd ?
> (I am building the initramsfs myself)
>

You need to get your root filesystem and /usr mounted. Just keep that
goal in mind and start adding files to support it.

There doesn't need to be anything systemd-specific.

[thegeezer]

my 2c would be to autobuild one using genkernel or dracut and then
dissemble it
this is because I always forget silly things like the special files
dev/tty and sda

[Helmut Jarausch]

I am not sure about "timing".

Initially the kernel has only the "mini" /usr partition contained
within the initramfs.
Then it switches to "real root" and only then it tries to mount the
"real" /usr partition.
Does this all happen before the kernel hands control over to the init
process (i.e. systemd) ?

Many thanks,
Helmut

[Canek Peláez Valdés]

On Tue, Aug 20, 2013 at 1:44 AM, Helmut Jarausch

<jara...@igpm.rwth-aachen.de> wrote:
> On 08/19/2013 04:55:29 PM, Mike Gilbert wrote:
>>
>> On Mon, Aug 19, 2013 at 5:58 AM, Helmut Jarausch
>> <jara...@igpm.rwth-aachen.de> wrote:
>> > Hi,
>> >
>> > what binaries and libraries have to be put into an initramfs for a
>> > system
>> > booting with init=/usr/lib/systemd/systemd ?
>> > (I am building the initramsfs myself)
>> >
>>
>> You need to get your root filesystem and /usr mounted. Just keep that
>> goal in mind and start adding files to support it.
>>
>> There doesn't need to be anything systemd-specific.
>>
>
> I am not sure about "timing".
>
> Initially the kernel has only the "mini" /usr partition contained within the
> initramfs.
> Then it switches to "real root" and only then it tries to mount the "real"
> /usr partition.
> Does this all happen before the kernel hands control over to the init
> process (i.e. systemd) ?

No, the kernel has a mini filesystem (doesn't matter which directory
structure has inside), and it executes the init script (or binary
program) in the root of the initramfs. This init program/script is the
responsible for mounting the real root and other partitions, and
handling control over to systemd (or OpenRC, or whatever).

Dracut is able to create an initramfs (with the systemd Dracut module)
that executes systemd inside the initramfs, which mounts /usr,
switches to the real root, and gives control to the "real" systemd
instance. At shutdown, the reverse happens: the "real" systemd
surrenders control to the initramfs systemd, it umounts everything,
and finish the shutdow process.

If you generate an initramfs with Dracut using the systemd module, the
init inside the initramfs is a link to /usr/lib/systemd/systemd.

Unless you want to learn the ins and outs of using an initramfs (and
having a lot of fun and failed boots in the process), I highly
recommend using Dracut. It does everything for you.

Regards.
--
Canek Peláez Valdés
Posgrado en Ciencia e Ingeniería de la Computación
Universidad Nacional Autónoma de México

[Alan McKinnon]

Yes.

init requires / to be properly mounted.

You have a typo in your description, it should be:

"Then it switches to "real root" and mounts the "real" / partition."

The kernel mounts /, init mounts /usr (read from fstab).
The kernel does not mount /usr (although there can be one in the
initramfs, this is discarded when real root is mounted)


--
--
Alan McKinnon
alan.m...@gmail.com

[Helmut Jarausch]

Yes, I'd like to generate the initramfs myself and let the kernel store
it within
its binary on /boot. This has been working just fine with openrc. Now I
only need
to now the additional requirements when using systemd as init process.

I'll have a look at what Dracut is doing.

Thanks to you and Alan,
Helmut

[Tanstaafl]

On 2013-08-20 2:54 AM, Canek Peláez Valdés <can...@gmail.com> wrote:
> Unless you want to learn the ins and outs of using an initramfs (and
> having a lot of fun and failed boots in the process), I highly
> recommend using Dracut. It does everything for you.

What about a previous posters comment that they don;t update the kernel
as often as userland stuff, and there is userland stuff in the
initramfs, so things can still get out of sync - and, apparently (I'm
inferring from the comments about nightmare scenarios of unbootable
systems because the initramfs got 'out of sync')...

So, how do/can you *guarantee* that nothing ever gets out of sync?

[Neil Bothwick]

On Tue, 20 Aug 2013 06:57:02 -0400, Tanstaafl wrote:

> > Unless you want to learn the ins and outs of using an initramfs (and
> > having a lot of fun and failed boots in the process), I highly
> > recommend using Dracut. It does everything for you.
>
> What about a previous posters comment that they don;t update the kernel
> as often as userland stuff, and there is userland stuff in the
> initramfs, so things can still get out of sync - and, apparently (I'm
> inferring from the comments about nightmare scenarios of unbootable
> systems because the initramfs got 'out of sync')...

That is only likely to happen if the initramfs file does't match the
kernel. An initramfs isn't magic, or even complex, it's just a few static
commands and a script to get the required filesystems mounted before
passing control to init.

> So, how do/can you *guarantee* that nothing ever gets out of sync?

You could add a custom postinst function to /etc/portage that would
check whether any of the files included in your initramfs are newer than
the initramfs/kernel and send an ewarn if so.


--
Neil Bothwick

Hello, this is an extension to the famous signature virus, called spymail.
Could you please copy me into your signature and send back what you were
doing last night between 10pm and 3am?

[Neil Bothwick]

On Tue, 20 Aug 2013 13:10:26 +0100, Neil Bothwick wrote:

> > So, how do/can you *guarantee* that nothing ever gets out of sync?
>
> You could add a custom postinst function to /etc/portage that would
> check whether any of the files included in your initramfs are newer than
> the initramfs/kernel and send an ewarn if so.

Running this once after each emerge world/system would be more efficient
than running it after every package, something like this

#!/bin/sh

KERNEL="/boot/vmlinuz-$(uname -r)"

for FILE in $(awk '/^file/ {print $3}' /usr/src/init.cfg); do
if [[ ${FILE} -nt ${KERNEL} ]]; then
echo "${FILE} is newer than initramfs"
fi
done

Incidentally, testing this showed that my busybox has been updated since
the kernel was built, but I have managed to reboot without the sky
falling in.


--
Neil Bothwick

On the other hand, you have different fingers.

[Walter Dnes]

On Tue, Aug 20, 2013 at 01:54:26AM -0500, Canek Pel�ez Vald�s wrote

>
> No, the kernel has a mini filesystem (doesn't matter which directory
> structure has inside), and it executes the init script (or binary
> program) in the root of the initramfs. This init program/script is the
> responsible for mounting the real root and other partitions, and
> handling control over to systemd (or OpenRC, or whatever).
>
> Dracut is able to create an initramfs (with the systemd Dracut module)
> that executes systemd inside the initramfs, which mounts /usr,
> switches to the real root, and gives control to the "real" systemd
> instance. At shutdown, the reverse happens: the "real" systemd
> surrenders control to the initramfs systemd, it umounts everything,
> and finish the shutdow process.

A possibly stupid question from a non-user of initramfs... why not
simply treat the initramfs as "the real system"? This would avoid the
hand-off to a second fs at start-up, and the reverse process at
shutdown. There would be no need to worry about keeping files synced in
2 different locations, because there would only be one location. If
necessary, one could use UnionFS http://en.wikipedia.org/wiki/Unionfs to
make the hard drive and userspace stuff all look like part of the
initramfs.

--
Walter Dnes <walt...@waltdnes.org>
I don't run "desktop environments"; I run useful applications

[Canek Peláez Valdés]

On Tue, Aug 20, 2013 at 7:32 PM, Walter Dnes <walt...@waltdnes.org> wrote:
> On Tue, Aug 20, 2013 at 01:54:26AM -0500, Canek Peláez Valdés wrote

>>
>> No, the kernel has a mini filesystem (doesn't matter which directory
>> structure has inside), and it executes the init script (or binary
>> program) in the root of the initramfs. This init program/script is the
>> responsible for mounting the real root and other partitions, and
>> handling control over to systemd (or OpenRC, or whatever).
>>
>> Dracut is able to create an initramfs (with the systemd Dracut module)
>> that executes systemd inside the initramfs, which mounts /usr,
>> switches to the real root, and gives control to the "real" systemd
>> instance. At shutdown, the reverse happens: the "real" systemd
>> surrenders control to the initramfs systemd, it umounts everything,
>> and finish the shutdow process.
>
> A possibly stupid question from a non-user of initramfs... why not
> simply treat the initramfs as "the real system"? This would avoid the
> hand-off to a second fs at start-up, and the reverse process at
> shutdown.

If I understand correctly, because it's a ramfs, as its name implies
(it lives on your RAM). You don't want it to hold all your system,
only the necessary parts to mount the filesystems necessary to boot.

> There would be no need to worry about keeping files synced in
> 2 different locations, because there would only be one location.

The worry about falling out of sync, although justified, I think it's
a little overreacted; even for things like LVM2 and NFS, how many
times changes the metadata or format used by different versions?
Normal filesystems present no problems: almost all of them are
future-proof.

And (as Allan suggested) you can (easily and automagically) regenerate
the initramfs everytime sometimes inside it changes.

> If
> necessary, one could use UnionFS http://en.wikipedia.org/wiki/Unionfs to
> make the hard drive and userspace stuff all look like part of the
> initramfs.

No unionfs has been accepted in the mainline kernel, although several
have been proposed. It's a hard problem, and nothing has satisfied the
kernel devs until now.

The initramfs has several advantages: it's supported directly by the
kernel, it works the same from embedded systems to big iron servers,
and it just works.

I just installed two systems (a server and a desktop) to systemd
directly, and Dracut just worked. And I like that now my kernels have
*EVERYTHING* as a module (that can be compiled as module). Even the
filesystems!

[Mike Gilbert]

On Tue, Aug 20, 2013 at 2:44 AM, Helmut Jarausch

<jara...@igpm.rwth-aachen.de> wrote:
> On 08/19/2013 04:55:29 PM, Mike Gilbert wrote:
>>
>> On Mon, Aug 19, 2013 at 5:58 AM, Helmut Jarausch
>> <jara...@igpm.rwth-aachen.de> wrote:
>> > Hi,
>> >
>> > what binaries and libraries have to be put into an initramfs for a
>> > system
>> > booting with init=/usr/lib/systemd/systemd ?
>> > (I am building the initramsfs myself)
>> >
>>
>> You need to get your root filesystem and /usr mounted. Just keep that
>> goal in mind and start adding files to support it.
>>
>> There doesn't need to be anything systemd-specific.
>>
>
> I am not sure about "timing".
>
> Initially the kernel has only the "mini" /usr partition contained within the
> initramfs.
> Then it switches to "real root" and only then it tries to mount the "real"
> /usr partition.
> Does this all happen before the kernel hands control over to the init
> process (i.e. systemd) ?
>

Here's the essence of what you need to accomplish (skip step 2 if you
don't have a separate /usr filesystem):

1. Mount the "real" root filesystem, which we will call /realroot.
2. Mount the /usr filesystem at /realroot/usr.
3. Pivot /realroot onto / and exec /usr/lib/systemd/systemd.

Step 3 is generally accomplished by a line like this:

exec switch_root /realroot /usr/bin/systemd/systemd "$@"

If you want to be able to switch inits at boot time, you will need to
parse the kernel command line from /proc/cmdline.

[thegeezer]

On 08/21/2013 02:13 AM, Canek Peláez Valdés wrote:
> On Tue, Aug 20, 2013 at 7:32 PM, Walter Dnes <walt...@waltdnes.org> wrote:
>> On Tue, Aug 20, 2013 at 01:54:26AM -0500, Canek Peláez Valdés wrote
>>> No, the kernel has a mini filesystem (doesn't matter which directory
>>> structure has inside), and it executes the init script (or binary
>>> program) in the root of the initramfs. This init program/script is the
>>> responsible for mounting the real root and other partitions, and
>>> handling control over to systemd (or OpenRC, or whatever).
>>>
>>> Dracut is able to create an initramfs (with the systemd Dracut module)
>>> that executes systemd inside the initramfs, which mounts /usr,
>>> switches to the real root, and gives control to the "real" systemd
>>> instance. At shutdown, the reverse happens: the "real" systemd
>>> surrenders control to the initramfs systemd, it umounts everything,
>>> and finish the shutdow process.
>> A possibly stupid question from a non-user of initramfs... why not
>> simply treat the initramfs as "the real system"? This would avoid the
>> hand-off to a second fs at start-up, and the reverse process at
>> shutdown.

you need as much memory as your / partition if you want to do this. it
is a good way of making a livecd if you know you are guaranteed the
memory, as once booted it no longer requires the cd (faster reads).
this boot time ramfs is released once unmounted so even if you have a
100MB initrd you get this memory back after boot normally.
not sure how unionFS works these days, previously i was using AUFS2 as
union didn't provide enough good features for me. but the premise is to
mount ramfs as readonly and then mount real_root as writable over the
top. again you stil have memory issues doing this; while memory is
cheap not all boards (intel atom) support huge amounts of memory.

> If I understand correctly, because it's a ramfs, as its name implies
> (it lives on your RAM). You don't want it to hold all your system,
> only the necessary parts to mount the filesystems necessary to boot.
>
>> There would be no need to worry about keeping files synced in
>> 2 different locations, because there would only be one location.
> The worry about falling out of sync, although justified, I think it's
> a little overreacted; even for things like LVM2 and NFS, how many
> times changes the metadata or format used by different versions?
> Normal filesystems present no problems: almost all of them are
> future-proof.

It happened to me just last week with LVM, and it wasn't a metadata
issue, it was a user space program/ service loaded service running issue.
> update LVM2
> kernel remains the same
> reboot
> initramfs finds all PVS and activates VG
> main system init
> /etc/init.d/lvm2 start
> error can't read from USB PVS
> login to system with missing PVS
> /etc/init.d/lvm2 restart
> all PVS listed
> reboot several times to verify it wasn't just a stuck service, exactly
the same
> now ok but restarting a boot service manually required (!)
I updated the initramfs and rebooted and all problems went away

[Tanstaafl]

On 2013-08-21 3:06 AM, thegeezer <theg...@thegeezer.net> wrote:
> On 08/21/2013 02:13 AM, Canek Peláez Valdés wrote:
>> The worry about falling out of sync, although justified, I think it's
>> a little overreacted; even for things like LVM2 and NFS, how many
>> times changes the metadata or format used by different versions?
>> Normal filesystems present no problems: almost all of them are
>> future-proof.

> It happened to me just last week with LVM, and it wasn't a metadata
> issue, it was a user space program/ service loaded service running issue.

>> update LVM2
>> kernel remains the same
>> reboot
>> initramfs finds all PVS and activates VG
>> main system init
>> /etc/init.d/lvm2 start
>> error can't read from USB PVS
>> login to system with missing PVS
>> /etc/init.d/lvm2 restart
>> all PVS listed
>> reboot several times to verify it wasn't just a stuck service, exactly
>> the same
>> now ok but restarting a boot service manually required (!)

> I updated the initramfs and rebooted and all problems went away

And this is *precisely* what scares me about this.

This simply should not be, period. Support for separate /usr without
initramfs simply SHOULD NOT be dropped unless/until things like this
(updating lvm) can *never* cause a system to fail to boot like this.

[Neil Bothwick]

On Wed, 21 Aug 2013 10:40:32 -0400, Tanstaafl wrote:

> >> update LVM2
> >> kernel remains the same
> >> reboot
> >> initramfs finds all PVS and activates VG
> >> main system init
> >> /etc/init.d/lvm2 start
> >> error can't read from USB PVS
> >> login to system with missing PVS
> >> /etc/init.d/lvm2 restart
> >> all PVS listed
> >> reboot several times to verify it wasn't just a stuck service,
> >> exactly the same
> >> now ok but restarting a boot service manually required (!)
>
> > I updated the initramfs and rebooted and all problems went away

This sounds like a bug in LVM. If it was down to a version clash, why did
a restart find the PVs?

> And this is *precisely* what scares me about this.
>
> This simply should not be, period. Support for separate /usr without
> initramfs simply SHOULD NOT be dropped unless/until things like this
> (updating lvm) can *never* cause a system to fail to boot like this.

This is irrelevant to separate /usr. an initramfs is required if / is on
a VM, whether or not /usr is on the same LV.


--
Neil Bothwick

Top Oxymorons Number 23: Sweet sorrow

[thegeezer]

On 08/21/2013 04:10 PM, Neil Bothwick wrote:
> On Wed, 21 Aug 2013 10:40:32 -0400, Tanstaafl wrote:
>
>>>> update LVM2
>>>> kernel remains the same
>>>> reboot
>>>> initramfs finds all PVS and activates VG
>>>> main system init
>>>> /etc/init.d/lvm2 start
>>>> error can't read from USB PVS
>>>> login to system with missing PVS
>>>> /etc/init.d/lvm2 restart
>>>> all PVS listed
>>>> reboot several times to verify it wasn't just a stuck service,
>>>> exactly the same
>>>> now ok but restarting a boot service manually required (!)
>>> I updated the initramfs and rebooted and all problems went away
> This sounds like a bug in LVM. If it was down to a version clash, why did
> a restart find the PVs?

well you can probably understand my confusion replugging usb devices and
trying to pvscan // vgchange -ay --partial etc
the errors i was getting are below. i genuinely thought with the
missing device it was a hardware failure of the usb disk, and a restart
of lvm was a last gasp chance; when it worked i realised the initram
might need updating.

Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
5242864
- Last output repeated twice -
Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
5242878
- Last output repeated twice -
Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block 0
- Last output repeated twice -
Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block 1
Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
5242879
- Last output repeated 2 times -

>
>> And this is *precisely* what scares me about this.
>>
>> This simply should not be, period. Support for separate /usr without
>> initramfs simply SHOULD NOT be dropped unless/until things like this
>> (updating lvm) can *never* cause a system to fail to boot like this.
> This is irrelevant to separate /usr. an initramfs is required if / is on
> a VM, whether or not /usr is on the same LV.
>
>

Perhaps though it highlights a need for a utility to identify if items
on an initramfs have been updated ?
in this case the problem was definitely between the keyboard and the
chair, but it is easily overlooked (yeah just trying to make myself feel
better)


either way i'm already using initramfs anyway -- i pesonally roll out
lvm on root on everything i can because of it's flexibility: so the
whole argument of whether or not split /usr is not my argument. i'm
just bringing things to light to make the overall process easier for
everyone by highlighting potential issues folks may have.

[Tanstaafl]

On 2013-08-21 11:10 AM, Neil Bothwick <ne...@digimed.co.uk> wrote:
> On Wed, 21 Aug 2013 10:40:32 -0400, Tanstaafl wrote:
>
>>>> update LVM2
>>>> kernel remains the same
>>>> reboot
>>>> initramfs finds all PVS and activates VG
>>>> main system init
>>>> /etc/init.d/lvm2 start
>>>> error can't read from USB PVS
>>>> login to system with missing PVS
>>>> /etc/init.d/lvm2 restart
>>>> all PVS listed
>>>> reboot several times to verify it wasn't just a stuck service,
>>>> exactly the same
>>>> now ok but restarting a boot service manually required (!)
>>
>> I updated the initramfs and rebooted and all problems went away

> This sounds like a bug in LVM. If it was down to a version clash, why did
> a restart find the PVs?

Sorry, ianap, but I do know that this kind of thing has never happened
to me in my 8+ years of running this old system with a separate /usr
*without* an initramfs...

So, the bottom line is, obviously (to me at least), there are a lot more
things that can go wrong when an initramfs is involved, that simply
don't or can't happen otherwise.

>> And this is *precisely* what scares me about this.
>>
>> This simply should not be, period. Support for separate /usr without
>> initramfs simply SHOULD NOT be dropped unless/until things like this
>> (updating lvm) can *never* cause a system to fail to boot like this.

> This is irrelevant to separate /usr. an initramfs is required if / is on
> a VM, whether or not /usr is on the same LV.

Sorry, I don't see where he said that this system was running on a VM...
or did you mean where he had / on an *LVM* partition - which, again, he
did not say he had.

[Canek Peláez Valdés]

As Neil said, this sounds like a bug in LVM and/or the initramfs you
were using (did you use genkernel, dracut, you roll your own?) If a
restart worked, the initramfs could do that, right?

So, it was not a problem with the initramfs, per se.

> Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
> 5242864
> - Last output repeated twice -
> Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
> 5242878
> - Last output repeated twice -
> Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
0
> - Last output repeated twice -
> Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block 1
> Aug 17 17:56:30 [kernel] Buffer I/O error on device dm-26, logical block
> 5242879
> - Last output repeated 2 times -
>
>
>>
>>> And this is *precisely* what scares me about this.
>>>
>>> This simply should not be, period. Support for separate /usr without
>>> initramfs simply SHOULD NOT be dropped unless/until things like this
>>> (updating lvm) can *never* cause a system to fail to boot like this.
>> This is irrelevant to separate /usr. an initramfs is required if / is on
>> a VM, whether or not /usr is on the same LV.
>>
>>
>
>
> Perhaps though it highlights a need for a utility to identify if items
> on an initramfs have been updated ?
> in this case the problem was definitely between the keyboard and the
> chair, but it is easily overlooked (yeah just trying to make myself feel
> better)

I'm under the impression that, with a properly generated initramfs,
this kind of things would be avoided when changing minor versions of
stuff like LVM or NFS. For major versions, a message from the ebuild
shoul be enough, and I'm not even sure about that.

We should also expect some responsibility from the user towards her
system; if she updates anything related to mounting /usr (and perhaps
/), then she should use her common sense and rebuild her initramfs.

> either way i'm already using initramfs anyway -- i pesonally roll out
> lvm on root on everything i can because of it's flexibility: so the
> whole argument of whether or not split /usr is not my argument. i'm
> just bringing things to light to make the overall process easier for
> everyone by highlighting potential issues folks may have.

If we had just one initramfs generator, this could be easily
automatized. The problem is that we have genkernel, dracut, busybox
with the sep-usr USE flag, and roll-your-own-initramfs (and whichever
one I don't know about). The cost of all this choice is that is
responsibility of the user to take care of her system.

I highly recommend to use dracut; it just works, it can be called at
any time, and it takes between 10 to 30 seconds (depends on the speed
of the machine) to build an initramfs. If you are really, *really*
paranoid, you can make an script for your "emerge -whatever world"
command, and add "dracut -f -H" afterwards, and then only upgrade your
world with that script.

That way, *every* time you upgrade your world, you update your
initramfs. It adds 10-30 seconds to your upgrade time, but the
initramfs is always in sync.

I only update my initramfs after a kernel update, but I follow
vanilla-sources, so that is every other week or something like that. I
have an script that compiles the new kernel, installs it, generates
the initramfs, and updates the GRUB (or GRUB2) config file, so
everything is automatized.

[thegeezer]

either way i'm not too fussed to find out - i found a temp solution
(restart lvm) and a complete resolution (update initramfs).
it is odd though huh, and is an important headsup to me (and maybe
others) to make sure to update the initramfs in future when i notice lvm
updates.
as it only happened last week it was i thought pertinent to the
discussion, thanks for the thoughts on finding a resolution but it is
resolved for me now.
i only post it as a potential warning to others.

> were using (did you use genkernel, dracut, you roll your own?) If a
> restart worked, the initramfs could do that, right?

genkernel initramfs --lvm

+1 for the user responsibility, as i said definitely pebkac.
however, a friendly notice wouldn't go amiss.
as i mentioned in a previous post where that notice comes from is to be
debated as i do not want an ubuntu style everything happens for you
automagically because then when you try to do anything different it all
falls down.
ya, i want full control and none of the responsibility for when it goes
wrong ;)

>
>> either way i'm already using initramfs anyway -- i pesonally roll out
>> lvm on root on everything i can because of it's flexibility: so the
>> whole argument of whether or not split /usr is not my argument. i'm
>> just bringing things to light to make the overall process easier for
>> everyone by highlighting potential issues folks may have.
> If we had just one initramfs generator, this could be easily
> automatized. The problem is that we have genkernel, dracut, busybox
> with the sep-usr USE flag, and roll-your-own-initramfs (and whichever
> one I don't know about). The cost of all this choice is that is
> responsibility of the user to take care of her system.
>
> I highly recommend to use dracut; it just works, it can be called at
> any time, and it takes between 10 to 30 seconds (depends on the speed
> of the machine) to build an initramfs. If you are really, *really*
> paranoid, you can make an script for your "emerge -whatever world"
> command, and add "dracut -f -H" afterwards, and then only upgrade your
> world with that script.
>
> That way, *every* time you upgrade your world, you update your
> initramfs. It adds 10-30 seconds to your upgrade time, but the
> initramfs is always in sync.

i'll have to try dracut, genkernel is considerably longer as it seems to
want to recompile everything each time i.e. busybox

>
> I only update my initramfs after a kernel update, but I follow
> vanilla-sources, so that is every other week or something like that. I
> have an script that compiles the new kernel, installs it, generates
> the initramfs, and updates the GRUB (or GRUB2) config file, so
> everything is automatized.
>
> Regards.

anyone have any good pointers to an initramfs interrogator, maybe that
takes as argument kernel command line ?

[Mike Gilbert]

On Wed, Aug 21, 2013 at 12:56 PM, thegeezer <theg...@thegeezer.net> wrote:
> anyone have any good pointers to an initramfs interrogator, maybe that
> takes as argument kernel command line ?

If you are looking to test an initramfs, qemu is handy.

qemu-kvm -kernel /boot/vmlinuz -initrd /boot/initramfs.img -append
"kernel command line"

[Neil Bothwick]

On Wed, 21 Aug 2013 17:56:36 +0100, thegeezer wrote:

> anyone have any good pointers to an initramfs interrogator, maybe that
> takes as argument kernel command line ?

I posted a script fragment that compares the contents of the kernel's
initramfs config file with the live filesystem. Dracut users could use
the same method by parsing the output from lsinitrd.


--
Neil Bothwick

For every action, there is an equal and opposite malfunction.

[Tanstaafl]

On 2013-08-21 2:52 PM, Neil Bothwick <ne...@digimed.co.uk> wrote:
> On Wed, 21 Aug 2013 17:56:36 +0100, thegeezer wrote:
>
>> anyone have any good pointers to an initramfs interrogator, maybe that
>> takes as argument kernel command line ?
>
> I posted a script fragment that compares the contents of the kernel's
> initramfs config file with the live filesystem. Dracut users could use
> the same method by parsing the output from lsinitrd.

My question is, why can't portage do this automatically as part of the
update process for these packages?

Again: support for separate /usr should absolutely NOT be dropped
unless/until this process can be handled properly by portage itself.

At an absolute bare minimum, the emerge output should advise the user
if/when they need to update their initramfs, but ideally, it should be
done automatically - automation is something that computers are good at.

[Mike Gilbert]

It really doesn't matter if your initramfs is out-of-date, so long as
it is able to mount your filesystems and pass control along to your
real system.

Personally, I only update my initramfs when I update my kernel; it
works out fine.

[Neil Bothwick]

On Wed, 21 Aug 2013 11:50:57 -0400, Tanstaafl wrote:

> > This sounds like a bug in LVM. If it was down to a version clash, why
> > did a restart find the PVs?
>
> Sorry, ianap, but I do know that this kind of thing has never happened
> to me in my 8+ years of running this old system with a separate /usr
> *without* an initramfs...

Which proves absolutely nothing. For all we know you don't use LVM either.

> So, the bottom line is, obviously (to me at least), there are a lot
> more things that can go wrong when an initramfs is involved, that
> simply don't or can't happen otherwise.

I'd take issue with "a lot" but there are things that can go wrong with
an initramfs (but this wasn't one of them, it was PEBKAC) just as there
are things that can go wrong when you use a separate /usr without an
initramfs.

> >> And this is *precisely* what scares me about this.
> >>
> >> This simply should not be, period. Support for separate /usr without
> >> initramfs simply SHOULD NOT be dropped unless/until things like this
> >> (updating lvm) can *never* cause a system to fail to boot like
> >> this.

No one has demonstrated that it can. An initramfs isn't magic, it
caries out a couple of trivial tasks before switching to the real root
partition.

Yes, an initramfs adds an extra step to the boot process, but so does
having a separate /usr in the first place. I think that if you took the
time to understand what an initramfs is and does instead of making an
emotional reaction to it, you would see that this is no big deal.

> > This is irrelevant to separate /usr. an initramfs is required if / is
> > on a VM, whether or not /usr is on the same LV.
>
> Sorry, I don't see where he said that this system was running on a
> VM... or did you mean where he had / on an *LVM* partition - which,
> again, he did not say he had.

Sorry, I meant LV.


--
Neil Bothwick

667 - The FAX number of the beast

[thegeezer]

On 08/21/2013 07:52 PM, Neil Bothwick wrote:
> On Wed, 21 Aug 2013 17:56:36 +0100, thegeezer wrote:
>
>> anyone have any good pointers to an initramfs interrogator, maybe that
>> takes as argument kernel command line ?
> I posted a script fragment that compares the contents of the kernel's
> initramfs config file with the live filesystem. Dracut users could use
> the same method by parsing the output from lsinitrd.
>
>

hiya,
the script you posted referenced /usr/src/init.cfg -- not sure if this
is a dracut thing but i don't have that.
i was actually thinking something like the following (warning needs work)

#!/bin/bash
# get contents of lsinitrd | awk for month,day,time, /filename | list
the awked filename
lsinitrd /boot/boot/initramfs-genkernel-x86_64-`uname -r` | awk '{ print
$6" "$7" "$8" /"$9; };' | xargs ls -alhd | awk '{ print $6" "$7" "$8"
"$9; };' > existingfiles.tmp
# get contents of lsinitrd
lsinitrd /boot/boot/initramfs-genkernel-x86_64-`uname -r` | awk '{ print
$6" "$7" "$8" /"$9; };' > initrdfiles.tmp
# do a diff to see whats newer in initrd
diff initrdfiles.tmp existingfiles.tmp | grep '<'

improvements gratefully received
:)

[Neil Bothwick]

On Wed, 21 Aug 2013 21:09:41 +0100, thegeezer wrote:

> > I posted a script fragment that compares the contents of the kernel's
> > initramfs config file with the live filesystem. Dracut users could use
> > the same method by parsing the output from lsinitrd.

> the script you posted referenced /usr/src/init.cfg -- not sure if this
> is a dracut thing but i don't have that.

It's the kernel's initramfs config file, or at least the location I use
(the location is set in the kernel config)

> i was actually thinking something like the following (warning needs
> work)
>
> #!/bin/bash
> # get contents of lsinitrd | awk for month,day,time, /filename | list
> the awked filename

There's no need to get the timestamps from the initramfs, just the list
of files. Then you compare the timestamp of the ones in your root
filesystem with the timestamp of the file containing the initramfs. it
saves parsing and comparing all those dates, just use the shell's -nt
test. The only difference between using it with Dracut or the kernel's
initramfs is how you generate the file list.


--
Neil Bothwick

Press every key to continue.

[Mark David Dumlao]

I'm confused here. initramfs, is, for all intents and purposes, an isolated
filesystem. It shouldn't be calling stuff in your real root except to
mount the real
root. Heck it should be able to mount pivot root on filesystems that
have absolutely
nothing to do with its construction, as for example, LTSP does.

So the only "out of sync" scenario that should matter is with the
kernel or kernel
modules. Even if it were out of sync with your current toolset it
should still be able
to perform the pivot. Shouldn't any "userland stuff" that breaks initramfs BE in
initramfs?
--
This email is: [ ] actionable [ ] fyi [x] social
Response needed: [ ] yes [ ] up to you [x] no
Time-sensitive: [ ] immediate [ ] soon [x] none

[J. Roeleveld]

> On Tue, Aug 20, 2013 at 6:57 PM, Tanstaafl <tans...@libertytrek.org>
wrote:
>> On 2013-08-20 2:54 AM, Canek Peláez Valdés <can...@gmail.com> wrote:
>>>
>>> Unless you want to learn the ins and outs of using an initramfs (and
>>> having a lot of fun and failed boots in the process), I highly
>>> recommend using Dracut. It does everything for you.
>>
>>
>> What about a previous posters comment that they don;t update the kernel as
>> often as userland stuff, and there is userland stuff in the initramfs, so
>> things can still get out of sync - and, apparently (I'm inferring from the
>> comments about nightmare scenarios of unbootable systems because the
>> initramfs got 'out of sync')...
>>
>> So, how do/can you *guarantee* that nothing ever gets out of sync?
>>
>
> I'm confused here. initramfs, is, for all intents and purposes, an
> isolated filesystem. It shouldn't be calling stuff in your real root
> except to mount the real
> root. Heck it should be able to mount pivot root on filesystems that
> have absolutely
> nothing to do with its construction, as for example, LTSP does.

Correct, and here lies the cause for the "out of sync" scenario.

> So the only "out of sync" scenario that should matter is with the
> kernel or kernel modules. Even if it were out of sync with your
> current toolset it should still be able
> to perform the pivot. Shouldn't any "userland stuff" that
> breaks initramfs BE in initramfs?

Incorrect, there are userland tools, like LVM and MDADM (layout 1.2 does
NOT support auto-assembly by kernel), that are needed to access of the
filesystems.

It is possible that an older version of one of these tools, after an
update, can no longer access the disks succesfully. When portage updates
this package, the initramfs is not automatically updated with the new
version.

I agree that it doesn't happen often. But on this list there has already
been a report of a recent occurence where LVM was updated, but the
initramfs was not, where the boot-sequence ended up being broken.

--
Joost

[J. Roeleveld]

On Wed, August 21, 2013 22:02, Neil Bothwick wrote:
> On Wed, 21 Aug 2013 11:50:57 -0400, Tanstaafl wrote:
>
>> > This sounds like a bug in LVM. If it was down to a version clash, why
>> > did a restart find the PVs?
>>
>> Sorry, ianap, but I do know that this kind of thing has never happened
>> to me in my 8+ years of running this old system with a separate /usr
>> *without* an initramfs...
>
> Which proves absolutely nothing. For all we know you don't use LVM either.

True, except that I have been using LVM for that long with a seperate LVM.
My partition scheme used to look like:
1) /boot (100M)
2) / (500M)
3) swap (whatever seemed logical)
4) LVM (rest)

And then the larger parts in seperate LVs. This always worked for me and I
never had any issues with any programs running on my machines.
As it looks like we are being forced to use an initramfs if /usr is
seperate, I decided to use the following partitioning:
1) /boot (300M)
2) LVM (rest)

And simply put everything into LVM.

>> So, the bottom line is, obviously (to me at least), there are a lot
>> more things that can go wrong when an initramfs is involved, that
>> simply don't or can't happen otherwise.
>
> I'd take issue with "a lot" but there are things that can go wrong with
> an initramfs (but this wasn't one of them, it was PEBKAC) just as there
> are things that can go wrong when you use a separate /usr without an
> initramfs.

I agree with the PEBKAC, but a simple method to identify when the
initramfs is out-of-sync with userland tools would help. Preferably
something integrated into portage that puts out a warning when a package
that has parts in the initramfs is updated mentions that the initramfs is
out-of-sync.

>> >> And this is *precisely* what scares me about this.
>> >>
>> >> This simply should not be, period. Support for separate /usr without
>> >> initramfs simply SHOULD NOT be dropped unless/until things like this
>> >> (updating lvm) can *never* cause a system to fail to boot like
>> >> this.
>
> No one has demonstrated that it can. An initramfs isn't magic, it
> caries out a couple of trivial tasks before switching to the real root
> partition.

The issue mentioned was an example. It was also:
1) The only one I can remember from the last 4 or 5 years
2) Easily avoided with a "rebuild initramfs" notice during upgrade

> Yes, an initramfs adds an extra step to the boot process, but so does
> having a separate /usr in the first place. I think that if you took the
> time to understand what an initramfs is and does instead of making an
> emotional reaction to it, you would see that this is no big deal.

I think part of the "problem" with it is that the documentation about it
isn't clear. There are tools (genkernel / dracut /..? ) that can automate
the generation of it. But it isn't clear what exactly it is doing.
If there would be a clear guide on how to do it manually, or a tool that
would assist in building the file(s) needed to have it build into the
kernel, then it might be more acceptable to some.

I currently use genkernel, simply because it works-for-me and I haven't
had the time to investigate how to get my setup supported with an
in-kernel version.

>> > This is irrelevant to separate /usr. an initramfs is required if / is
>> > on a VM, whether or not /usr is on the same LV.
>>
>> Sorry, I don't see where he said that this system was running on a
>> VM... or did you mean where he had / on an *LVM* partition - which,
>> again, he did not say he had.
>
> Sorry, I meant LV.

The person with the issue did not mention having / on LVM.

I also never had any issues with /usr on LVM while / was not.

--
Joost

[Alan McKinnon]

On 22/08/2013 08:20, J. Roeleveld wrote:
> On Wed, August 21, 2013 22:02, Neil Bothwick wrote:
>> On Wed, 21 Aug 2013 11:50:57 -0400, Tanstaafl wrote:
>>

[snip]

>> No one has demonstrated that it can. An initramfs isn't magic, it
>> caries out a couple of trivial tasks before switching to the real root
>> partition.
>
> The issue mentioned was an example. It was also:
> 1) The only one I can remember from the last 4 or 5 years
> 2) Easily avoided with a "rebuild initramfs" notice during upgrade

Take out that word "Easily", it doesn't belong there.

What is the trigger condition that causes the message to be emitted?

Will it be in each ebuild whose files might end up in an intramfs?
Expect much bitching and refusing from the devs who will have to
maintain that, so not gonna fly.

Will it be at the tail end of all emerge actions? Ain't gonna fly either
as that is completely not a portage function. The PM installs and
updates packages, it does not check what you then do with the results.
And portage doesn't really know if you a) have an initramfs, b) where it
is, c) what you want in it (as opposed to what you have in it)

Will it be a reminder after installing kernel sources? Changed sources
are not the cause of the isolated problems being mentioned here. Changed
userland is.

Face it. If you want to use an initramf, IT IS THE USERS JOB to keep it
working as he wants it to work. It's a somewhat similar problem to out
of tree modules and keeping them installed and in sync - portage makes
zero effort to help with that one too.--
--
Alan McKinnon
alan.m...@gmail.com

[J. Roeleveld]

True, but with the scripts that are floating in this thread, a usable
solution can be build.
Only need to finalize that into a generic option and document it in the
wiki. Preferably as a "post-emerge" command.

--
Joost

[Neil Bothwick]

On Thu, 22 Aug 2013 10:23:57 +0200, J. Roeleveld wrote:

> True, but with the scripts that are floating in this thread, a usable
> solution can be build.

It took me a few seconds to do that, so why hasn't it been done before?
It's obviously not due to the work involved, so it must be that no one
involved believes there is a need for it, an belief I share.

> Only need to finalize that into a generic option and document it in the
> wiki. Preferably as a "post-emerge" command.

No thanks you. As Alan said, portage doesn't have a mechanism to run a
command after a batch emerge, it would have to be done after every single
package. So the next time a bunch of Perl packages and their virtuals are
updated, look forward to it being run fifty times.


--
Neil Bothwick

When there's a will, I want to be in it.

[Neil Bothwick]

On Thu, 22 Aug 2013 07:26:38 +0200, J. Roeleveld wrote:

> > So the only "out of sync" scenario that should matter is with the
> > kernel or kernel modules. Even if it were out of sync with your
> > current toolset it should still be able
> > to perform the pivot. Shouldn't any "userland stuff" that
> > breaks initramfs BE in initramfs?
>
> Incorrect, there are userland tools, like LVM and MDADM (layout 1.2 does
> NOT support auto-assembly by kernel), that are needed to access of the
> filesystems.
>
> It is possible that an older version of one of these tools, after an
> update, can no longer access the disks succesfully. When portage updates
> this package, the initramfs is not automatically updated with the new
> version.

The userland tools either work or they don't. Nothing stops working as
soon as an update becomes available. The versions of the tools in the
root filesystem don't matter at this point, because the root filesystem
is not mounted. The initramfs is a standalone environment that doesn't
stop working just because someone changed a file on an unmounted
filesystem.


--
Neil Bothwick

No, you *can't* call 999 now. I'm downloading my mail.

[Neil Bothwick]

On Thu, 22 Aug 2013 08:20:23 +0200, J. Roeleveld wrote:

> No one has demonstrated that it can. An initramfs isn't magic, it
> > caries out a couple of trivial tasks before switching to the real root
> > partition.
>
> The issue mentioned was an example. It was also:
> 1) The only one I can remember from the last 4 or 5 years
> 2) Easily avoided with a "rebuild initramfs" notice during upgrade

3) spurious as the poster then realised that this was a PEBKAC problem.
So in 5 years you have seen one problem blamed on the initramfs, and all
but one of those reported problems were actually down the the initramfs.

> I think part of the "problem" with it is that the documentation about it
> isn't clear.

No argument there.

> There are tools (genkernel / dracut /..? ) that can
> automate the generation of it. But it isn't clear what exactly it is
> doing. If there would be a clear guide on how to do it manually, or a
> tool that would assist in building the file(s) needed to have it build
> into the kernel, then it might be more acceptable to some.

There are two. A rather terse one in the kernel documentation, I posted
the location earlier in the thread, and a page on the Wiki that describes
the process in more detail, including an example init script. I've just
looked for it and it has expanded since I last need to look at it

http://wiki.gentoo.org/wiki/Early_Userspace_Mounting

Or if you look at the official Gentoo documentation it links to the
various resources.

http://www.gentoo.org/doc/en/initramfs-guide.xml


--
Neil Bothwick

Top Oxymorons Number 31: Small crowd

[Mark David Dumlao]

On Aug 22, 2013 1:28 PM, "J. Roeleveld" <jo...@antarean.org> wrote:
> Correct, and here lies the cause for the "out of sync" scenario.
>
> > So the only "out of sync" scenario that should matter is with the
> > kernel or kernel modules. Even if it were out of sync with your
> > current toolset it should still be able
> > to perform the pivot. Shouldn't any "userland stuff" that
> > breaks initramfs BE in initramfs?
>
> Incorrect, there are userland tools, like LVM and MDADM (layout 1.2 does
> NOT support auto-assembly by kernel), that are needed to access of the
> filesystems.
>
> It is possible that an older version of one of these tools, after an
> update, can no longer access the disks succesfully. When portage updates
> this package, the initramfs is not automatically updated with the new
> version.

Ok. I don't use raid / lvm on my desktop so I missed the obvious case of a user tool that needs to be in initramfs.

But it makes sense. Any tool that affects filesystem mounting at the very least of /usr, even if its cifs or nfs or whatev, should be included in initramfs. This is gentoo, not ubuntu. Blind updates are known to be irresponsible behavior. Is this a big deal?

[Alan McKinnon]

On 22/08/2013 13:47, Mark David Dumlao wrote:
>
> On Aug 22, 2013 1:28 PM, "J. Roeleveld" <jo...@antarean.org

No it's not a big deal.

If you omit support in the kernel for your mb chipset, the machine does
not boot.
If you forget to recompile nvidia drivers when updating the kernel, the
screen does not work.
If you delete the modules for ext* from /lib/modules, the kernel cannot
read the file system.

All these things need to be fixed manually, need to be remembered
manually, are not detectable in any sane fashion and Gentoo users are
expected to know what to do and how to fix it. All this talk of user
tools in initramfs that might not work one time in three years is
exactly the same IMNSHO.

The conventional notification method is via an elog message from ebuilds
that the maintainer knows will change metadata formats



--
Alan McKinnon
alan.m...@gmail.com

[J. Roeleveld]

On Thu, August 22, 2013 11:17, Neil Bothwick wrote:
> On Thu, 22 Aug 2013 08:20:23 +0200, J. Roeleveld wrote:
>
>> No one has demonstrated that it can. An initramfs isn't magic, it
>> > caries out a couple of trivial tasks before switching to the real root
>> > partition.
>>
>> The issue mentioned was an example. It was also:
>> 1) The only one I can remember from the last 4 or 5 years
>> 2) Easily avoided with a "rebuild initramfs" notice during upgrade
>
> 3) spurious as the poster then realised that this was a PEBKAC problem.
> So in 5 years you have seen one problem blamed on the initramfs, and all
> but one of those reported problems were actually down the the initramfs.

Also the reason why I don't see a problem with the current methods for
initramfs usage.

>> I think part of the "problem" with it is that the documentation about it
>> isn't clear.
>
> No argument there.
>
>> There are tools (genkernel / dracut /..? ) that can
>> automate the generation of it. But it isn't clear what exactly it is
>> doing. If there would be a clear guide on how to do it manually, or a
>> tool that would assist in building the file(s) needed to have it build
>> into the kernel, then it might be more acceptable to some.
>
> There are two. A rather terse one in the kernel documentation, I posted
> the location earlier in the thread, and a page on the Wiki that describes
> the process in more detail, including an example init script. I've just
> looked for it and it has expanded since I last need to look at it
>
> http://wiki.gentoo.org/wiki/Early_Userspace_Mounting
>
> Or if you look at the official Gentoo documentation it links to the
> various resources.
>
> http://www.gentoo.org/doc/en/initramfs-guide.xml

Last time I looked, the information was too minimal to create the config
needed for my setup. Genkernel was the simpler method.
The current version seems more complete. I will give it another go when I
find a spare moment.

--
Joost

[J. Roeleveld]

I would not want it run fifty times. I would prefer if there was a way to
have a script run automagically when emerge finished with a batch-update.

That script is a usefull addition to the set of commands I use when
updating the server. It would help me to avoid rebuilding the initramfs
each time I do an update.

--
Joost

[Neil Bothwick]

On Thu, 22 Aug 2013 14:55:58 +0200, J. Roeleveld wrote:

> > No thanks you. As Alan said, portage doesn't have a mechanism to run a
> > command after a batch emerge, it would have to be done after every
> > single package. So the next time a bunch of Perl packages and their
> > virtuals are updated, look forward to it being run fifty times.
>
> I would not want it run fifty times. I would prefer if there was a way
> to have a script run automagically when emerge finished with a
> batch-update.

There maybe times that would be useful, but it is not possible with
portage. The only way I can think of is a wrapper script that you call
instead of emerge.


--
Neil Bothwick

Power outage at a department store yesterday, Twenty people were
trapped on the escalators.

[Stefan G. Weichinger]

Am 20.08.2013 08:54, schrieb Canek Peláez Valdés:

> Unless you want to learn the ins and outs of using an initramfs (and
> having a lot of fun and failed boots in the process), I highly
> recommend using Dracut. It does everything for you.

I'd dig a short and working howto "systemd and dracut".

My approach in the last weeks/months was a mixed one (which is nearly
always bad):

I ran genkernel to compile kernel and modules ... additionally generated
initramfs with a quick-and-dirty-script hacked by myself (it uses
dracut) ... and then executed grub2-mkconfig to update my grub2-config.

This works so far ... until yesterday when my thinkpad didn't boot
correctly anymore with the gentoo-sources-based kernel/initramfs 3.10.9
(interesting: my desktop worked out fine with the same sources ... the
config might be a bit different).

I found that the genkernel-based grub2-entry (with the initramfs
generated via genkernel) didn't create /dev/shm anymore .. leading to
various things failing.

Another grub2-entry pointing to the dracut-based initramfs works fine. Cool.

So I have to get rid of genkernel, right? At least for now.

Additionally I would really like to understand how to influence the
default entry for grub2 ... letting grub2-mkconfig detect the available
options is one thing ... but do I really have to count down the
available kernels and edit the number?

I would prefer to be able to explicitly select my default kernel by
editing some file and for example choose "3.10.9" somewhere ...

Stefan

[Neil Bothwick]

On Thu, 22 Aug 2013 23:08:43 +0200, Stefan G. Weichinger wrote:

> Additionally I would really like to understand how to influence the
> default entry for grub2 ... letting grub2-mkconfig detect the available
> options is one thing ... but do I really have to count down the
> available kernels and edit the number?
>
> I would prefer to be able to explicitly select my default kernel by
> editing some file and for example choose "3.10.9" somewhere ...

/etc/grub.d/40_custom

Add you entries there, and change the number in the filename to have them
appear before the autogenerated entries.


--
Neil Bothwick

Modulation in all things.

[Stefan G. Weichinger]

Am 23.08.2013 00:44, schrieb Neil Bothwick:

> /etc/grub.d/40_custom
>
> Add you entries there, and change the number in the filename to
> have them appear before the autogenerated entries.

Thanks for the pointer. Gotta play with that.