Unstable naming of dev/mmcblk0

[Alan Martinovic]

Hi,
I've noticed that mender decides which partition is being booted
by passing a kernel paramter root=/dev/mmcblk0p2 to the kernel.

So the parameter is hard codded into and the booting is decided
by the partition index (mmcblk0p1 or mmcblk0p2).


What I've been witnessing recently after needing to update the
kernel on my device is that sometimes the naming of mmcblk0 changes.
It becomes mmcblk1 or mmcblk2.
When this happens the system stops since it can't mount the rootfs.

In an attempt to see why has this started to happen, I've discovered
that it's normal kernel behavior not to provide stable indexes for device
files and that it randomly depends on the probing order.

Here is a link
https://lkml.org/lkml/2016/4/29/817


Has anyone experienced issues like these?
How do you ensure that the naming won't change at random
when updating the kernel in the field?

Be Well,
Alan

[Harry ten Berge]

Have you tried using UUID's? 

Like in:

/dev/disk/by-uuid/

Regards Harry

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.
To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

[Alan Martinovic]

Is this something addressed in newer mender versions?

I'm currently running 1.3 and the uboot scripts seem to depend on boot partitions 

being indexed 1 and 2.

<partial output of env print in uboot>

    mender_boot_part=2

    mender_dtb_name=sun8i-h3-senic-hub.dtb

    mender_kernel_name=zImage

    mender_setup=setenv mender_kernel_root /dev/mmcblk0p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk0p2; else setenv mender_boot_part_name /dev/mmcblk0p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs

Besides that, I need a solution so that a single built image would work for multiple 

devices and as far as I understood UUID would be generated differently for every 

device and mapped to a dev/mmcblk*.

Say that I build the image so that uboot passes root=/dev/disk/by-uuid/91c2b390-272f-4160-a9eb-e97fa4f484cf

How do I scale that? 

To be honest I'm not even sure why that would work for a single device.

Am I missing something in your suggestion?

Be Well,

Alan

To unsubscribe from this group and stop receiving emails from it, send an email to mender+unsubscribe@lists.mender.io.

To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

--

You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+unsubscribe@lists.mender.io.

[Harry ten Berge]

Yeah. I didn't thought of scalability and the fact that those devices are hardcoded.

With respect to booting from u-boot I can imagine that you do something with a variable.

So first decide what's the right device (mmcblk0, mmcblk1 or mmcblk2), place that in a var, and use that var in the u-boot script.

But that doesn't solve the issue that the mender client is streaming directly to the partition.

On the other hand: you could change Mender's configuration file before you start the Mender client....

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.

To post to this group, send email to men...@lists.mender.io.
Visit this group at https://groups.google.com/a/lists.mender.io/group/mender/.

--

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+un...@lists.mender.io.

[Mirza Krak]

On Wed, Aug 1, 2018 at 3:33 PM, Alan Martinovic <alan.ma...@senic.com> wrote:

Hi,
I've noticed that mender decides which partition is being booted
by passing a kernel paramter root=/dev/mmcblk0p2 to the kernel.

Correct.

 

So the parameter is hard codded into and the booting is decided
by the partition index (mmcblk0p1 or mmcblk0p2). 

This is configurable at build-time but only affects the .sdimg file. 

What I've been witnessing recently after needing to update the
kernel on my device is that sometimes the naming of mmcblk0 changes.
It becomes mmcblk1 or mmcblk2.
When this happens the system stops since it can't mount the rootfs.

In an attempt to see why has this started to happen, I've discovered
that it's normal kernel behavior not to provide stable indexes for device
files and that it randomly depends on the probing order.

Here is a link
https://lkml.org/lkml/2016/4/29/817


Has anyone experienced issues like these?

This is fairly well known that in theory the indexing could change between Linux kernel updates, especially on "major" kernel versions. Should definitely not change on stable kernels.

 

How do you ensure that the naming won't change at random
when updating the kernel in the field?

I would guess that the current option is to patch the kernel should the indexing change, or more specifically patch the device-tree of the Linux kernel because that is where it could change probe order and hence change the indexing.

UUID`s are probably a more stable approach and it might be possible to set these to fixed values during build time, I know at least that the wic tool has options like "--fsuuid" but I have not really tried this out but could be worth investigating.

-- 

Mirza Krak | Embedded Solutions Architect | https://mender.io

 Northern.tech AS | @northerntechHQ

[Alan Martinovic]

Thanks for the reply Mirza.

> This is configurable at build-time but only affects the .sdimg file. 

There is a hard coded env variable (script) as a result of mender's integration into uboot: 

mender_setup=setenv mender_kernel_root /dev/mmcblk2p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk2p2; else setenv mender_boot_part_name /dev/mmcblk2p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs

Aha...

I guess that's what you meant. It influences the .sdimage as opposed to .mender one.

> UUID`s are probably a more stable approach and it might be possible to set these to fixed values during build time, I know at least that the wic tool has options like "--fsuuid" but I have not really tried this out but could be worth investigating

I don't yet understand how the suggested UUID approach scales when you want to use

the same image on say 10000 production devices. 

So far labels seem to be perhaps a more scalable solution.

i.e. /dev/disk/by-label/primary

> I would guess that the current option is to patch the kernel should the indexing change, or more specifically patch 

the device-tree of the Linux kernel because that is where it could change probe order and hence change the indexing.

That seems like our best approach for now yes,

to be honest it does seem hacky :)

Changing the device tree until the naming gets right again by chance and considering

that a reliable option...

Have tried getting a PoC with initramfs and them manually map "root=/dev/mmcblk0p2"
to the correct /dev/disk/by-label/ regardless what index the kernel gave the mmcblk.

This, as I see it, promises to both keep backwards compatibility with deployed devices

and is immune to kernels naming changes in the future (one less regressions to 

worry about when updating device trees and kernels in the field).

Struggling still to integrate initramfs into the kernel with yocto though.... :( 

--
You received this message because you are subscribed to the Google Groups "Mender List mender.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mender+unsubscribe@lists.mender.io.

[Mirza Krak]

On Mon, Aug 6, 2018 at 12:16 PM, Alan Martinovic <alan.ma...@senic.com> wrote:

Thanks for the reply Mirza.

> This is configurable at build-time but only affects the .sdimg file. 

There is a hard coded env variable (script) as a result of mender's integration into uboot: 

mender_setup=setenv mender_kernel_root /dev/mmcblk2p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk2p2; else setenv mender_boot_part_name /dev/mmcblk2p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs

Aha...

I guess that's what you meant. It influences the .sdimage as opposed to .mender one.

> UUID`s are probably a more stable approach and it might be possible to set these to fixed values during build time, I know at least that the wic tool has options like "--fsuuid" but I have not really tried this out but could be worth investigating

I don't yet understand how the suggested UUID approach scales when you want to use

the same image on say 10000 production devices. 

So far labels seem to be perhaps a more scalable solution.

i.e. /dev/disk/by-label/primary

It is quite common to use UUID to get around this problem, and I do not really see a problem with scale.

The kernel supports following root arguments:

- root=PARTUUID=partition_uuid, 
- root=UUID=fs_uuid

If one is able to set "fs_uuid" to a fixed value on the rootfs parts (different of course), in that case the UUID will _always_ be the same and be resilient to indexing changes due to kernel updates.

The idea is _not_ to set a unique ID on every device, which you might be refereeing to as you mention number of devices.

Again this is in theory and I haven't actually tried this and Mender does not yet support this, but I believe that it might be possible as the "wic" tool has an "--fsuuid" option [1] and "wic" is used to generate the .sdimg files.

 

> I would guess that the current option is to patch the kernel should the indexing change, or more specifically patch 

the device-tree of the Linux kernel because that is where it could change probe order and hence change the indexing.

That seems like our best approach for now yes,

to be honest it does seem hacky :)

Changing the device tree until the naming gets right again by chance and considering

that a reliable option...

Yeah, I did not say it was pretty :). But I do not really understand what you mean "by chance"?

[1]. https://patchwork.openembedded.org/patch/145542/

[Alan Martinovic]

> The idea is _not_ to set a unique ID on every device, which you might be refereeing to as you mention number of devices.

Oh, this is pretty cool. 

Was always under the impression that the kernel generates the UUIDS dynamically.

Yeah this would work great.

Thanks for sharing.

> But I do not really understand what you mean "by chance"?

Well...

If you need another entry in a device tree at some point in the future, there is always 

this regression lurking by that the change will result in the mmcblk name change.

And knowing your system to a level in which you can tell the kernels probing

order and relating that to the naming is a lot of determinism on a kernel level.

My point is that it seems like an expensive determinism to gain and if you aren't a big company

with dedicated kernel engineers. your best bet is trial and error device tree node

reordering (or "fixing it by chance" and hoping it doesn't break on the next change again) :)