mmcblk dynamic device numbering and U-Boot root= kernel command-line generation

[Stephen Warren]

On Tegra, U-Boot assumes that its SD/MMC device numbering exactly matches
kernel mmcblk device numbering. This assumption is encoded into the kernel
command-line root= options that U-Boot (and the ChromeOS U-Boot scripts on
disk) generate. In practice, this is not always true.

I believe U-Boot statically assigns IDs to specific host controllers; on
Seaboard, ID 0 is internal MMC, and ID 1 is SD slot.

The kernel dynamically assigns mmcblk device IDs to whichever device is
identified first.

In practice, at least on Seaboard, I believe through some timing fluke,
kernel chromeos-2.6.37 usually identifies the devices in the same order
as U-Boot's ID assignments, and hence ends up with the same numbering as
U-Boot. However, this isn't always true; occasionally this kernel will
identify things "backwards".

In practice, at least with my Seaboard clamshell, kernel chromeos-2.6.38
usually identifies the devices in the reverse order to U-Boot.

When U-Boot and kernel order happen to match, everything just works.

When U-Boot and kernel order mismatch, two things can happen: (a) the
kernel mounts the root fs from a device other than expected, but the
system then probably boots OK, or (b) the other device has no filesystem
on it, and the kernel panics when it attempts to mount the root filesystem
from there, and can never boot.

I started a discussion on the linux-mmc mailing list re: whether the
mmcblk device IDs could be assigned statically; see
http://comments.gmane.org/gmane.linux.kernel.mmc/7140. In that thread,
Chris Ball (MMC maintainer) indicated that this is typically solved by
an initrd rather than the kernel.

So, should ChromeOS start to use an initrd to solve this? Or, should
ChromeOS carry something like that patch I mentioned, which causes the
kernel to assign mmcblk device IDs statically based on the host controller
that contains the block device? Note that with that patch, for Seaboard,
the mapping from U-Boot to kernel device IDs isn't exactly 1:1, even though
it is static:

U-Boot ID Kernel ID
0 2
1 1

(perhaps a more complex patch could be both static and fix the mapping
table to be 1:1)

I'd be pleased to hear anyone's thoughts on this.

Note: I've observed similar issues on our Ventana boards when using
mainline kernels there too.

--
nvpublic

[Olof Johansson]

Hi,

On Mon, Apr 4, 2011 at 2:25 PM, Stephen Warren <swa...@nvidia.com> wrote:

[description of how having dynamic block device numbers makes life complicated]

> I started a discussion on the linux-mmc mailing list re: whether the
> mmcblk device IDs could be assigned statically; see
> http://comments.gmane.org/gmane.linux.kernel.mmc/7140. In that thread,
> Chris Ball (MMC maintainer) indicated that this is typically solved by
> an initrd rather than the kernel.
>
> So, should ChromeOS start to use an initrd to solve this? Or, should
> ChromeOS carry something like that patch I mentioned, which causes the
> kernel to assign mmcblk device IDs statically based on the host controller
> that contains the block device? Note that with that patch, for Seaboard,
> the mapping from U-Boot to kernel device IDs isn't exactly 1:1, even though
> it is static:

[...]

> I'd be pleased to hear anyone's thoughts on this.

Yeah, this is a generic issue with anyone trying to assume which
device will probe first on the kernel. For SATA and USB, there's
similar issues (but there might be less runtime variation there).

We had some long discussions over this on x86 back when we first
started looking at recovery mode booting, etc.

On a chromeos build, we have a uuid specifier for the filesystem to
use, so there are no references to actual device names. This is
similar to how ramdisks handle it, but it's done in-kernel.

We don't want a ramdisk for boottime performance reasons (and for the
hassle of having one).

The problem still exists for developer builds where you build your
image with --noenable_rootfs_verification. We have not yet worried
about solving this, but I guess there's some need to do so now.. :)

-Olof

[Olof Johansson]

[from an address that will post on the list this time]

Hi,

On Mon, Apr 4, 2011 at 2:25 PM, Stephen Warren <swa...@nvidia.com> wrote:

[description of how having dynamic block device numbers makes life complicated]

> I started a discussion on the linux-mmc mailing list re: whether the

> mmcblk device IDs could be assigned statically; see
> http://comments.gmane.org/gmane.linux.kernel.mmc/7140. In that thread,
> Chris Ball (MMC maintainer) indicated that this is typically solved by
> an initrd rather than the kernel.
>
> So, should ChromeOS start to use an initrd to solve this? Or, should
> ChromeOS carry something like that patch I mentioned, which causes the
> kernel to assign mmcblk device IDs statically based on the host controller
> that contains the block device? Note that with that patch, for Seaboard,
> the mapping from U-Boot to kernel device IDs isn't exactly 1:1, even though
> it is static:

[...]

> I'd be pleased to hear anyone's thoughts on this.

[Scott James Remnant]

On Mon, Apr 4, 2011 at 2:25 PM, Stephen Warren <swa...@nvidia.com> wrote:

 

On Tegra, U-Boot assumes that its SD/MMC device numbering exactly matches
kernel mmcblk device numbering. This assumption is encoded into the kernel
command-line root= options that U-Boot (and the ChromeOS U-Boot scripts on
disk) generate. In practice, this is not always true.

Are there any reasons why you can't use the UUID of the filesystem for this, rather than hardcoding a particular block device?

Scott

[Olof Johansson]

See comment about ramdisk -- the UUID to block device translation has
traditionally been done from userspace on a ramdisk. And we don't want
one.

There's been recent work to do it in-kernel though. Not sure if it's
all in place though, I haven't keep track of it.

-Olof

[Scott James Remnant]

Yeah, I thought there had been kernel work to be able to do this now for the useful filesystems.

After all, the kernel knows all the block devices, and has all of the different filesystem code which tend to know how to read UUIDs.

My e-mail was meant more in the "could we use this?" tone

Scott

[Bill Richardson]

On x86, the way this works is that the verified boot stuff works in cooperation with the custom Chrome OS BIOS to return the GPT's Unique Partition GUID for the verfied kernel. This is passed to the bootstub, which then expands any %U strings in the default kernel command line into the actual kernel partition's GUID. The dm-verity driver uses a %U to identify the kernel partition, and a %U+1 to identify the rootfs partition (that is, find the kernel partition, then mount the next partition on the same drive). For ARM, U-boot needs to perform the same %U expansion before handing off to the kernel. I believe that Will successfully upstreamed a kernel change to understand root=%U, although I'm not certain he has yet gotten rootfs=%U+n, which is really what's needed to boot reliably without dm-verity.

Note that this all works using the UUID from the GPT itself, not from any filesystem-specific UUIDs. As long as the UUIDs are really unique, there should only be one matching partition on the system, which will be correctly identified by both the bootloader and the kernel.

There is some discussion in http://code.google.com/p/chromium-os/issues/detail?id=5081, and I'm sure there was more at one time, but I can't seem to find the original bug.

Will touched this last; maybe he'll have more to add.

Bill

--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

--

Art for Art's Sake

Engineering for Money

[Bill Richardson]

On Mon, Apr 4, 2011 at 4:06 PM, Bill Richardson <wfri...@chromium.org> wrote:

Note that this all works using the UUID from the GPT itself, not from any filesystem-specific UUIDs. As long as the UUIDs are really unique, there should only be one matching partition on the system, which will be correctly identified by both the bootloader and the kernel.

We don't use the filesystem's UUIDs for two reasons: the drivers that understand those take extra time to load and run while the UUIDs in the GPT don't require any filesystem scanning, and more importantly, the filesystem UUID is unique only to the filesystem, not to the content of the filesystem. If we install or backup using dd instead of cp -ra, the new filesystem will also have the same UUID, and we can't tell one from the other.

Bill

[Scott James Remnant]

Makes sense; using the GPT UUID would be even better - I always forget about them because we had to care about MBR in Ubuntu-land and couldn't use EFI because it made the nvidia binary blob driver not work. We did vaguely talk about switching to GPT at various points, but it never happened.

Does the ARM boot loader not understand these? We're using GPT on /dev/mmcblk0 right?

Scott

[Mandeep Singh Baines]

Bill Richardson (wfri...@chromium.org) wrote:
> On x86, the way this works is that the verified boot stuff works in
> cooperation with the custom Chrome OS BIOS to return the GPT's Unique
> Partition GUID for the verfied kernel. This is passed to the bootstub, which
> then expands any %U strings in the default kernel command line into the
> actual kernel partition's GUID. The dm-verity driver uses a %U to identify
> the kernel partition, and a %U+1 to identify the rootfs partition (that is,
> find the kernel partition, then mount the next partition on the same drive).
> For ARM, U-boot needs to perform the same %U expansion before handing off to
> the kernel. I believe that Will successfully upstreamed a kernel change to
> understand root=%U, although I'm not certain he has yet gotten rootfs=%U+n,
> which is really what's needed to boot reliably without dm-verity.
>

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b5af921ec02333e943efb59aca4f56b78fc0e100

[Will Drewry]

On Mon, Apr 4, 2011 at 6:06 PM, Bill Richardson <wfri...@chromium.org> wrote:
> On x86, the way this works is that the verified boot stuff works in
> cooperation with the custom Chrome OS BIOS to return the GPT's Unique
> Partition GUID for the verfied kernel. This is passed to the bootstub, which
> then expands any %U strings in the default kernel command line into the
> actual kernel partition's GUID. The dm-verity driver uses a %U to identify
> the kernel partition, and a %U+1 to identify the rootfs partition (that is,
> find the kernel partition, then mount the next partition on the same drive).
> For ARM, U-boot needs to perform the same %U expansion before handing off to
> the kernel. I believe that Will successfully upstreamed a kernel change to
> understand root=%U, although I'm not certain he has yet gotten rootfs=%U+n,
> which is really what's needed to boot reliably without dm-verity.
> Note that this all works using the UUID from the GPT itself, not from any
> filesystem-specific UUIDs. As long as the UUIDs are really unique, there
> should only be one matching partition on the system, which will be correctly
> identified by both the bootloader and the kernel.
> There is some discussion
> in http://code.google.com/p/chromium-os/issues/detail?id=5081, and I'm sure
> there was more at one time, but I can't seem to find the original bug.
> Will touched this last; maybe he'll have more to add.

Yeah - I haven't tried to push up a root=PARTUUID=%U+n yet, but
support for PARTUUID=%U is upstream. The code in the kernel is
implemented in two parts:
1. the root= parser
2. partition scanner.

When the partitions are scanned, we add extra metadata about the
partition UUID if supplied. I only landed support for EFI upstream
but the mechanism is generic. Then during root=[device] resolution,
the [device] scan is forked based on whether it looks like a UUID and
punts it over to the uuid parser then walks all registered block
devices looking for a matching partition uuid. First-come-first serve
since they're unique... right :)

that's kinda where it's at :/
will

[Will Drewry]

On Tue, Apr 5, 2011 at 11:11 AM, Mandeep Singh Baines <m...@chromium.org> wrote:
> Bill Richardson (wfri...@chromium.org) wrote:
>> On x86, the way this works is that the verified boot stuff works in
>> cooperation with the custom Chrome OS BIOS to return the GPT's Unique
>> Partition GUID for the verfied kernel. This is passed to the bootstub, which
>> then expands any %U strings in the default kernel command line into the
>> actual kernel partition's GUID. The dm-verity driver uses a %U to identify
>> the kernel partition, and a %U+1 to identify the rootfs partition (that is,
>> find the kernel partition, then mount the next partition on the same drive).
>> For ARM, U-boot needs to perform the same %U expansion before handing off to
>> the kernel. I believe that Will successfully upstreamed a kernel change to
>> understand root=%U, although I'm not certain he has yet gotten rootfs=%U+n,
>> which is really what's needed to boot reliably without dm-verity.
>>
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=b5af921ec02333e943efb59aca4f56b78fc0e100

Thanks! Jens cleaned up some messed up attributes I had in that
patch, but on the whole that is it. It's a pretty easy hack to add +n
to the change. Just change the length check for PARTUUID to < 36
(instead of !=) then add a parser to the split for the offset (+/-%u)
like we did in dm-verity.

cheers!
will

[Bill Richardson]

On Tue, Apr 5, 2011 at 9:16 AM, Will Drewry <w...@chromium.org> wrote:

Thanks!  Jens cleaned up some messed up attributes I had in that

patch, but on the whole that is it.  It's a pretty easy hack to add +n
to the change.  Just change the length check for PARTUUID to < 36
(instead of !=) then add a parser to the split for the offset (+/-%u)
like we did in dm-verity.

Right. The +/-%n stuff is in dm_get_device_by_uuid(), in  

third_party/kernel-next/drivers/md/dm-verity.c.

Are any kernel people working to push the corresponding change upstream, so we can use

"root=%U+1" in the kernel command line? That would be nice (hint, hint). :-)

Bill