Lenovo n20p: kernel blob and boot options

[gm]

Hi there,

I've recently started using a Lenovo n20p chromebook and - since I discovered it has no SeaBIOS - I'd like to experiment alternative ways of booting.

In particular, I am interested in the following:

1) creating and flashing a different kernel blob, with a v3.10.18 kernel built from official sources (https://chromium.googlesource.com/chromiumos/third_party/kernel/) but with my own config (enabling NFS module etc)

2) booting from an external USB device with a different root= kernel command line parameter, either using the official ChromeOS kernel or my own (as per [1] above)

Regarding (1), I have come to a blocker because I don't know what to provide to vbutil_kernel for the --bootloader option. Documentation is not clear about this apparently outdated kernel version. My verbose config is currently as follows (vanilla ChromeOS):

```

Key block:

  Signature:           ignored

  Size:                0x4b8

  Flags:               7  !DEV DEV !REC

  Data key algorithm:  4 RSA2048 SHA256

  Data key version:    1

  Data key sha1sum:    REDACTED

Preamble:

  Size:                0xfb48

  Header version:      2.2

  Kernel version:      1

  Body load address:   0x100000

  Body size:           0x440e00

  Bootloader address:  0x539000

  Bootloader size:     0x4000

  Vmlinuz header address: 0x53d000

  Vmlinuz header size:    0x3e00

  Flags          :       0x0

Body verification succeeded.

Config:

console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro disablevmx=off lsm.module_locking=0 dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 2506752 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=2506752 alg=sha1 root_hexdigest=REDACTED salt=REDACTED" noinitrd vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic 

```

However I have not been able to extract the bootloader as I do not understand what is the absolute offset in the current kernel partition #4. Can somebody provide insight?

Also, regarding both (1) and (2), will I risk to make the device completely unbootable? There is a valid kernel (probably older version, as updated by ChromeOS) on partition #2, so I am currently living under the assumption that when I'll change the kernel command line as per (2) or boot my own blob (as per [1]), I'd eventually still be able to get it booting thanks to the backup kernel. But I wonder: what if the kernel I try to boot hangs up? How can I in that case boot the 2nd one?

Thanks to anybody that will have the patience to help a bit, as I am new to this type of devices :)

--

  gm

[Mike Frysinger]

we really should make the --bootloader argument optional at this point.  you can try giving it a blank file:

  truncate -s 512b stub.bin

in the CrOS world, we feed it the EFI stub:

  /lib64/bootstub/bootstub.efi

but afaik that only matters on EFI systems and doesn't apply to official Chrome OS devices like the n20

as long as you do not disable the firmware write protect, you cannot brick the system.  the worst that would happen is that you'd have to get a usb recovery image and have the bootloader (which doesn't live on the ssd) reimage the device.

wrt kernel selection, make sure your gpt flags have your kernel set with a higher priority otherwise the bootloader will keep selecting the older one.

-mike

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[gm]

On Saturday, 30 May 2015 06:43:40 UTC+2, Mike Frysinger wrote:

we really should make the --bootloader argument optional at this point.  you can try giving it a blank file:

  truncate -s 512b stub.bin

in the CrOS world, we feed it the EFI stub:

  /lib64/bootstub/bootstub.efi

but afaik that only matters on EFI systems and doesn't apply to official Chrome OS devices like the n20

The docs already say that it can be fed a dummy file, but I wasn't sure it would apply also to my device, which as I understand is not really cutting-edge. Many thanks for your reply Mike!

On Saturday, 30 May 2015 06:43:40 UTC+2, Mike Frysinger wrote: 

as long as you do not disable the firmware write protect, you cannot brick the system.  the worst that would happen is that you'd have to get a usb recovery image and have the bootloader (which doesn't live on the ssd) reimage the device.

Ok, good to know. I haven't opened the device so the firmware write protect is still there in place.

 

wrt kernel selection, make sure your gpt flags have your kernel set with a higher priority otherwise the bootloader will keep selecting the older one.

Thanks, I got this part right. I will try to boot a debian-style vmlinuz and a plain ChromeOS one, should be able to find my way. What I have noticed is that by default I get only a blank screen if there is something wrong with the kernel, maybe it's because the video is not correctly initialized.  Of course I'd be drooling to see what's going on via serial interface, but that's a bit out of reach for me :)

--

  gm

[Mike Frysinger]

you have to feed it a bzImage regardless as that's the only one that's executable on x86.  CrOS doesn't use an initramfs, but i think Debian does ... but that shouldn't matter either way.  don't confuse it with a vmlinux (which is typically an ELF).  usually distros install it as /boot/vmlinuz*.

if you have a USB RS232 dongle, you could try enabling that driver and setting that as a console.

-mike

[gm]

On Saturday, 30 May 2015 16:25:21 UTC+2, Mike Frysinger wrote:

you have to feed it a bzImage regardless as that's the only one that's executable on x86.  CrOS doesn't use an initramfs, but i think Debian does ... but that shouldn't matter either way.  don't confuse it with a vmlinux (which is typically an ELF).  usually distros install it as /boot/vmlinuz*.

What I fed to --vmlinuz option was exactly the one put at /boot/vmlinuz* after building and installing a deb with make-kpkg, my bad for not reading it must be a bzImage (vbutil_kernel help says so already). This time I'll use `make bzImage` and the artifact generated by that.

Is there any way to iterate faster? What I did now after this screwup was enable verification (that accounts for a powerwash), then enable developer mode (which is a second powerwash). Ideally I'd just tell the bootloader "hey please go with the kernel in 2nd-priority partition now".

 

if you have a USB RS232 dongle, you could try enabling that driver and setting that as a console.

Mmh..good tip, there must be a franken-RS232 dongle around here, if I dig deep enough..

Thanks again!

[Mike Frysinger]

if your kernel is failing to boot fully (and it sounds like it is), then the firmware/update logic has fallback support built in.  it'll try to boot the active kernel a few times before giving up and trying the next one.  the key is to not mark the partition as successful.  see this doc for details:

http://dev.chromium.org/chromium-os/chromiumos-design-docs/disk-format#TOC-Selecting-the-kernel

i don't have a device next to me to double check, but i think you want to set the flags like:

 - tries field to 1

 - successful to 0

 - highest priority

then when it fails to boot the first time, power cycle/reboot it, and it should fallback to the other.

-mike

[gm]

On Saturday, 30 May 2015 17:23:52 UTC+2, Mike Frysinger wrote:

if your kernel is failing to boot fully (and it sounds like it is), then the firmware/update logic has fallback support built in.  it'll try to boot the active kernel a few times before giving up and trying the next one.  the key is to not mark the partition as successful.  see this doc for details:

http://dev.chromium.org/chromium-os/chromiumos-design-docs/disk-format#TOC-Selecting-the-kernel

i don't have a device next to me to double check, but i think you want to set the flags like:

 - tries field to 1

 - successful to 0

 - highest priority

then when it fails to boot the first time, power cycle/reboot it, and it should fallback to the other.

-mike

Thanks mike! I used gdisk and flipped the bits so that the partition I am flashing is no more considered successful and it'll try max twice (bit #53).

After a few failures it booted the developer-mode powerwash, and since I flashed only kernel B so far I guess that's what is stored on the next-priority kernel.

Regarding the kernel I built myself, I don't know what might be going wrong. I assembled the blob with:

vbutil_kernel --pack newkernel.blob \

  --keyblock /usr/share/vboot/devkeys/kernel.keyblock \

  --version 1 \

  --bootloader dummy-512-zero \

  --signprivate /usr/share/vboot/devkeys/kernel_data_key.vbprivk \

  --config chromeos-cmdline \

  --vmlinuz bzImage \

  --arch x86_64

The `chromeos-cmdline` contained the kernel commandline options of the previously running ChromeOS (a.k.a. vbutil_kernel --verify /dev/mmcblk0p4 --verbose | tail -1).

I also disabled rootfs verification and replaced /lib/modules/3.10.18 on the rootfs with my built modules, and merged /lib/firmware too.

Right now I am trying to boot the built kernel using https://chromium.googlesource.com/chromiumos/third_party/kernel/ branch v3.10.18, then I might try some other Linux OS.

--

  gm

[gm]

Forgot to add: except for the first attempt, when the Chromebook rebooted itself, I had always to poweroff the device with a power button longpress, to avoid staring at a blank screen.

--

  gm

[Mike Frysinger]

i think you should start from a known working place and change things step by step

for example, vbutil_kernel has a --get-vmlinuz option that'll extract the bzImage from an existing verified boot kernel image.  you could use that to pull the kernel out of the working kernel A slot, then try running your custom vbutil_kernel --pack options, and write the result to kernel B slot.  once you have a working combination, you can be confident in which parts are OK and focus on the new parts.

note: "v3.10.18" is not a branch, it's a tag.  that's the upstream stable kernel which contains none of our changes.  if you want the same kernel that is used on n20p, you should use one of the release branches instead like release-R44-7077.B-chromeos-3.10.  you probably should start with the kernel config that your n20p is currently running as well and once you have that booting, start changing options to match your needs.

-mike

[gm]

On Saturday, 30 May 2015 21:36:47 UTC+2, Mike Frysinger wrote:

i think you should start from a known working place and change things step by step

for example, vbutil_kernel has a --get-vmlinuz option that'll extract the bzImage from an existing verified boot kernel image.  you could use that to pull the kernel out of the working kernel A slot, then try running your custom vbutil_kernel --pack options, and write the result to kernel B slot.  once you have a working combination, you can be confident in which parts are OK and focus on the new parts.

Yes, I had just noticed that option and this was exactly the next test on my list.

 

note: "v3.10.18" is not a branch, it's a tag.  that's the upstream stable kernel which contains none of our changes.  if you want the same kernel that is used on n20p, you should use one of the release branches instead like release-R44-7077.B-chromeos-3.10.  you probably should start with the kernel config that your n20p is currently running as well and once you have that booting, start changing options to match your needs.

That is probably the issue. Actually now the "third-party" of the path makes sense...I will indeed try with one of the release branches, many thanks again Mike!

Yes, I am using the current kernel config with only difference being NFS enabled, but I'll eventually try again with the vanilla config.

--

  gm

[gm]

A repacked kernel (by using the extracted vmlinuz) works if arch == x86. By reading this page https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/custom-firmware I thought I had depthcharge bootloader, however seems like the --arch arm advice given there might not be valid for my device (I have tried both arm and x86_64). Will keep digging, however I might switch to using the CROS SDK and an external USB instead of my current approach.

--

  gm

[gm]

I've started using the SDK and read enough documentation to realize that everything is in high flux, I should go through it much more carefully :)

Summary:

- the bzImage must be served to the bootloader as an x86 (--arch x86). This doc page (https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/custom-firmware) might need to be amended (I know, probably not the only one..)

- the bootloader is coreboot, as documented here: https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices/lenovo-chromebook-n20 (there are even sources, that's nice :)

- according to the above page, which refers to the Acer C720 page for development instructions, it *IS* possible to boot from USB :) when I originally posted here I had given up on this

Also in the developers guide, may I suggest that the section "Installing Chromium OS on your Device" gets changed? In particular where it says "This section is NOT for use on the Google Chrome devices", yet - AFAIK - it is legit to use cros flash if you want to boot a kernel with Ctrl+U from a Google Chrome device.

Right now I am waiting to build everything via SDK, so that I'll try the boot from USB.

By the way, nice to see the good old Gentoo inside the chroot :)

--

  gm

[Mike Frysinger]

i've cleaned up & merged the sections in the developer guide as you pointed out.  thanks!

-mike

[gm]

On Sunday, 31 May 2015 05:00:35 UTC+2, Mike Frysinger wrote:

i've cleaned up & merged the sections in the developer guide as you pointed out.  thanks!

Thanks, much appreciated! It reads better now :)

An update on my quest.

My latest Google-provided ChromeOS has firmware Google_Clapper.5216.199.7, platform 6812.88.0 stable-channel clapper.

If I press Cltr+I at the scary screen, I see (amongst others):

...

dev_boot_usb: 1

dev_boot_legacy: 1

dev_boot_signed_only: 1

gbb.flags: 0x0000000

...

Yesterday I built release-R44-7077.B (passed to the ~/chromiumos repo init) for board clapper and then after about 6 hours of Gentoo love (and 60GB of byproducts) I could flash to USB. Out of desperation, I also built a ChrUbuntu stick (using http://goo.gl/9sgchs).

Well, with both USB sticks and regardless of the dev_boot_signed_only value, I get either 2 low-tone beeps for a Ctrl+L press or a single high-pitch beep for Ctrl+U. These are the same beeps I get without anything plugged in at all, so I know it's not even trying to check what's on the USB sticks. I remember though, on another day, that I could see a screen saying something like "invalid SD/USB inserted", I wonder how I could get back there...

I also tried a dd of the currently-working kernel on the SSD (B) directly to the USB highest-prio kernel partition (I know it would still point to the SSD's rootfs), but same result. There's something not clicking here...

These are the coreboot sources I should be running: https://chromium.googlesource.com/chromiumos/third_party/coreboot/+/firmware-clapper-5216.199.B however I don't know which payload(s) I have enabled :(

On a side note, for my CROS SDK usage would it be worth building another branch for example? Utterly ignorant of the differences, I was looking with interest at these:

• firmware-clapper-5216.199.B
  
• stabilize-6812.85.B-chromeos-3.10
  

For now I'll make other attempts by using the original-vmlinuz-repacking technique which seemed more promising; as I have huge faith in human errors, am I right thinking that there are just some technical quirks between me and booting my own kernel, and not any purposefully-added blocking mechanism?

[gm]

It turns out that a specific type of USB sticks of mine are not seen at all by the bootloader.

For reference:

> New USB device found, idVendor=058f, idProduct=6387

> 

They report a "Generic" manufacturer; these little critters will now be marked with a big red X.. :s

I could successfully boot without issues my SDK-built image by using a Kingston DataTraveler 2.0 instead, and I instantly recognized the Chromium logo. That is progress, now :)

I noticed that switching to VT2 gives a nasty flickering/overlapping effect, maybe I'll try some other branch.

--

  gm

[gm]

No more issues at booting from USB and/or with a custom kernel (I use cros_sdk to build it) :) I have an issue with wifi probably due to the new defaults of the kernel, but I know how to troubleshoot that.

Many thanks again Mike for the support!

May I suggest that the following gets added somewhere on page https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/kernel-configuration ? Perhaps as sub-header 9?

[~ snip

Activate your commit in the kernel ebuild

=========================

Once your changes to the kernel configuration are complete, commit them and take note of the commit hash, then place it in the ebuild declaration CROS_WORKON_COMMIT in the corresponding chromeos-kernel ebuild, for example: (cros_sdk chroot path) ~/trunk/src/third-party/chromiumos-overlay/sys-kernel/chromeos-kernel-3_10-3.10.18-r681.ebuild.

You can use ``emerge --search chromeos-kernel`` to find available ebuilds.

~snip]

If there's a more streamlined way to do this, then please ignore the above..I am really new to all the cros* tools, the SDK is huge :)

--

  gm

[Yuly Novikov]

cros_workon --board=${BOARD} start chromeos-kernel-3_10

[gm]

Thanks Yuly, will definitively try this.

Please note that the USB issue I reported with idVendor=058f, idProduct=6387 sticks is still valid, also for recovery mode - although might be a known issue and/or not important.

--

  gm

[Mike Frysinger]

i've tweaked the docs slightly to mention the need for chromeos-kernel-$VER selection

-mike