Modified kernel command line does not have any effect.

[Andrey Tarasevich]

Hello!

I'm using Chromium OS qemu image which is build in a standard way and
modified for tests. Then I modify kernel command line like described
here http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format#TOC-Quick-development

The original cmd is following:
quiet loglevel=1 console=tty2 init=/sbin/init add_efi_memmap
boot=local noresume noswap i915.modeset=1 cros_secure kern_guid=%U
tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic mutant=1
root=PARTUUID=%U/PARTNROFF=1 rootwait ro dm_verity.error_behavior=3
dm_verity.max_bios=-1 dm_verity.dev_wait=0 dm="vroot none ro,0 1740800
verity ROOT_DEV HASH_DEV 1740800 0 sha1
2d90263c9c30d1201dad4336c6e2b7376b3b3cdb" noinitrd cros_debug

The modified:
quiet loglevel=1 console=tty2 init=/sbin/init add_efi_memmap
boot=local noresume noswap i915.modeset=1 cros_secure kern_guid=%U
tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic mutant=1
root=PARTUUID=%U/PARTNROFF=1 rootwait ro dm_verity.error_behavior=3
dm_verity.max_bios=-1 dm_verity.dev_wait=0 dm="vroot none ro,0 1740800
verity ROOT_DEV HASH_DEV 1740800 0 sha1
2d90263c9c30d1201dad4336c6e2b7376b3b3cdb" noinitrd cros_debug mutant=1


At the moment I just check the argument value by simple printk() and
then just look at kernel logs of the image. But it does not make any
effect. It seems to me like argument is not passed to the kernel at
all. Although I checked the kernel partition with vbutil_kernel --
verify kernel.old --verbose
And this what I get.
Key block:
Signature: ignored
Size: 0x4b8
Flags: 7 !DEV DEV !REC
Data key algorithm: 4 RSA2048 SHA256
Data key version: 1
Data key sha1sum: d6170aa480136f1f29cf339a5ab1b960585fa444
Preamble:
Size: 0xfb48
Header version: 2.0
Kernel version: 1
Body load address: 0x100000
Body size: 0x33a000
Bootloader address: 0x436000
Bootloader size: 0x4000
Body verification succeeded.
Config:
quiet loglevel=1 console=tty2 init=/sbin/init add_efi_memmap
boot=local noresume noswap i915.modeset=1 cros_secure kern_guid=%U
tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic
root=PARTUUID=%U/PARTNROFF=1 rootwait ro dm_verity.error_behavior=3
dm_verity.max_bios=-1 dm_verity.dev_wait=0 dm="vroot none ro,0 1740800
verity ROOT_DEV HASH_DEV 1740800 0 sha1
5b515546749feba739bb3a762422bc947aea1fdc" noinitrd cros_debug mutant=1

So it seems to me that kernel command line is fine. I'm sure, that
from the side of the kernel everything is correct, since I tried to
modify build_image.sh script and hardcoded new argument in the script
and it worked. But this is not what I need at the moment, since
building whole image takes too much time.

Am I missing something?

thanks,
Andrey Tarasevich

[Andrey Tarasevich]

The funny thing is that I'm aware of the different kernel partitions and I'm running the right one. Here is what I do and what I get:

sevich@sevich-desktop:~/master/chromiumos/src/scripts$ ./bin/cros_start_vm --snapshot --image_path=../build/images/x86-generic/0.15.895.2011_08_15_2137-a1/chromiumos_qemu_image.bin 

Starting a KVM instance

pci_add_option_rom: failed to find romfile "pxe-virtio.bin"

INFO    : KVM started with pid stored in /tmp/kvm.5217.pid

ssh root@sevich-desktop -p 9222 -o StrictHostKeyChecking=no

sevich@sevich-desktop:~/master/chromiumos/src/scripts$ ssh root@localhost -p 9222 -o StrictHostKeyChecking=no

Warning: Permanently added '[localhost]:9222' (RSA) to the list of known hosts.

Password: 

localhost ~ # cat /proc/cmdline 

quiet console=tty2 init=/sbin/init boot=local rootwait ro noresume noswap loglevel=1 noinitrd root=/dev/sda3 i915.modeset=1 cros_legacy cros_debug BOOT_IMAGE=vmlinuz.A 

localhost ~ # dd if=/dev/sda2 of=/var/kernel.old

32768+0 records in

32768+0 records out

16777216 bytes (17 MB) copied, 0.162956 s, 103 MB/s

localhost ~ # vbutil_kernel --verify /var/kernel.old --verbose

Key block:

  Signature:           ignored

  Size:                0x4b8

  Flags:               7  !DEV DEV !REC

  Data key algorithm:  4 RSA2048 SHA256

  Data key version:    1

  Data key sha1sum:    d6170aa480136f1f29cf339a5ab1b960585fa444

Preamble:

  Size:                0xfb48

  Header version:      2.0

  Kernel version:      1

  Body load address:   0x100000

  Body size:           0x33a000

  Bootloader address:  0x436000

  Bootloader size:     0x4000

Body verification succeeded.

Config:

quiet loglevel=1 console=tty2 init=/sbin/init add_efi_memmap boot=local noresume noswap i915.modeset=1 cros_secure kern_guid=%U tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic root=PARTUUID=%U/PARTNROFF=1 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=0 dm="vroot none ro,0 1740800 verity ROOT_DEV HASH_DEV 1740800 0 sha1 5b515546749feba739bb3a762422bc947aea1fdc" noinitrd cros_debug mutant=5 

localhost ~ # rootdev -s

/dev/sda3

As you can see the modification "mutant=5" is present in kernel partition, but not in /proc/cmdline

Cheers, 

Andrey Tarasevich

2011/8/22 Vadim Bendebury <vbe...@chromium.org>

On Mon, Aug 22, 2011 at 11:16 AM, Andrey Tarasevich

<tarasevi...@googlemail.com> wrote:

> Nope, it does not have these modifications.
> Cheers,
> Andrey Tarasevich
>

well, this means you are not running the modified kernel. I see that
the instructions on the page you referenced call for placing the
modified kernel into the partition 4 (sudo dd if=/tmp/new_kern.bin
/dev/sda4) of the SSD, not sure wwhy, as by default the system uses
partition 2. If `rootdev -s' on your machine returns /dev/sda3 - you
the kernel is loaded from partition 2, not 4.

If this is the case, I am not sure how comfortable you are with
hacking around this, but a fairly simple and safe way of doing it
would be

- bring up your system as is
- copy the running kernel partition to /var (dd if=/dev/sda2
of=/var/kernel.backup)
- scp the modified kernel to your machine as described
- install the new kernel on partition 2 (sudo dd if=/tmp/new_kern.bin /dev/sda2)
- reboot the system

if it fails to come up at this point (shows the recovery screen) - it
would be an indication of the fact that the new kernel was not signed
properly (the page you are looking at *might* be outdated). In that
case you would need to plug in the recovery USB stick, bring up the
system in recovery mode, enter shell, mount the SSD and use dd to copy
<mountpoint>/var/kernel.backup to /dev/sda2. This would return the
system to the original state.

Let me know if this is not clear and you have questions, good luck :)

cheers,
/vb

> 2011/8/22 Vadim Bendebury <vbe...@chromium.org>

>> does /proc/cmdline on the running system contain your modifications?
>>
>> cheers,
>> /vb
>>
>>
>> > thanks,
>> > Andrey Tarasevich
>> >
>> > --
>> > Chromium OS Developers mailing list: chromiu...@chromium.org
>> > View archives, change email options, or unsubscribe:
>> > http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en
>> >
>
>

[Bill Richardson]

I just had a brief conversation with Vadim about this, so I suspect he's preparing a detailed reply. But I can tell you quickly that the problem you're having is that you're modifying the wrong kernel. 

Short version: For each image there are three different kernels, each with its own command line. The type of BIOS determines which one gets used, and kvm uses legacy BIOS.

Long version is here: https://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/disk-format

Bill

--

Art for Art's Sake

Engineering for Money

[Andrey Tarasevich]

I'm posting this message to all in case anyone is interested. 

2011/8/22 Vadim Bendebury <vbe...@chromium.org>

Ah, sorry, you obviously know what you are doing, but  I missed the
fact that you are using a virtual machine.

What's there in /proc/cmdline sure looks very different from what is
in the kernel blob as shown by vbutil_kernel. The thing is that it is
the chromeOS firmware (H2C bios)  who gets the command line out of the
blob and supplies it to the kernel, but VM does not use the chrome OS
firmware, so the command line and the kernel itself come from the
/boot directory of the SSD if I remember correctly. This doc
(http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format)
describes it at length, please have a look.

cheers,
/vb

Sorry, that I did not mentioned from beginning, that I'm using VM in order to run Chromium OS image. 

I read the document more carefully and as far as I understand I should modify configs in /boot/syslinux/ 

But as far as I can see the command line in all files, that present in this folder, i.e. root.A.cfg, root.B.cfg and usb.A.cfg, does not fit the one, that is in /proc/cmdline on running image. Anyway I modified them also just to be sure, but that didn't work either. I really don't know where to dig. 

2011/8/22 Bill Richardson <wfri...@chromium.org>

I just had a brief conversation with Vadim about this, so I suspect he's preparing a detailed reply. But I can tell you quickly that the problem you're having is that you're modifying the wrong kernel. 

Short version: For each image there are three different kernels, each with its own command line. The type of BIOS determines which one gets used, and kvm uses legacy BIOS.

Long version is here: https://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/disk-format

 If I'm using wrong kernel how can I get access to the correct one? Since I used a bit of a brute force ;) and run vbutil_kernel --verify <partition> on all the partitions I got from running script "unpack_partitions.sh" and besides partition with the real kernel all I get is that there is no kernel blob. 

Although "vbutil_kernel --verify part_8" returned "ERROR: key_block_size advances past the end of the blob" but I don't know what does it mean, since I didn't find any source for the vbutil_kernel and googling it didn't provide any results. 

Cheers,

Andrey Tarasevich

[Vadim Bendebury]

On Mon, Aug 22, 2011 at 2:13 PM, Andrey Tarasevich
<tarasevi...@googlemail.com> wrote:
> I'm posting this message to all in case anyone is interested.
> 2011/8/22 Vadim Bendebury <vbe...@chromium.org>
>>
>> Ah, sorry, you obviously know what you are doing, but  I missed the
>> fact that you are using a virtual machine.
>>
>> What's there in /proc/cmdline sure looks very different from what is
>> in the kernel blob as shown by vbutil_kernel. The thing is that it is
>> the chromeOS firmware (H2C bios)  who gets the command line out of the
>> blob and supplies it to the kernel, but VM does not use the chrome OS
>> firmware, so the command line and the kernel itself come from the
>> /boot directory of the SSD if I remember correctly. This doc
>> (http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format)
>> describes it at length, please have a look.
>>
>> cheers,
>> /vb
>
> Sorry, that I did not mentioned from beginning, that I'm using VM in order
> to run Chromium OS image.
> I read the document more carefully and as far as I understand I should
> modify configs in /boot/syslinux/
> But as far as I can see the command line in all files, that present in this
> folder, i.e. root.A.cfg, root.B.cfg and usb.A.cfg, does not fit the one,
> that is in /proc/cmdline on running image. Anyway I modified them also just
> to be sure, but that didn't work either. I really don't know where to dig.

I never tried running a VM and don't have a ChromeOS machine where I
can get a shell to investigate various partitions. If you are not done
by Tuesday morning PST - let me know, I'll check.

> 2011/8/22 Bill Richardson <wfri...@chromium.org>
>>
>> I just had a brief conversation with Vadim about this, so I suspect he's
>> preparing a detailed reply. But I can tell you quickly that the problem
>> you're having is that you're modifying the wrong kernel.
>> Short version: For each image there are three different kernels, each with
>> its own command line. The type of BIOS determines which one gets used, and
>> kvm uses legacy BIOS.
>> Long version is
>> here: https://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/disk-format
>
>  If I'm using wrong kernel how can I get access to the correct one? Since I
> used a bit of a brute force ;) and run vbutil_kernel --verify <partition> on
> all the partitions I got from running script "unpack_partitions.sh" and
> besides partition with the real kernel all I get is that there is no kernel
> blob.
> Although "vbutil_kernel --verify part_8" returned "ERROR: key_block_size
> advances past the end of the blob" but I don't know what does it mean, since
> I didn't find any source for the vbutil_kernel and googling it didn't
> provide any results.
> Cheers,
> Andrey Tarasevich

There is no point to run vbutil_kernel on anything but the contents of
partitions 2 and 4, these are the only places where verified boot
looks for the kernel.

All other kernel instances on the SSD device are for booting by legacy
BIOS, they are not signed and can't be verified. The "key_block_size
advances past the end of the blob"error message indicates that
vbutil_kernel is trying to interpret some junk contents as a valid
signed kernel, and fails.

cheers,
/vb

[Andrey Tarasevich]

I tried the following approach. 

According to the Disk Format design document http://www.chromium.org/chromium-os/chromiumos-design-docs/disk-format#TOC-x86-legacy-BIOS partition 12 is used also for legacy boot. So when I build image for the qemu-kvm virtual machine I execute following script "./image_to_vm --board=x86-generic --test_image". The workflow eventually executes script "update_bootloader.sh" where bootloader partition 12 is generated. After all work is done and before clean up I copy already created partition for my further manipulations. Now I have ready-to-use partition which I can mount and manipulate it how I want. Actually I just realized, that I could simply use unpack_partitions.sh script on qemu image and bootloader partition will be "part_12". D'oh! Anyway I change parameters of the kernel command line by editing /syslinux/usb.A.cfg since this config is used by default in syslinux.cfg as far as I understand. But just to be sure I modified root.A.cfg and root.B.sfg also. Then I copy partition back to image using command from automatically generate script "pack_partitions.sh", i.e. it looks something like this

dd if=part_12 of=chromiumos_qemu_image.bin bs=512 count=32768 seek=3940352 conv=notrun

And it does not have any effect. "cat /proc/cmdline" still shows me old command line. Well if I don't missed anything then I'm really confused. I don't have any clue how to solve this problem. I would appreciate any advice. 

by the way. This is cmd line I get from running image:

quiet console=tty2 init=/sbin/init boot=local rootwait ro noresume noswap loglevel=1 noinitrd mutant=1 root=/dev/sda3 i915.modeset=1 cros_legacy cros_debug BOOT_IMAGE=vmlinuz.A

and this is the cmd line from bootloader:

quiet console=tty2 init=/sbin/init boot=local rootwait ro noresume noswap loglevel=1 noinitrd mutant=6 root=/dev/sda3 i915.modeset=1 cros_legacy cros_debug

Cheers,

Andrey Tarasevich

2011/8/23 Vadim Bendebury <vbe...@chromium.org>