Re: Issue 222959 in chromium: Chrome doesn't automatically reboot after a panic with corrupt rootfs

[chro...@googlecode.com]

Comment #1 on issue 222959 by tay...@chromium.org: Chrome doesn't
automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

eventlog.txt Look at event #101 and 102.

0 | 2013-03-21 08:01:20 | Log area cleared | 4096
1 | 2013-03-21 08:01:20 | Kernel Event | Clean Shutdown
2 | 2013-03-21 08:01:26 | System boot | 231
3 | 2013-03-21 08:01:26 | SUS Power Fail
4 | 2013-03-21 08:01:26 | System Reset
5 | 2013-03-21 08:01:26 | ACPI Wake | S5
6 | 2013-03-21 08:01:27 | Chrome OS Developer Mode
7 | 2013-03-21 09:04:17 | Kernel Event | Clean Shutdown
8 | 2013-03-21 09:04:17 | System boot | 232
9 | 2013-03-21 09:04:17 | System Reset
10 | 2013-03-21 09:04:18 | Chrome OS Developer Mode
11 | 2013-03-21 09:18:16 | Kernel Event | Clean Shutdown
12 | 2013-03-21 09:18:16 | ACPI Enter | S5
13 | 2013-03-21 09:18:22 | System boot | 233
14 | 2013-03-21 09:18:22 | SUS Power Fail
15 | 2013-03-21 09:18:22 | System Reset
16 | 2013-03-21 09:18:22 | ACPI Wake | S5
17 | 2013-03-21 09:18:23 | Chrome OS Developer Mode
18 | 2013-03-21 09:21:49 | System boot | 234
19 | 2013-03-21 09:21:49 | SUS Power Fail
20 | 2013-03-21 09:21:49 | Power Button Override
21 | 2013-03-21 09:21:49 | System Reset
22 | 2013-03-21 09:21:49 | ACPI Wake | S5
23 | 2013-03-21 09:21:49 | Chrome OS Developer Mode
24 | 2013-03-21 09:29:30 | Kernel Event | Clean Shutdown
25 | 2013-03-21 09:29:31 | System boot | 235
26 | 2013-03-21 09:29:31 | System Reset
27 | 2013-03-21 09:29:31 | Chrome OS Developer Mode
28 | 2013-03-21 09:30:39 | Kernel Event | Clean Shutdown
29 | 2013-03-21 09:30:39 | System boot | 236
30 | 2013-03-21 09:30:39 | System Reset
31 | 2013-03-21 09:30:40 | Chrome OS Developer Mode
32 | 2013-03-21 09:32:03 | Kernel Event | Clean Shutdown
33 | 2013-03-21 09:32:03 | System boot | 237
34 | 2013-03-21 09:32:03 | System Reset
35 | 2013-03-21 09:32:04 | Chrome OS Developer Mode
36 | 2013-03-21 09:32:10 | Kernel Event | Panic
37 | 2013-03-21 09:59:37 | System boot | 238
38 | 2013-03-21 09:59:37 | SUS Power Fail
39 | 2013-03-21 09:59:37 | Power Button Override
40 | 2013-03-21 09:59:37 | System Reset
41 | 2013-03-21 09:59:37 | ACPI Wake | S5
42 | 2013-03-21 09:59:37 | Chrome OS Developer Mode
43 | 2013-03-21 10:08:39 | System boot | 239
44 | 2013-03-21 10:08:39 | SUS Power Fail
45 | 2013-03-21 10:08:39 | Power Button Override
46 | 2013-03-21 10:08:39 | System Reset
47 | 2013-03-21 10:08:39 | ACPI Wake | S5
48 | 2013-03-21 10:08:39 | Chrome OS Developer Mode
49 | 2013-03-21 10:14:36 | Kernel Event | Clean Shutdown
50 | 2013-03-21 10:14:36 | System boot | 240
51 | 2013-03-21 10:14:36 | System Reset
52 | 2013-03-21 10:14:37 | Chrome OS Developer Mode
53 | 2013-03-21 10:14:43 | Kernel Event | Panic
54 | 2013-03-21 10:49:21 | System boot | 241
55 | 2013-03-21 10:49:21 | SUS Power Fail
56 | 2013-03-21 10:49:21 | Power Button Override
57 | 2013-03-21 10:49:21 | System Reset
58 | 2013-03-21 10:49:21 | ACPI Wake | S5
59 | 2013-03-21 10:49:22 | Chrome OS Recovery Mode | OS Kernel Failed
Signature Check
60 | 2013-03-21 10:49:41 | Power Button
61 | 2013-03-21 10:49:41 | ACPI Enter | S5
62 | 2013-03-21 10:49:47 | System boot | 242
63 | 2013-03-21 10:49:47 | SUS Power Fail
64 | 2013-03-21 10:49:47 | System Reset
65 | 2013-03-21 10:49:47 | ACPI Wake | S5
66 | 2013-03-21 10:49:48 | Chrome OS Developer Mode
67 | 2013-03-21 10:53:31 | System boot | 243
68 | 2013-03-21 10:53:31 | SUS Power Fail
69 | 2013-03-21 10:53:31 | Power Button Override
70 | 2013-03-21 10:53:31 | System Reset
71 | 2013-03-21 10:53:31 | ACPI Wake | S5
72 | 2013-03-21 10:53:32 | Chrome OS Developer Mode
73 | 2013-03-21 11:01:26 | Kernel Event | Clean Shutdown
74 | 2013-03-21 11:01:26 | System boot | 244
75 | 2013-03-21 11:01:26 | System Reset
76 | 2013-03-21 11:01:27 | Chrome OS Developer Mode
77 | 2013-03-21 11:01:33 | Kernel Event | Panic
78 | 2013-03-21 11:14:59 | System boot | 245
79 | 2013-03-21 11:14:59 | SUS Power Fail
80 | 2013-03-21 11:14:59 | Power Button Override
81 | 2013-03-21 11:14:59 | System Reset
82 | 2013-03-21 11:14:59 | ACPI Wake | S5
83 | 2013-03-21 11:15:00 | Chrome OS Developer Mode
84 | 2013-03-21 16:58:57 | Kernel Event | Clean Shutdown
85 | 2013-03-21 16:58:58 | ACPI Enter | S5
86 | 2013-03-21 16:59:07 | System boot | 246
87 | 2013-03-21 16:59:07 | SUS Power Fail
88 | 2013-03-21 16:59:07 | System Reset
89 | 2013-03-21 16:59:07 | ACPI Wake | S5
90 | 2013-03-21 16:59:07 | Chrome OS Developer Mode
91 | 2013-03-21 17:05:57 | System boot | 247
92 | 2013-03-21 17:05:57 | SUS Power Fail
93 | 2013-03-21 17:05:57 | Power Button Override
94 | 2013-03-21 17:05:57 | System Reset
95 | 2013-03-21 17:05:57 | ACPI Wake | S5
96 | 2013-03-21 17:05:57 | Chrome OS Developer Mode
97 | 2013-03-21 17:09:50 | Kernel Event | Clean Shutdown
98 | 2013-03-21 17:09:51 | System boot | 248
99 | 2013-03-21 17:09:51 | System Reset
100 | 2013-03-21 17:09:52 | Chrome OS Developer Mode
101 | 2013-03-21 17:09:58 | Kernel Event | Panic
102 | 2013-03-21 17:18:10 | System boot | 249
103 | 2013-03-21 17:18:10 | SUS Power Fail
104 | 2013-03-21 17:18:10 | Power Button Override
105 | 2013-03-21 17:18:10 | System Reset
106 | 2013-03-21 17:18:10 | ACPI Wake | S5
107 | 2013-03-21 17:18:11 | Chrome OS Developer Mode


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[chro...@googlecode.com]

Updates:
Status: Assigned
Owner: m...@chromium.org
Labels: -Pri-2 Pri-1 M-26

Comment #2 on issue 222959 by arsc...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Mandeep, can you help out here? It looks like the device is hanging in the
panic code.

[chro...@googlecode.com]

Comment #4 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Paul's scripts:

dd if=/dev/sda2 of=/dev/sda4 bs=64K # copy kernel A to B
dd if=/dev/sda3 of=/dev/sda5 bs=64K # copy rootfs A to B

cgpt add -i 2 -T 5 -P 9 -S 1 /dev/sda # set attributes on kernel A
cgpt add -i 4 -T 5 -P 9 -S 1 /dev/sda # set attributes on kernel B

dd if=/dev/zero of=/dev/sda2 count=1 # Corrupt kernel A
dd if=/dev/zero of=/dev/sda4 count=1 # Corrupt kernel B

# Corrupting the root partition in deveoper mode from TOT
# seems to just panic and do nothing else. Doesn't reboot.

dd if=/dev/zero of=/dev/sda3 count=16 # Corrupt rootfs A
dd if=/dev/zero of=/dev/sda5 count=16 # Corrupt rootfs B

dd if=/dev/zero of=/dev/sda3 seek=2129920 # Destroy bootcache A
dd if=/dev/zero of=/dev/sda5 seek=2129920 # Destroy bootcache B

dd if=/dev/zero of=/dev/sda3 seek=2129920 count=1 # Corrupt bootcache A
dd if=/dev/zero of=/dev/sda5 seek=2129920 count=1 # Corrupt bootcache B

dd if=/dev/sda4 of=/dev/sda2 bs=64K # copy kernel B to A
dd if=/dev/sda5 of=/dev/sda3 bs=64K # copy rootfs B to A

[chro...@googlecode.com]

Comment #5 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Trying to reproduce on daisy. The machine just goes into recovery mode.

[chro...@googlecode.com]

Comment #6 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

After also using the exact same cgpt settings as Paul, the system now
successfully falls back to KERN-B and ROOT-B.

[chro...@googlecode.com]

Comment #7 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Finally reproduced. cgpt flags get changed by bootloader so that was
confusing me.

[chro...@googlecode.com]

Comment #8 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Attached console-ramoops.

The problems is that we don't set /proc/sys/kernel/panic and panic_on_oops
until the init script which processes /etc/sysctl.conf is run. In this
case, Paul was corrupting the superblock so we were crashing before this
values were updated. The kernel default in not to reboot.

Fix should be pretty straight forward. We could either add "oops=panic
panic=-1" to the kernel commandline or we could hard-code these values into
the kernel.

Attachments:
console-ramoops 35.4 KB

[chro...@googlecode.com]

Updates:
Cc: tay...@chromium.org gaur...@chromium.org ol...@chromium.org

Comment #9 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

I can change the kernel command-line, add a new kconfig for setting the
default or add a tiny hack in the kernel and change the defaults for panic
and panic_on_oops.

My first thought is to change the kernel command-line but I've heard that
hard to do because of release scripts.

Anyone with a preference.

[chro...@googlecode.com]

Updates:
Owner: tay...@chromium.org
Cc: -tay...@chromium.org m...@chromium.org

Comment #10 on issue 222959 by m...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Verified that changing the command-line fixes this issue. Assigning to Paul
who knows how to change the command-line.

[chro...@googlecode.com]

Comment #11 on issue 222959 by tay...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

I have this working with my set of changes and verifies that it recovers.
Now, testing with signing.

[chro...@googlecode.com]

Comment #14 on issue 222959 by tay...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

I have things working on stumpy, not sure why restart didn't completely
work on parrot, but parrot had problem with r26 but those just required a
manual restart.

[chro...@googlecode.com]

Comment #15 on issue 222959 by bugdro...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs

http://code.google.com/p/chromium/issues/detail?id=222959#c15

Project: chromeos/cros-signing
Branch : master
Author : Paul Taysom <tay...@chromium.org>
Commit : fab25cc83c4472e9d9ed82032e0336cb13546d1b

Code Review +2: Mike Frysinger
Verified +1: Paul Taysom
Change-Id : I7370666ef89f9b6706adcc00343b1047f5515643
Reviewed-at : https://gerrit-int.chromium.org/34415

Added oops=panic and panic=-1 to optional kparams

These options have to be on the command line so the system is
in the right state so if the root volume is corrupt, the bios
will reboot. Without this change, the reboot has to be done manually.

BUG=chromium:222959
TEST=./security_test_image --board=$B
--image=/home/taysom/trunk/src/build/images/$B/latest/recovery_image.bin

Commit-Queue: Paul Taysom <tay...@google.com>

M security_test_baselines/ensure_secure_kernelparams.config

[chro...@googlecode.com]

Updates:
Labels: ReleaseBlock-Stable

Comment #16 on issue 222959 by benh...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

(No comment was entered for this change.)

[chro...@googlecode.com]

Updates:
Labels: Merge-Approved

Comment #17 on issue 222959 by benh...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Mike, mind merging these to R26 singers?

[chro...@googlecode.com]

Comment #18 on issue 222959 by bugdro...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs

http://code.google.com/p/chromium/issues/detail?id=222959#c18

Project: chromiumos/platform/crosutils

Branch : master
Author : Paul Taysom <tay...@chromium.org>

Commit : afbd21e4eed71007116986708f9c1f672e648b18

Code Review +2: Mike Frysinger
Verified +1: Paul Taysom

Change-Id : I59f64158d63efdc6f7b3d2efdbb9bb509cd68a4a
Reviewed-at : https://gerrit.chromium.org/gerrit/46442

Added "oops=panic panic=-1" to linux command line

These options are needed to be added to the command line to
that if the root volume is corrupt, the system will failover
to the alternate patitions automatically.

BUG=chromium:222959
TEST=manually corrupted the root volume, rebooted

Commit-Queue: Paul Taysom <tay...@chromium.org>

M build_kernel_image.sh

[chro...@googlecode.com]

Updates:
Labels: -M-26 -Merge-Approved M-27

Comment #20 on issue 222959 by benh...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Moving to 27. This isn't a regression, seems to fix something that affects
very few users and is relatively risky for this stage.

That said, we should get this in 27 as it's a nice thing to do for people
with Chromebooks, i.e. auto-reboot when corruption is detected.

[chro...@googlecode.com]

Updates:
Labels: Merge-Requested

Comment #21 on issue 222959 by tay...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

(No comment was entered for this change.)

[chro...@googlecode.com]

Updates:
Labels: -Merge-Requested Merge-Approved

Comment #22 on issue 222959 by josa...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Approved for M-27

[chro...@googlecode.com]

Updates:
Owner: m...@chromium.org
Cc: tay...@chromium.org

Comment #23 on issue 222959 by tay...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

Mandeep, Your doesn't appear to work on snow. Should I look at it or will
you?

[chro...@googlecode.com]

Updates:
Status: Fixed

Comment #28 on issue 222959 by tay...@chromium.org: Chrome doesn't

automatically reboot after a panic with corrupt rootfs
http://code.google.com/p/chromium/issues/detail?id=222959

(No comment was entered for this change.)