[syzbot] [ext4?] kernel panic: EXT4-fs (device loop0): panic forced after error (3)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://..
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000
kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf
dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/3b9bad020898/disk-ae545c32.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/4073566d0a4b/vmlinux-ae545c32.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b163e2a2c47c/bzImage-ae545c32.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/004395fabe81/mount_0.gz

Bisection is inconclusive: the issue happens on the oldest tested release.

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=12890d53a80000
final oops: https://syzkaller.appspot.com/x/report.txt?x=11890d53a80000
console output: https://syzkaller.appspot.com/x/log.txt?x=16890d53a80000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+27eece...@syzkaller.appspotmail.com

loop0: detected capacity change from 0 to 512
ext2: Unknown parameter '����'
EXT4-fs (loop0): feature flags set on rev 0 fs, running e2fsck is recommended
EXT4-fs (loop0): warning: checktime reached, running e2fsck is recommended
EXT4-fs warning (device loop0): ext4_update_dynamic_rev:1084: updating to rev 1 because of new feature flag, running e2fsck is recommended
EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap
Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error
CPU: 1 PID: 5061 Comm: syz-executor211 Not tainted 6.5.0-rc5-syzkaller-00353-gae545c3283dc #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
panic+0x30f/0x770 kernel/panic.c:340
ext4_handle_error+0x84e/0x8b0 fs/ext4/super.c:685
__ext4_error+0x277/0x3b0 fs/ext4/super.c:776
ext4_validate_block_bitmap+0xdf5/0x1200 fs/ext4/balloc.c:429
ext4_read_block_bitmap+0x40/0x80 fs/ext4/balloc.c:595
ext4_mb_clear_bb fs/ext4/mballoc.c:6503 [inline]
ext4_free_blocks+0xfd3/0x2e20 fs/ext4/mballoc.c:6748
ext4_remove_blocks fs/ext4/extents.c:2545 [inline]
ext4_ext_rm_leaf fs/ext4/extents.c:2710 [inline]
ext4_ext_remove_space+0x216e/0x4d90 fs/ext4/extents.c:2958
ext4_ext_truncate+0x164/0x1f0 fs/ext4/extents.c:4408
ext4_truncate+0xa0a/0x1150 fs/ext4/inode.c:4127
ext4_process_orphan+0x1aa/0x2d0 fs/ext4/orphan.c:339
ext4_orphan_cleanup+0xb71/0x1400 fs/ext4/orphan.c:474
__ext4_fill_super fs/ext4/super.c:5577 [inline]
ext4_fill_super+0x63ef/0x6ce0 fs/ext4/super.c:5696
get_tree_bdev+0x468/0x6c0 fs/super.c:1318
vfs_get_tree+0x8c/0x270 fs/super.c:1519
do_new_mount+0x28f/0xae0 fs/namespace.c:3335
do_mount fs/namespace.c:3675 [inline]
__do_sys_mount fs/namespace.c:3884 [inline]
__se_sys_mount+0x2d9/0x3c0 fs/namespace.c:3861
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f86c9eb1a99
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 17 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffc2eecbce8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 0030656c69662f2e RCX: 00007f86c9eb1a99
RDX: 00000000200001c0 RSI: 00000000200006c0 RDI: 0000000020000640
RBP: 000000000000ec37 R08: 0000000000000000 R09: 00005555562044c0
R10: 000000003f000000 R11: 0000000000000246 R12: 00007ffc2eecbd10
R13: 00007ffc2eecbcfc R14: 431bde82d7b634db R15: 00007f86c9efa03b
</TASK>
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Theodore Ts'o]

On Wed, Aug 16, 2023 at 03:48:49PM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://..
> git tree: upstream
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000
> kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf
> dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000
>

> EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap
> Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error

#syz invalid

This is fundamentally a syzbot bug. The file system is horrifically
corrupted, *and* the superblock has the "panic on error" (aka "panic
onfile system corruption") bit set.

This can be desireable because in a failover situation, if the file
system is found to be corrupted, you *want* the primary server to
fail, and let the secondary server to take over. This is a technique
which is decades old.

So this is Working As Intended, and is a classic example of (a) if you
are root, you can force the file system to crash, and (b) a classic
example of syzbot noise. (Five minutes of my life that I'm never
getting back. :-)

- Ted

[Aleksandr Nogikh]

On Thu, Aug 17, 2023 at 4:21 PM Theodore Ts'o <ty...@mit.edu> wrote:
>
> On Wed, Aug 16, 2023 at 03:48:49PM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://..
> > git tree: upstream
> > console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf
> > dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7
> > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000
> >
> > EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap
> > Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error
>
> #syz invalid
>
> This is fundamentally a syzbot bug. The file system is horrifically
> corrupted, *and* the superblock has the "panic on error" (aka "panic
> onfile system corruption") bit set.

The console log has the following line:

[ 60.708717][ T5061] Kernel panic - not syncing: EXT4-fs (device

loop0): panic forced after error

Can we consider a "panic forced after error" line to be a reliable
indicator that syzbot must ignore the report?

>
> This can be desireable because in a failover situation, if the file
> system is found to be corrupted, you *want* the primary server to
> fail, and let the secondary server to take over. This is a technique
> which is decades old.
>
> So this is Working As Intended, and is a classic example of (a) if you
> are root, you can force the file system to crash, and (b) a classic
> example of syzbot noise. (Five minutes of my life that I'm never
> getting back. :-)
>
> - Ted
>

> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20230817142103.GA2247938%40mit.edu.

[Theodore Ts'o]

On Thu, Aug 17, 2023 at 04:28:33PM +0200, Aleksandr Nogikh wrote:
> The console log has the following line:
>
> [ 60.708717][ T5061] Kernel panic - not syncing: EXT4-fs (device
> loop0): panic forced after error
>
> Can we consider a "panic forced after error" line to be a reliable
> indicator that syzbot must ignore the report?

Yes. And the file system image that generated this bug should be
discarded, because otherwise successive mutations will generate a
large number of crashes that syzbot will then need to ignore, thus
consuming syzbot resources.

Alternatively, you can do the moral equivalent of "tune2fs -e continue
foo.img" on any mutated file system seed, which will clear the "panic
on error".

(The other alternative is "tune2fs -e remount-ro", but given syzbot's
desire to find kernel crashes, "tune2fs -e continue" is more likely
find ways in which the kernel will find itself into trouble. Some
sysadmins will want to chose "remount-ro", however, since that is more
likely to limit file system damage once the file system is discovered
to be corrupted.)

- Ted

[Eric Sandeen]

On 8/17/23 9:21 AM, Theodore Ts'o wrote:
> On Wed, Aug 16, 2023 at 03:48:49PM -0700, syzbot wrote:
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: ae545c3283dc Merge tag 'gpio-fixes-for-v6.5-rc6' of git://..
>> git tree: upstream
>> console+strace: https://syzkaller.appspot.com/x/log.txt?x=13e5d553a80000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=171b698bc2e613cf
>> dashboard link: https://syzkaller.appspot.com/bug?extid=27eece6916b914a49ce7
>> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13433207a80000
>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=109cd837a80000
>>
>> EXT4-fs error (device loop0): ext4_validate_block_bitmap:430: comm syz-executor211: bg 0: block 46: invalid block bitmap
>> Kernel panic - not syncing: EXT4-fs (device loop0): panic forced after error
>
> #syz invalid
>
> This is fundamentally a syzbot bug. The file system is horrifically
> corrupted, *and* the superblock has the "panic on error" (aka "panic
> onfile system corruption") bit set.
>
> This can be desireable because in a failover situation, if the file
> system is found to be corrupted, you *want* the primary server to
> fail, and let the secondary server to take over. This is a technique
> which is decades old.

Just to play devil's advocate here - (sorry) - I don't see this as any
different from any other "malicious" filesystem image.

I've never been a fan of the idea that malicious images are real
security threats, but whether the parking lot USB stick paniced the box
in an unexpected way or "on purpose," the result is the same ...

I wonder if it might make sense to put EXT4_MOUNT_ERRORS_PANIC under a
sysctl or something, so that admins can enable it only when needed.

Sorry for stealing another 5 minutes of your life.

-Eric

[Theodore Ts'o]

On Thu, Aug 17, 2023 at 09:47:48AM -0500, Eric Sandeen wrote:
>
> Just to play devil's advocate here - (sorry) - I don't see this as any
> different from any other "malicious" filesystem image.
>
> I've never been a fan of the idea that malicious images are real security
> threats, but whether the parking lot USB stick paniced the box in an
> unexpected way or "on purpose," the result is the same ...
>
> I wonder if it might make sense to put EXT4_MOUNT_ERRORS_PANIC under a
> sysctl or something, so that admins can enable it only when needed.

Well, if someone is stupid enough to plug in a parking lot USB stick
into their system, they get everything they deserve. And a forced
panic isn't going to lead a more privilege escalation attack, so I
really don't see a problem if a file system which is marked "panic on
error", well, causes a panic. It's a good way of (harmlessly)
punishing stupid user tricks. :-)

The other way of thinking about it is that if your threat model
includes an attacker with physical access to the server with a USB
port, attacks include a cable which has a USB port on one side, and a
120V/240V AC mains plug on the the other. This will very likely cause
a system shutdown, even if they don't have automount enabled. :-)

- Ted

[Eric Biggers]

Eric S. is correct that for a filesystem image to enable panic on error, support
for panic on error should have to be properly consented to by the kernel
configuration, for example through an fs.allow_panic_on_error sysctl.

It can be argued that this not important, or not worth implementing when the
default will need to remain 1 for backwards compatibility. Or even that
syzkaller should work around it in the mean time. But it is incorrect to write

"This is fundamentally a syzbot bug."

- Eric

[Theodore Ts'o]

On Thu, Aug 17, 2023 at 09:47:39AM -0700, Eric Biggers wrote:
>
> Eric S. is correct that for a filesystem image to enable panic on error, support
> for panic on error should have to be properly consented to by the kernel
> configuration, for example through an fs.allow_panic_on_error sysctl.

I disagree. It's up to the system administrator, not the kernel ---
and the system adminsitrator is perfectly free to run e2fsck on a
random file system, or to use tune2fs to adjust the panic on error
setting on the file system, befure using their root powers to mount
the file system.

Root can do many things that cause the system to reboot. For example,
the system adminsirtator could run /sbin/reboot. Should the kernel
"consent" by setting fs.allow_reboot_system_call_to_work before the
root user can run the /sbin/reboot binary? Hopefully it's obvious why
this makes absolutely no sense.

> It can be argued that this not important, or not worth implementing when the
> default will need to remain 1 for backwards compatibility. Or even that
> syzkaller should work around it in the mean time. But it is incorrect to write
> "This is fundamentally a syzbot bug."

Well, the current behaviour is Working as Intended. And if syzbot is
going about whining about things that are Working as Intended, it's
not fit for the upostream developers' purpose.

As another example, root can set a real-time priority of a process to
be at a level where it will prempt all other processes, including
kernel threads. Do enough of these, and you *will* lock up the
kernel. Again, should there be a sysctl that allows real-time
priorities to work? Or do we teach syzbot that doing things that are
documented to cause the kernel to lock up are not something that's
worthy of a report. In the past, syzbot issued a *huge* amount of
noise caused by precisely to this. Upstream developers complained
that it was a false positive, and syzbot was adjusted to Stop Doing
That.

- Ted

[Eric Biggers]

Obviously it's up to the system administrator; that should have been clear since
I suggested a sysctl. Sorry if I wasn't clear. The point is that there are
certain conventions for what is allowed to break the safety guarantees that the
kernel provides to userspace, which includes causing a kernel panic. Panics on
various problems are configured by /proc/sys/kernel/panic_*. So having to
opt-in to panic-on-error, or at least being able to opt-out, by setting a sysctl
seems natural. Whereas having mount() being able to automatically panic the
kernel with no way to opt-out seems like a violation of broader kernel
conventions, even if it happens to be "working as intended" in the ext4 context.

Anyway, I'm not actually saying this issue is important. I just get frustrated
by the total denial that it could even possibly be considered something that
could be improved in the kernel...

- Eric

[Aleksandr Nogikh]

I've taken a closer look at the issue.

Documentation/filesystems/ext4.txt says that the "errors=" mount
parameter "override the errors behavior specified in the superblock".
So syzbot can prevent it by passing "errors=continue" as a mount
argument and there's no need to filter out such reports.

Syzkaller actually already does that in the C reproducer. It just
seems that this time the tool has mutated the mount options so much
that the simple patching no longer worked (most likely because of \0
characters in between). I'll update the syz_mount_image() code.

[Theodore Ts'o]

On Thu, Aug 17, 2023 at 07:52:55PM -0700, Eric Biggers wrote:
> Obviously it's up to the system administrator; that should have been clear since
> I suggested a sysctl. Sorry if I wasn't clear. The point is that there are
> certain conventions for what is allowed to break the safety guarantees that the
> kernel provides to userspace, which includes causing a kernel panic. Panics on
> various problems are configured by /proc/sys/kernel/panic_*. So having to
> opt-in to panic-on-error, or at least being able to opt-out, by setting a sysctl
> seems natural. Whereas having mount() being able to automatically panic the
> kernel with no way to opt-out seems like a violation of broader kernel
> conventions, even if it happens to be "working as intended" in the ext4 context.

The reason why a sysctl isn't really great is because the system
administrator might want to configure the behavior on a per-file
system basis. And you *can* configure it as a mount option, via
"mount -o errors=continue" or "mount -o "errors=panic". The
superblock setting is just the default if something isn't explicitly
specified as a mount option (either on the command line or in
/etc/fstab).

So mount does not "automatically" panic the kernel, and there are
*plenty* of ways to opt-out. You can use the mount option; you can
run "tune2fs -e continue"; you can just !@#!?! run fsck.ext4 before
mounting the file system. There are all ways of "opting out." Some
of them, such as the last, is even considered best practice --- just
as picking up a USB stick, or worse, a firewire drive, in a parking
lot, and *not* plugging it into your laptop is considered best practice.

- Ted

[Aleksandr Nogikh]

On Fri, Aug 18, 2023 at 1:43 PM Aleksandr Nogikh <nog...@google.com> wrote:
>
> I've taken a closer look at the issue.
>
> Documentation/filesystems/ext4.txt says that the "errors=" mount
> parameter "override the errors behavior specified in the superblock".
> So syzbot can prevent it by passing "errors=continue" as a mount
> argument and there's no need to filter out such reports.
>
> Syzkaller actually already does that in the C reproducer. It just
> seems that this time the tool has mutated the mount options so much
> that the simple patching no longer worked (most likely because of \0
> characters in between). I'll update the syz_mount_image() code.

Ah, it's a bit trickier -- the syz_mount_image() code is fine. The
reproducer first mounts an ext4 image via syz_mount_image(), which
appends "errors=continue" to the options and it doesn't lead to the
panic. But then the reproducer does a direct mount() call for the loop
device previously created in syz_mount_image(), this time _without_
mount options.

I've sent https://github.com/google/syzkaller/pull/4143 to sanitize
plain mount() calls.