Re: [GIT PULL] first round of SCSI updates for the 5.13+ merge window

[Marco Elver]

On Fri, Jul 02, 2021 at 09:11AM +0100, James Bottomley wrote:
[...]
> scsi: core: Kill DRIVER_SENSE
[...]

As of this being merged, most of our syzbot instances are broken with:

| Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
| CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.13.0-syzkaller #0
| Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
| Call Trace:
| __dump_stack lib/dump_stack.c:79 [inline]
| dump_stack_lvl+0x6e/0x91 lib/dump_stack.c:96
| panic+0x192/0x4c7 kernel/panic.c:232
| mount_block_root+0x268/0x31a init/do_mounts.c:439
| mount_root+0x162/0x18d init/do_mounts.c:555
| prepare_namespace+0x1ff/0x234 init/do_mounts.c:607
| kernel_init_freeable+0x2c4/0x2d6 init/main.c:1604
| kernel_init+0x1a/0x1c0 init/main.c:1483
| ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

I've bisected the problem to 464a00c9e0ad ("scsi: core: Kill DRIVER_SENSE"):

| git bisect start
| # bad: [79160a603bdb51916226caf4a6616cc4e1c58a58] Merge tag 'usb-5.14-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
| git bisect bad 79160a603bdb51916226caf4a6616cc4e1c58a58
| # good: [62fb9874f5da54fdb243003b386128037319b219] Linux 5.13
| git bisect good 62fb9874f5da54fdb243003b386128037319b219
| # good: [c288d9cd710433e5991d58a0764c4d08a933b871] Merge tag 'for-5.14/io_uring-2021-06-30' of git://git.kernel.dk/linux-block
| git bisect good c288d9cd710433e5991d58a0764c4d08a933b871
| # bad: [bd31b9efbf549d9630bf2f269a3a56dcb29fcac1] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
| git bisect bad bd31b9efbf549d9630bf2f269a3a56dcb29fcac1
| # good: [b322a50d17ede5cff6622040f345228afecdcc45] Merge tag 'amd-drm-next-5.14-2021-06-22-1' of https://gitlab.freedesktop.org/agd5f/linux into drm-next
| git bisect good b322a50d17ede5cff6622040f345228afecdcc45
| # good: [406254918b232db198ed60f5bf1f8b84d96bca00] Merge tag 'perf-tools-for-v5.14-2021-07-01' of git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux
| git bisect good 406254918b232db198ed60f5bf1f8b84d96bca00
| # good: [e4aa67b023167c570be95a39149a918cca9e36a0] Merge tag 'jfs-5.14' of git://github.com/kleikamp/linux-shaggy
| git bisect good e4aa67b023167c570be95a39149a918cca9e36a0
| # bad: [d1f2ce77638d681c1e34d597e432a3b28ce15c78] scsi: qedi: Fix host removal with running sessions
| git bisect bad d1f2ce77638d681c1e34d597e432a3b28ce15c78
| # good: [44dc724f5eecf6ef8ea493e0f0198f297e1c7306] scsi: mpi3mr: Wait for pending I/O completions upon detection of VD I/O timeout
| git bisect good 44dc724f5eecf6ef8ea493e0f0198f297e1c7306
| # bad: [3d45cefc8edd7f560e6c97a8d9928ad571f76dec] scsi: core: Drop obsolete Linux-specific SCSI status codes
| git bisect bad 3d45cefc8edd7f560e6c97a8d9928ad571f76dec
| # bad: [9a588d239f2392fca24aa9e95d2a9174048db8e9] scsi: qlogicfas408: make ql_pcmd() a void function
| git bisect bad 9a588d239f2392fca24aa9e95d2a9174048db8e9
| # bad: [16576ad8a8f0c05f8cd86779e3f2c0ee9f5e5552] scsi: core: Do not use DRIVER_INVALID
| git bisect bad 16576ad8a8f0c05f8cd86779e3f2c0ee9f5e5552
| # good: [7e26e3ea028740f934477ec01ba586ab033c35aa] scsi: scsi_dh_alua: Check for negative result value
| git bisect good 7e26e3ea028740f934477ec01ba586ab033c35aa
| # good: [f2b1e9c6f867ec8f929e96ba4e4010e267587448] scsi: core: Introduce scsi_build_sense()
| git bisect good f2b1e9c6f867ec8f929e96ba4e4010e267587448
| # bad: [464a00c9e0ad45e3f42ff6ea705491a356df818e] scsi: core: Kill DRIVER_SENSE
| git bisect bad 464a00c9e0ad45e3f42ff6ea705491a356df818e
| # good: [d0672a03e0af5dd4b07dc9175b38e44290722192] scsi: core: Introduce scsi_status_is_check_condition()
| git bisect good d0672a03e0af5dd4b07dc9175b38e44290722192
| # first bad commit: [464a00c9e0ad45e3f42ff6ea705491a356df818e] scsi: core: Kill DRIVER_SENSE

The problem can be reproduced with the attached config and the following
qemu command (identical disk image not necessary):

qemu-system-x86_64 -kernel $KERNEL -append "console=ttyS0 root=/dev/sda earlycon earlyprintk=serial" \
-nographic -smp 2 -snapshot -enable-kvm -cpu host -m 1G \
-device virtio-scsi-pci,id=scsi \
-drive discard=unmap,file=vm.qcow2,if=none,id=hd0 \
-device scsi-hd,drive=hd0

Is the problem known?

Thanks,
-- Marco

[Martin K. Petersen]

Marco,

> On Fri, Jul 02, 2021 at 09:11AM +0100, James Bottomley wrote:
> [...]
>> scsi: core: Kill DRIVER_SENSE
> [...]
>
> As of this being merged, most of our syzbot instances are broken with:

I believe this should fix it:

https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=5.14/scsi-queue&id=c43ddbf97f46b93727718408d60a47ce8c08f30c

--
Martin K. Petersen Oracle Linux Engineering

[Marco Elver]

On Tue, Jul 06, 2021 at 10:27AM -0400, Martin K. Petersen wrote:
> Marco,
>
> > On Fri, Jul 02, 2021 at 09:11AM +0100, James Bottomley wrote:
> > [...]
> >> scsi: core: Kill DRIVER_SENSE
> > [...]
> >
> > As of this being merged, most of our syzbot instances are broken with:
>
> I believe this should fix it:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/mkp/scsi.git/commit/?h=5.14/scsi-queue&id=c43ddbf97f46b93727718408d60a47ce8c08f30c

Indeed it does! For "scsi: virtio_scsi: Do not overwrite SCSI status":

Tested-by: Marco Elver <el...@google.com>

Thanks,
-- Marco

[Dan Carpenter]

On Tue, Jul 06, 2021 at 03:59:51PM +0200, 'Marco Elver' via syzkaller-bugs wrote:
> On Fri, Jul 02, 2021 at 09:11AM +0100, James Bottomley wrote:
> [...]
> > scsi: core: Kill DRIVER_SENSE
> [...]
>
> As of this being merged, most of our syzbot instances are broken with:
>
> | Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
> | CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.13.0-syzkaller #0
> | Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> | Call Trace:
> | __dump_stack lib/dump_stack.c:79 [inline]
> | dump_stack_lvl+0x6e/0x91 lib/dump_stack.c:96
> | panic+0x192/0x4c7 kernel/panic.c:232
> | mount_block_root+0x268/0x31a init/do_mounts.c:439
> | mount_root+0x162/0x18d init/do_mounts.c:555
> | prepare_namespace+0x1ff/0x234 init/do_mounts.c:607
> | kernel_init_freeable+0x2c4/0x2d6 init/main.c:1604
> | kernel_init+0x1a/0x1c0 init/main.c:1483
> | ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
>
> I've bisected the problem to 464a00c9e0ad ("scsi: core: Kill DRIVER_SENSE"):

Here is one of syzbot reports.

https://groups.google.com/g/syzkaller-bugs/c/6aqmRNRYI7E/m/V7BNerRfDAAJ

If you look at the console output link, init_mount() is failing with
-ENXIO. It looks the sda drive is not found at all. It's supposed to
print a list of available partitions but the list is empty.

regards,
dan carpenter