[syzbot] [bcachefs?] general protection fault in proc_sys_compare

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: b91872c56940 Merge tag 'dmaengine-fix-6.14' of git://git.k..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1485e8b7980000
kernel config: https://syzkaller.appspot.com/x/.config?x=8de9cc84d5960254
dashboard link: https://syzkaller.appspot.com/bug?extid=4364ec1693041cad20de
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=149d55a8580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/4b855669df70/disk-b91872c5.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/e44f3c546271/vmlinux-b91872c5.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b106e670346a/bzImage-b91872c5.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/68b26fa478ee/mount_0.gz

The issue was bisected to:

commit 579cd64b9df8a60284ec3422be919c362de40e41
Author: Hector Martin <mar...@marcan.st>
Date: Sat Feb 8 00:54:35 2025 +0000

ASoC: tas2770: Fix volume scale

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=14aa03a8580000
final oops: https://syzkaller.appspot.com/x/report.txt?x=16aa03a8580000
console output: https://syzkaller.appspot.com/x/log.txt?x=12aa03a8580000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+4364ec...@syzkaller.appspotmail.com
Fixes: 579cd64b9df8 ("ASoC: tas2770: Fix volume scale")

Oops: general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN PTI
KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007]
CPU: 0 UID: 0 PID: 5490 Comm: dhcpcd Not tainted 6.14.0-rc4-syzkaller-00295-gb91872c56940 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2025
RIP: 0010:sysctl_is_seen fs/proc/proc_sysctl.c:907 [inline]
RIP: 0010:proc_sys_compare+0x1ba/0x260 fs/proc/proc_sysctl.c:932
Code: 09 89 c5 31 ff 89 c6 e8 04 b2 5e ff 85 ed 74 56 80 3d d2 ae c2 0d 01 75 6f e8 b2 ad 5e ff e9 74 ff ff ff 48 89 e8 48 c1 e8 03 <42> 80 3c 28 00 74 08 48 89 ef e8 d7 61 c3 ff 48 8b 5d 00 48 85 db
RSP: 0018:ffffc90003547718 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff8c3b6218 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: 0000000000000004 RDI: ffffc900035476a0
RBP: 0000000000000000 R08: 0000000000000003 R09: fffff520006a8ed4
R10: dffffc0000000000 R11: fffff520006a8ed4 R12: 0000000000000004
R13: dffffc0000000000 R14: ffff88806c232c30 R15: ffff88806c040020
FS: 00007f2e2350e740(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056042d48d000 CR3: 000000001e6d2000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
d_same_name fs/dcache.c:2137 [inline]
__d_lookup+0x533/0x7b0 fs/dcache.c:2361
lookup_fast+0x79/0x590 fs/namei.c:1751
walk_component fs/namei.c:2110 [inline]
link_path_walk+0x672/0xea0 fs/namei.c:2479
path_openat+0x266/0x3590 fs/namei.c:3985
do_filp_open+0x27f/0x4e0 fs/namei.c:4016
do_sys_openat2+0x13e/0x1d0 fs/open.c:1428
do_sys_open fs/open.c:1443 [inline]
__do_sys_openat fs/open.c:1459 [inline]
__se_sys_openat fs/open.c:1454 [inline]
__x64_sys_openat+0x247/0x2a0 fs/open.c:1454
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f2e235d89a4
Code: 24 20 48 8d 44 24 30 48 89 44 24 28 64 8b 04 25 18 00 00 00 85 c0 75 2c 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 76 60 48 8b 15 55 a4 0d 00 f7 d8 64 89 02 48 83
RSP: 002b:00007ffcef5bc920 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00000000000100a0 RCX: 00007f2e235d89a4
RDX: 0000000000000000 RSI: 00007ffcef5ccbb8 RDI: 00000000ffffff9c
RBP: 00007ffcef5ccbb8 R08: 0000000000000000 R09: 00007ffcef5ccb28
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffcef5bca38 R14: 00007ffcef5bca38 R15: 0000000000000000
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:sysctl_is_seen fs/proc/proc_sysctl.c:907 [inline]
RIP: 0010:proc_sys_compare+0x1ba/0x260 fs/proc/proc_sysctl.c:932
Code: 09 89 c5 31 ff 89 c6 e8 04 b2 5e ff 85 ed 74 56 80 3d d2 ae c2 0d 01 75 6f e8 b2 ad 5e ff e9 74 ff ff ff 48 89 e8 48 c1 e8 03 <42> 80 3c 28 00 74 08 48 89 ef e8 d7 61 c3 ff 48 8b 5d 00 48 85 db
RSP: 0018:ffffc90003547718 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff8c3b6218 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: 0000000000000004 RDI: ffffc900035476a0
RBP: 0000000000000000 R08: 0000000000000003 R09: fffff520006a8ed4
R10: dffffc0000000000 R11: fffff520006a8ed4 R12: 0000000000000004
R13: dffffc0000000000 R14: ffff88806c232c30 R15: ffff88806c040020
FS: 00007f2e2350e740(0000) GS:ffff8880b8600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056042d48d000 CR3: 000000001e6d2000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
----------------
Code disassembly (best guess), 1 bytes skipped:
0: 89 c5 mov %eax,%ebp
2: 31 ff xor %edi,%edi
4: 89 c6 mov %eax,%esi
6: e8 04 b2 5e ff call 0xff5eb20f
b: 85 ed test %ebp,%ebp
d: 74 56 je 0x65
f: 80 3d d2 ae c2 0d 01 cmpb $0x1,0xdc2aed2(%rip) # 0xdc2aee8
16: 75 6f jne 0x87
18: e8 b2 ad 5e ff call 0xff5eadcf
1d: e9 74 ff ff ff jmp 0xffffff96
22: 48 89 e8 mov %rbp,%rax
25: 48 c1 e8 03 shr $0x3,%rax
* 29: 42 80 3c 28 00 cmpb $0x0,(%rax,%r13,1) <-- trapping instruction
2e: 74 08 je 0x38
30: 48 89 ef mov %rbp,%rdi
33: e8 d7 61 c3 ff call 0xffc3620f
38: 48 8b 5d 00 mov 0x0(%rbp),%rbx
3c: 48 85 db test %rbx,%rbx


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Hector Martin]

On 2025/03/07 11:45, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: b91872c56940 Merge tag 'dmaengine-fix-6.14' of git://git.k..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=1485e8b7980000
> kernel config: https://syzkaller.appspot.com/x/.config?x=8de9cc84d5960254
> dashboard link: https://syzkaller.appspot.com/bug?extid=4364ec1693041cad20de
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=149d55a8580000
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/4b855669df70/disk-b91872c5.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/e44f3c546271/vmlinux-b91872c5.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/b106e670346a/bzImage-b91872c5.xz
> mounted in repro: https://storage.googleapis.com/syzbot-assets/68b26fa478ee/mount_0.gz
>
> The issue was bisected to:
>
> commit 579cd64b9df8a60284ec3422be919c362de40e41
> Author: Hector Martin <mar...@marcan.st>
> Date: Sat Feb 8 00:54:35 2025 +0000
>
> ASoC: tas2770: Fix volume scale
>
> bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=14aa03a8580000
> final oops: https://syzkaller.appspot.com/x/report.txt?x=16aa03a8580000
> console output: https://syzkaller.appspot.com/x/log.txt?x=12aa03a8580000

[...]

This is a bad bisect. Not sure what the appropriate syzbot action is in
this case.

- Hector

[Kent Overstreet]

Better bisection algorithm? Standand bisect does really badly when fed
noisy data, but it wouldn't be hard to fix that: after N successive
passes or fails, which is unlikely because bisect tests are coinflips,
backtrack and gather more data in the part of the commit history where
you don't have much.

[Theodore Ts'o]

On Fri, Mar 07, 2025 at 06:51:23AM -0500, Kent Overstreet wrote:
>
> Better bisection algorithm? Standand bisect does really badly when fed
> noisy data, but it wouldn't be hard to fix that: after N successive
> passes or fails, which is unlikely because bisect tests are coinflips,
> backtrack and gather more data in the part of the commit history where
> you don't have much.

My general approach when handling some test failure is to try running
the reproducer 5-10 times on the original commit where the failure was
detected, to see if the reproducer is reliable. Once it's been
established whether the failure reproduces 100% of the time, or some
fraction of the time, say 25% of the time, then we can estalbish how
times we should try running the reproducer before we can conclude the
that a particular commit is "good" --- and the first time we detect a
failure, we can declare the commit is "bad", even if it happens on the
2nd out of the 25 tries that we might need to run a test if it is
particularly flaky.

Maybe this is something Syzbot could implement?

And if someone is familiar with the Go language, patches to implement
this in gce-xfstests's ltm server would be great! It's something I've
wanted to do, but I haven't gotten around to implementing it yet so it
can be fully automated. Right now, ltm's git branch watcher reruns
any failing test 5 times, so I get an idea of whether a failure is
flaky or not. I'll then manually run a potentially flaky test 30
times, and based on how reliable or flaky the test failure happens to
be, I then tell gce-xfstests to do a bisect running each test N times,
without having it stop once the test fails. It wasts a bit of test
resources, but since it doesn't block my personal time (results land
in my inbox when the bisect completes), it hasn't risen to the top of
my todo list.

Cheers,

- Ted

[Kent Overstreet]

On Fri, Mar 07, 2025 at 08:31:26AM -0500, Theodore Ts'o wrote:
> On Fri, Mar 07, 2025 at 06:51:23AM -0500, Kent Overstreet wrote:
> >
> > Better bisection algorithm? Standand bisect does really badly when fed
> > noisy data, but it wouldn't be hard to fix that: after N successive
> > passes or fails, which is unlikely because bisect tests are coinflips,
> > backtrack and gather more data in the part of the commit history where
> > you don't have much.
>
> My general approach when handling some test failure is to try running
> the reproducer 5-10 times on the original commit where the failure was
> detected, to see if the reproducer is reliable. Once it's been
> established whether the failure reproduces 100% of the time, or some
> fraction of the time, say 25% of the time, then we can estalbish how
> times we should try running the reproducer before we can conclude the
> that a particular commit is "good" --- and the first time we detect a
> failure, we can declare the commit is "bad", even if it happens on the
> 2nd out of the 25 tries that we might need to run a test if it is
> particularly flaky.

That does sound like a nice trick. I think we'd probably want both
approaches though, I've seen cases where a test starts out failing
perhasp 5% of the time and then jumps up to 40% later on - some other
behavioural change makes your race or what have you easier to hit.

Really what we're trying to do is determine the shape of an unknown
function sampling; we hope it's just a single stepwise change
but if not we need to keep gathering more data until we get a clear
enough picture (and we need a way to present that data, too).

>
> Maybe this is something Syzbot could implement?

Wouldn't it be better to have it in 'git bisect'?

> And if someone is familiar with the Go language, patches to implement
> this in gce-xfstests's ltm server would be great! It's something I've
> wanted to do, but I haven't gotten around to implementing it yet so it
> can be fully automated. Right now, ltm's git branch watcher reruns
> any failing test 5 times, so I get an idea of whether a failure is
> flaky or not. I'll then manually run a potentially flaky test 30
> times, and based on how reliable or flaky the test failure happens to
> be, I then tell gce-xfstests to do a bisect running each test N times,
> without having it stop once the test fails. It wasts a bit of test
> resources, but since it doesn't block my personal time (results land
> in my inbox when the bisect completes), it hasn't risen to the top of
> my todo list.

If only we had interns and grad students for this sort of thing :)

[Theodore Ts'o]

On Fri, Mar 07, 2025 at 09:33:11AM -0500, Kent Overstreet wrote:
>
> > Maybe this is something Syzbot could implement?
>
> Wouldn't it be better to have it in 'git bisect'?

"Git bisect" is the wrong layer of abstraction. It doesn't know
anything about (a) how build the software package (which might not be
the kernel, remember), nor how to run a test, nor how to tell whether
a test run was successful or a failure.

It's expected that those tools need to be built on top of "git
bisect." For example Steve Rostedt has contributed ktest.pl, which is
in the kernel sources in tools/teseting/ktest/. This assumes that the
system under test is a bare metal machine where is accessible via
ssh/scp, and that builds are done on the local machine where ktest.pl
is run.

ktest.pl is not something I've used myself, since I do pretty much all
of my testing using VM's, and in the case of gce-xfstests, we spin up
a fast compile VM which does the kernel compilation and uploads the
freshly compiled kernel to Google Cloud Storage (GCS), and then kicks
off one or more test VM's which fetches the kernel from GCS, with the
command of which tests to run encoded in the test VM metadata. I also
have a monitoring VM that detects if a test VM hangs, so it can
automatically restart the test VM. All of this is logic which is't
supported by ktest.pl, and obviously *far* beyond the scope of "git
bisect".

And while Syzbot can use Google Cloud in addition to qemu, its
infrastruture for how it compiles kernels and runs its test VM's is
sufficiently different that it's not clear that software written for
one infrastructure would be easily applicable to another.

> If only we had interns and grad students for this sort of thing :)

The lightweight test manager (ltm) for gce-xfstests was implemented by
an intern. And the kernel compilation service (kcs), git branch
watcher, and git bisection engine for gce-xfstests was done by a group
of undergraduates at a unversity in Boston as part of a software
engineering class project.

Mentoring interns and undergraduates was incredibly fulfilling, and I
very much enjoyed the experience. I'd like to think I helped them to
become better software engineers. However, mentoring students takes a
significant amount of time, and on net, it's not clear it was a win
from a personal time ROI perspective.

We did manage to recruit the intern to become a SWE at Google after he
graduated, so that was definitely considered a win from my company's
perspective. :-)

- Ted

[Kent Overstreet]

On Sat, Mar 08, 2025 at 04:53:05PM -0500, Theodore Ts'o wrote:
> On Fri, Mar 07, 2025 at 09:33:11AM -0500, Kent Overstreet wrote:
> >
> > > Maybe this is something Syzbot could implement?
> >
> > Wouldn't it be better to have it in 'git bisect'?
>
> "Git bisect" is the wrong layer of abstraction. It doesn't know
> anything about (a) how build the software package (which might not be
> the kernel, remember), nor how to run a test, nor how to tell whether
> a test run was successful or a failure.

Eh?

It has a mode for automatic bisections, you just give it a test that
runs pass/fail.

This works with my ktest, which runs tests in a VM and in non
interactive gives you that pass/fail in the exit code - I've used it
that way before.

> > If only we had interns and grad students for this sort of thing :)
>
> The lightweight test manager (ltm) for gce-xfstests was implemented by
> an intern. And the kernel compilation service (kcs), git branch
> watcher, and git bisection engine for gce-xfstests was done by a group
> of undergraduates at a unversity in Boston as part of a software
> engineering class project.
>
> Mentoring interns and undergraduates was incredibly fulfilling, and I
> very much enjoyed the experience. I'd like to think I helped them to
> become better software engineers. However, mentoring students takes a
> significant amount of time, and on net, it's not clear it was a win
> from a personal time ROI perspective.
>
> We did manage to recruit the intern to become a SWE at Google after he
> graduated, so that was definitely considered a win from my company's
> perspective. :-)

Cool :)

[syzbot]

Auto-closing this bug as obsolete.
No recent activity, existing reproducers are no longer triggering the issue.