[syzbot] INFO: rcu detected stall in sys_lsetxattr

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 40354149f4d7 Add linux-next specific files for 20220414
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=16ae0bd0f00000
kernel config: https://syzkaller.appspot.com/x/.config?x=a44d62051576f6f5
dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=164417ccf00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=104d63d0f00000

The issue was bisected to:

commit e257039f0fc7da36ac3a522ef9a5cb4ae7852e67
Author: Al Viro <vi...@zeniv.linux.org.uk>
Date: Tue Mar 1 04:04:20 2022 +0000

mount_setattr(): clean the control flow and calling conventions

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=14622210f00000
final oops: https://syzkaller.appspot.com/x/report.txt?x=16622210f00000
console output: https://syzkaller.appspot.com/x/log.txt?x=12622210f00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+306090...@syzkaller.appspotmail.com
Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions")

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 1-....: (10499 ticks this GP) idle=23b/1/0x4000000000000000 softirq=5447/5447 fqs=5210
(t=10500 jiffies g=4401 q=63 ncpus=2)
NMI backtrace for cpu 1
CPU: 1 PID: 3614 Comm: syz-executor153 Not tainted 5.18.0-rc2-next-20220414-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
nmi_cpu_backtrace.cold+0x47/0x144 lib/nmi_backtrace.c:111
nmi_trigger_cpumask_backtrace+0x1e6/0x230 lib/nmi_backtrace.c:62
trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
rcu_dump_cpu_stacks+0x262/0x3f0 kernel/rcu/tree_stall.h:369
print_cpu_stall kernel/rcu/tree_stall.h:665 [inline]
check_cpu_stall kernel/rcu/tree_stall.h:749 [inline]
rcu_pending kernel/rcu/tree.c:4010 [inline]
rcu_sched_clock_irq.cold+0x144/0x8fc kernel/rcu/tree.c:2675
update_process_times+0x16d/0x200 kernel/time/timer.c:1811
tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:243
tick_sched_timer+0xee/0x120 kernel/time/tick-sched.c:1473
__run_hrtimer kernel/time/hrtimer.c:1685 [inline]
__hrtimer_run_queues+0x1c0/0xe50 kernel/time/hrtimer.c:1749
hrtimer_interrupt+0x31c/0x790 kernel/time/hrtimer.c:1811
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline]
__sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103
sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:649
RIP: 0010:__mnt_want_write+0xdd/0x2e0 fs/namespace.c:348
Code: 00 02 00 00 89 de e8 22 64 9c ff 85 db 74 42 48 b8 00 00 00 00 00 fc ff df 4d 89 ec 49 c1 ec 03 49 01 c4 e8 e5 61 9c ff f3 90 <41> 0f b6 04 24 84 c0 74 08 3c 03 0f 8e 99 01 00 00 8b 5d 10 31 ff
RSP: 0018:ffffc90003acfdf0 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
RDX: ffff8880209957c0 RSI: ffffffff81dda00b RDI: 0000000000000003
RBP: ffff888140174c20 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffff81dda030 R11: 1ffffffff1f09899 R12: ffffed102802e986
R13: ffff888140174c30 R14: ffff888140174c50 R15: 0000000000000000
mnt_want_write+0x13d/0x3e0 fs/namespace.c:394
path_setxattr+0xb2/0x1c0 fs/xattr.c:627
__do_sys_lsetxattr fs/xattr.c:652 [inline]
__se_sys_lsetxattr fs/xattr.c:648 [inline]
__x64_sys_lsetxattr+0xbd/0x150 fs/xattr.c:648
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f7262608cc9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 11 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f72625992f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000bd
RAX: ffffffffffffffda RBX: 00007f72626904b0 RCX: 00007f7262608cc9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000600
RBP: 00007f726265e074 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0030656c69662f2e
R13: 695f70756f72672c R14: 695f726573752c30 R15: 00007f72626904b8
</TASK>
----------------
Code disassembly (best guess):
0: 00 02 add %al,(%rdx)
2: 00 00 add %al,(%rax)
4: 89 de mov %ebx,%esi
6: e8 22 64 9c ff callq 0xff9c642d
b: 85 db test %ebx,%ebx
d: 74 42 je 0x51
f: 48 b8 00 00 00 00 00 movabs $0xdffffc0000000000,%rax
16: fc ff df
19: 4d 89 ec mov %r13,%r12
1c: 49 c1 ec 03 shr $0x3,%r12
20: 49 01 c4 add %rax,%r12
23: e8 e5 61 9c ff callq 0xff9c620d
28: f3 90 pause
* 2a: 41 0f b6 04 24 movzbl (%r12),%eax <-- trapping instruction
2f: 84 c0 test %al,%al
31: 74 08 je 0x3b
33: 3c 03 cmp $0x3,%al
35: 0f 8e 99 01 00 00 jle 0x1d4
3b: 8b 5d 10 mov 0x10(%rbp),%ebx
3e: 31 ff xor %edi,%edi


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

[Christian Brauner]

On Tue, Apr 19, 2022 at 09:16:20AM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 40354149f4d7 Add linux-next specific files for 20220414
> git tree: linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=16ae0bd0f00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=a44d62051576f6f5
> dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=164417ccf00000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=104d63d0f00000
>
> The issue was bisected to:
>
> commit e257039f0fc7da36ac3a522ef9a5cb4ae7852e67
> Author: Al Viro <vi...@zeniv.linux.org.uk>
> Date: Tue Mar 1 04:04:20 2022 +0000
>
> mount_setattr(): clean the control flow and calling conventions
>
> bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=14622210f00000
> final oops: https://syzkaller.appspot.com/x/report.txt?x=16622210f00000
> console output: https://syzkaller.appspot.com/x/log.txt?x=12622210f00000
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+306090...@syzkaller.appspotmail.com
> Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions")

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git fs.mount_setattr.cleanup

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+306090...@syzkaller.appspotmail.com

Tested on:

commit: bbc1e8c5 fs: unset MNT_WRITE_HOLD on failure
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git fs.mount_setattr.cleanup
kernel config: https://syzkaller.appspot.com/x/.config?x=c01066a8395ef6d7

dashboard link: https://syzkaller.appspot.com/bug?extid=306090cfa3294f0bbfb3
compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.

[Christian Brauner]

After mnt_hold_writers() has been called we will always have set MNT_WRITE_HOLD
and consequently we always need to pair mnt_hold_writers() with
mnt_unhold_writers(). After the recent cleanup in [1] where Al switched from a
do-while to a for loop the cleanup currently fails to unset MNT_WRITE_HOLD for
the first mount that was changed. Fix this and make sure that the first mount
will be cleaned up and add some comments to make it more obvious.

Reported-by: syzbot+10a16d...@syzkaller.appspotmail.com
Reported-by: syzbot+306090...@syzkaller.appspotmail.com
Fixes: e257039f0fc7 ("mount_setattr(): clean the control flow and calling conventions") [1]
Link: https://lore.kernel.org/lkml/0000000000007c...@google.com
Link: https://lore.kernel.org/lkml/00000000000080...@google.com
Cc: Hillf Danton <hda...@sina.com>
Cc: Christoph Hellwig <h...@lst.de>
Cc: Al Viro <vi...@zeniv.linux.org.uk>
Signed-off-by: Christian Brauner (Microsoft) <bra...@kernel.org>
---
This should fix the syzbot issue. This is only relevant for making a
mount or mount tree read-only:
1. successul recursive read-only mount tree change:
Cleanup loop isn't executed.
2. failed recursive read-only mount tree change:
m will point to the mount we failed to handle. The cleanup loop will
run until p == m and then terminate.
3. successful single read-only mount change:
Cleanup loop won't be executed.
4. failed single read-only mount change:
m will point to mnt and the cleanup loop will terminate if p == m.
I don't think there's any other weird corner cases since we now that
MNT_WRITE_HOLD can only have been set by us as it requires
lock_mount_hash() which we hold. So unconditionally unsetting it is
fine. But please make sure to take a close look at the changed loop.
---
fs/namespace.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/fs/namespace.c b/fs/namespace.c
index a0a36bfa3aa0..afe2b64b14f1 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -4058,10 +4058,22 @@ static int mount_setattr_prepare(struct mount_kattr *kattr, struct mount *mnt)
if (err) {
struct mount *p;

- for (p = mnt; p != m; p = next_mnt(p, mnt)) {
+ /*
+ * If we had to call mnt_hold_writers() MNT_WRITE_HOLD will
+ * be set in @mnt_flags. The loop unsets MNT_WRITE_HOLD for all
+ * mounts and needs to take care to include the first mount.
+ */
+ for (p = mnt; p; p = next_mnt(p, mnt)) {
/* If we had to hold writers unblock them. */
if (p->mnt.mnt_flags & MNT_WRITE_HOLD)
mnt_unhold_writers(p);
+
+ /*
+ * We're done once the first mount we changed got
+ * MNT_WRITE_HOLD unset.
+ */
+ if (p == m)
+ break;
}
}
return err;

base-commit: b2d229d4ddb17db541098b83524d901257e93845
--
2.32.0

[Christian Brauner]

Unless I hear objections I'll route this upstream before -rc4 to get
this fixed because it's pretty trivial to trigger this.

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <h...@lst.de>