WARNING: bad unlock balance in rcu_lock_release

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: bed3c0d8 Merge tag 'for-5.2-rc5-tag' of git://git.kernel.o..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=148ef681a00000
kernel config: https://syzkaller.appspot.com/x/.config?x=28ec3437a5394ee0
dashboard link: https://syzkaller.appspot.com/bug?extid=f9545ab3e9f85cd43a3a
compiler: clang version 9.0.0 (/home/glider/llvm/clang
80fee25776c2fb61e74c1ecb1a523375c2500b69)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+f9545a...@syzkaller.appspotmail.com

=====================================
WARNING: bad unlock balance detected!
5.2.0-rc5+ #3 Not tainted
-------------------------------------
syz-executor.3/1203 is trying to release lock (rcu_callback) at:
[<ffffffff81636ec4>] rcu_lock_release+0x4/0x20 include/linux/rcupdate.h:212
but there are no more locks to release!

other info that might help us debug this:
1 lock held by syz-executor.3/1203:
#0: 00000000ae396ab9 (&type->s_umount_key#48/1){+.+.}, at:
alloc_super+0x15f/0x740 fs/super.c:228

stack backtrace:
CPU: 0 PID: 1203 Comm: syz-executor.3 Not tainted 5.2.0-rc5+ #3
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1d8/0x2f8 lib/dump_stack.c:113
print_unlock_imbalance_bug+0x26d/0x2b0 kernel/locking/lockdep.c:3846
__lock_release kernel/locking/lockdep.c:4062 [inline]
lock_release+0x435/0x790 kernel/locking/lockdep.c:4322
rcu_lock_release+0x1c/0x20 include/linux/rcupdate.h:214
__rcu_reclaim kernel/rcu/rcu.h:223 [inline]
rcu_do_batch kernel/rcu/tree.c:2092 [inline]
invoke_rcu_callbacks kernel/rcu/tree.c:2310 [inline]
rcu_core+0x8e2/0xf90 kernel/rcu/tree.c:2291
__do_softirq+0x340/0x7b0 arch/x86/include/asm/paravirt.h:777
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x21a/0x230 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0xf8/0x260 arch/x86/kernel/apic/apic.c:1068
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:806
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:767
[inline]
RIP: 0010:console_unlock+0xa65/0xf20 kernel/printk/printk.c:2471
Code: 20 00 74 0c 48 c7 c7 90 63 aa 88 e8 a5 44 50 00 48 83 3d fd cc 4b 07
00 0f 84 7d 04 00 00 e8 52 1b 17 00 48 8b 7c 24 28 57 9d <0f> 1f 44 00 00
f6 44 24 1f 01 75 52 e8 3a 1b 17 00 eb 63 0f 1f 84
RSP: 0018:ffff88805b6c7540 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: ffffffff815e969e RBX: 0000000000000200 RCX: 0000000000040000
RDX: ffffc9000c450000 RSI: 000000000002643c RDI: 0000000000000282
RBP: ffff88805b6c7670 R08: ffff88805cb14040 R09: fffffbfff115a505
R10: fffffbfff115a505 R11: 1ffffffff115a504 R12: dffffc0000000000
R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff88c31298
vprintk_emit+0x239/0x3a0 kernel/printk/printk.c:1986
vprintk_default+0x28/0x30 kernel/printk/printk.c:2013
vprintk_func+0x158/0x170 kernel/printk/printk_safe.c:386
printk+0xc4/0x11d kernel/printk/printk.c:2046
__ntfs_error+0x21b/0x230 fs/ntfs/debug.c:89
parse_options+0x481/0x1f80 fs/ntfs/super.c:234
ntfs_fill_super+0x19b/0x2940 fs/ntfs/super.c:2748
mount_bdev+0x31c/0x440 fs/super.c:1346
ntfs_mount+0x34/0x40 fs/ntfs/super.c:3051
legacy_get_tree+0xf9/0x1a0 fs/fs_context.c:661
vfs_get_tree+0x8f/0x360 fs/super.c:1476
do_new_mount fs/namespace.c:2791 [inline]
do_mount+0x1813/0x2730 fs/namespace.c:3111
ksys_mount+0xcc/0x100 fs/namespace.c:3320
__do_sys_mount fs/namespace.c:3334 [inline]
__se_sys_mount fs/namespace.c:3331 [inline]
__x64_sys_mount+0xbf/0xd0 fs/namespace.c:3331
do_syscall_64+0xfe/0x140 arch/x86/entry/common.c:301
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45bd1a
Code: b8 a6 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 9d 8d fb ff c3 66 2e 0f
1f 84 00 00 00 00 00 66 90 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff
ff 0f 83 7a 8d fb ff c3 66 0f 1f 84 00 00 00 00 00
RSP: 002b:00007f49071c9a88 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 00007f49071c9b40 RCX: 000000000045bd1a
RDX: 00007f49071c9ae0 RSI: 0000000020000140 RDI: 00007f49071c9b00
RBP: 0000000000000000 R08: 00007f49071c9b40 R09: 00007f49071c9ae0
R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000003
R13: 00000000004c82e2 R14: 00000000004deb40 R15: 00000000ffffffff
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_0
net_ratelimit: 21 callbacks suppressed
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1
protocol 88fb is buggy, dev hsr_slave_0
protocol 88fb is buggy, dev hsr_slave_1


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit: da940012 Merge tag 'char-misc-5.4-rc3' of git://git.kernel..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=12cfdf4f600000
kernel config: https://syzkaller.appspot.com/x/.config?x=2d2fd92a28d3e50

dashboard link: https://syzkaller.appspot.com/bug?extid=f9545ab3e9f85cd43a3a
compiler: clang version 9.0.0 (/home/glider/llvm/clang
80fee25776c2fb61e74c1ecb1a523375c2500b69)

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=148c9fc7600000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=100d3f8b600000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+f9545a...@syzkaller.appspotmail.com

=====================================
WARNING: bad unlock balance detected!

5.4.0-rc2+ #0 Not tainted
-------------------------------------
syz-executor111/7877 is trying to release lock (rcu_callback) at:
[<ffffffff81612bd4>] rcu_lock_release+0x4/0x20 include/linux/rcupdate.h:212

but there are no more locks to release!

other info that might help us debug this:

1 lock held by syz-executor111/7877:
#0: ffff8880a3c600d8 (&type->s_umount_key#42/1){+.+.}, at:
alloc_super+0x15f/0x790 fs/super.c:229

stack backtrace:
CPU: 1 PID: 7877 Comm: syz-executor111 Not tainted 5.4.0-rc2+ #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1d8/0x2f8 lib/dump_stack.c:113

print_unlock_imbalance_bug+0x20b/0x240 kernel/locking/lockdep.c:4008
__lock_release kernel/locking/lockdep.c:4244 [inline]
lock_release+0x473/0x780 kernel/locking/lockdep.c:4506

rcu_lock_release+0x1c/0x20 include/linux/rcupdate.h:214
__rcu_reclaim kernel/rcu/rcu.h:223 [inline]

rcu_do_batch kernel/rcu/tree.c:2157 [inline]
rcu_core+0x84f/0x1050 kernel/rcu/tree.c:2377
rcu_core_si+0x9/0x10 kernel/rcu/tree.c:2386
__do_softirq+0x333/0x7c4 arch/x86/include/asm/paravirt.h:766
invoke_softirq kernel/softirq.c:373 [inline]
irq_exit+0x227/0x230 kernel/softirq.c:413
exiting_irq arch/x86/include/asm/apic.h:536 [inline]
smp_apic_timer_interrupt+0x113/0x280 arch/x86/kernel/apic/apic.c:1137
apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
</IRQ>
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:756
[inline]
RIP: 0010:console_unlock+0xe35/0xef0 kernel/printk/printk.c:2477
Code: 20 00 74 0c 48 c7 c7 f0 91 8a 88 e8 65 da 4f 00 48 83 3d ad fd 2d 07
00 0f 84 b1 00 00 00 e8 d2 9c 16 00 48 8b 7c 24 10 57 9d <0f> 1f 44 00 00
eb 91 e8 bf 9c 16 00 eb 8a e8 b8 9c 16 00 eb 83 48
RSP: 0018:ffff888092daf930 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
RAX: ffffffff815c944e RBX: 0000000000000200 RCX: ffff888094422640
RDX: 0000000000000000 RSI: ffffffff815c8129 RDI: 0000000000000282
RBP: ffff888092daf9f0 R08: ffff888094422640 R09: fffffbfff111a4f1
R10: fffffbfff111a4f1 R11: 0000000000000000 R12: dffffc0000000000
R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff88a3d748
vprintk_emit+0x239/0x3a0 kernel/printk/printk.c:1996
vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
vprintk_func+0x158/0x170 kernel/printk/printk_safe.c:386
printk+0x62/0x8d kernel/printk/printk.c:2056
__ntfs_error+0x185/0x190 fs/ntfs/debug.c:89
read_ntfs_boot_sector fs/ntfs/super.c:675 [inline]
ntfs_fill_super+0x720/0x2a40 fs/ntfs/super.c:2784
mount_bdev+0x27c/0x390 fs/super.c:1415
ntfs_mount+0x34/0x40 fs/ntfs/super.c:3051
legacy_get_tree+0xf9/0x1a0 fs/fs_context.c:647
vfs_get_tree+0x8b/0x2a0 fs/super.c:1545
do_new_mount fs/namespace.c:2823 [inline]
do_mount+0x16c0/0x2510 fs/namespace.c:3143
ksys_mount+0xcc/0x100 fs/namespace.c:3352
__do_sys_mount fs/namespace.c:3366 [inline]
__se_sys_mount fs/namespace.c:3363 [inline]
__x64_sys_mount+0xbf/0xd0 fs/namespace.c:3363
do_syscall_64+0xf7/0x1c0 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x441e99
Code: e8 fc ad 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 3b 0a fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fff8d7174e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000441e99
RDX: 0000000020000140 RSI: 0000000020000280 RDI: 00000000200004c0
RBP: 000000000000f000 R08: 0000000000000000 R09: 00007fff8d717698
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006ccdc8
R13: 00000000006cd440 R14: 0000000000000000 R15: 0000000000000000

[Jan Kara]

On Sun 13-10-19 14:28:06, syzbot wrote:
> syzbot has found a reproducer for the following crash on:
>
> HEAD commit: da940012 Merge tag 'char-misc-5.4-rc3' of git://git.kernel..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=12cfdf4f600000
> kernel config: https://syzkaller.appspot.com/x/.config?x=2d2fd92a28d3e50
> dashboard link: https://syzkaller.appspot.com/bug?extid=f9545ab3e9f85cd43a3a
> compiler: clang version 9.0.0 (/home/glider/llvm/clang
> 80fee25776c2fb61e74c1ecb1a523375c2500b69)
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=148c9fc7600000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=100d3f8b600000
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+f9545a...@syzkaller.appspotmail.com
>
> =====================================
> WARNING: bad unlock balance detected!
> 5.4.0-rc2+ #0 Not tainted
> -------------------------------------
> syz-executor111/7877 is trying to release lock (rcu_callback) at:
> [<ffffffff81612bd4>] rcu_lock_release+0x4/0x20 include/linux/rcupdate.h:212
> but there are no more locks to release!

Hum, this is really weird. Look:

> other info that might help us debug this:
> 1 lock held by syz-executor111/7877:
> #0: ffff8880a3c600d8 (&type->s_umount_key#42/1){+.+.}, at:
> alloc_super+0x15f/0x790 fs/super.c:229
>
> stack backtrace:
> CPU: 1 PID: 7877 Comm: syz-executor111 Not tainted 5.4.0-rc2+ #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
> <IRQ>
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x1d8/0x2f8 lib/dump_stack.c:113
> print_unlock_imbalance_bug+0x20b/0x240 kernel/locking/lockdep.c:4008
> __lock_release kernel/locking/lockdep.c:4244 [inline]
> lock_release+0x473/0x780 kernel/locking/lockdep.c:4506
> rcu_lock_release+0x1c/0x20 include/linux/rcupdate.h:214
> __rcu_reclaim kernel/rcu/rcu.h:223 [inline]

__rcu_reclaim_kernel() has:

rcu_lock_acquire(&rcu_callback_map);
if (__is_kfree_rcu_offset(offset)) {
trace_rcu_invoke_kfree_callback(rn, head, offset);
kfree((void *)head - offset);
rcu_lock_release(&rcu_callback_map);
return true;
} else {
trace_rcu_invoke_callback(rn, head);
f = head->func;
WRITE_ONCE(head->func, (rcu_callback_t)0L);
f(head);
rcu_lock_release(&rcu_callback_map);
return false;
}

So RCU locking is clearly balanced there. The only possibility I can see
how this can happen is that RCU callback we have called actually released
rcu_callback_map but grepping the kernel doesn't show any other place where
that would get released? Confused.

But apparently there is even a reproducer for this so we could dig
further...

Honza

--
Jan Kara <ja...@suse.com>
SUSE Labs, CR

[Eric Biggers]

It's probably the same cause as "WARNING: bad unlock balance in rcu_core", see
the thread: https://lkml.kernel.org/linux-fsdevel/000000000000c0...@google.com/T/#u
Looks related to the lockdep_off() in ntfs_fill_super().

- Eric

[syzbot]

syzbot suspects this bug was fixed by commit:

commit 10476e6304222ced7df9b3d5fb0a043b3c2a1ad8
Author: Peter Zijlstra <pet...@infradead.org>
Date: Fri Mar 13 08:56:38 2020 +0000

locking/lockdep: Fix bad recursion pattern

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=16d64bac100000
start commit: 7d194c21 Linux 5.4-rc4
git tree: upstream
kernel config: https://syzkaller.appspot.com/x/.config?x=6c03e4d33fa96d51
dashboard link: https://syzkaller.appspot.com/bug?extid=f9545ab3e9f85cd43a3a
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=10a0a8c0e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13dd2dd8e00000

If the result looks correct, please mark the bug fixed by replying with:

#syz fix: locking/lockdep: Fix bad recursion pattern

For information about bisection process see: https://goo.gl/tpsmEJ#bisection