[syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2)

4 views
Skip to first unread message

syzbot

unread,
May 19, 2021, 1:35:22 PM5/19/21
to Ja...@zx2c4.com, da...@davemloft.net, ku...@kernel.org, linux-...@vger.kernel.org, net...@vger.kernel.org, syzkall...@googlegroups.com, wire...@lists.zx2c4.com
Hello,

syzbot found the following issue on:

HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec
git tree: net
console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000
kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5
dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+a70a63...@syzkaller.appspotmail.com

BUG: MAX_LOCKDEP_KEYS too low!
turning off the locking correctness validator.
CPU: 0 PID: 5917 Comm: syz-executor.4 Not tainted 5.12.0-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:79 [inline]
dump_stack+0x141/0x1d7 lib/dump_stack.c:120
register_lock_class.cold+0x14/0x19 kernel/locking/lockdep.c:1281
__lock_acquire+0x102/0x5230 kernel/locking/lockdep.c:4781
lock_acquire kernel/locking/lockdep.c:5512 [inline]
lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5477
flush_workqueue+0x110/0x13e0 kernel/workqueue.c:2786
drain_workqueue+0x1a5/0x3c0 kernel/workqueue.c:2951
destroy_workqueue+0x71/0x800 kernel/workqueue.c:4382
alloc_workqueue+0xc40/0xef0 kernel/workqueue.c:4343
wg_newlink+0x43d/0x9e0 drivers/net/wireguard/device.c:335
__rtnl_newlink+0x1062/0x1710 net/core/rtnetlink.c:3452
rtnl_newlink+0x64/0xa0 net/core/rtnetlink.c:3500
rtnetlink_rcv_msg+0x44e/0xad0 net/core/rtnetlink.c:5562
netlink_rcv_skb+0x153/0x420 net/netlink/af_netlink.c:2502
netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline]
netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1338
netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1927
sock_sendmsg_nosec net/socket.c:654 [inline]
sock_sendmsg+0xcf/0x120 net/socket.c:674
____sys_sendmsg+0x6e8/0x810 net/socket.c:2350
___sys_sendmsg+0xf3/0x170 net/socket.c:2404
__sys_sendmsg+0xe5/0x1b0 net/socket.c:2433
do_syscall_64+0x3a/0xb0 arch/x86/entry/common.c:47
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x4665d9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fb25febe188 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 000000000056c0b0 RCX: 00000000004665d9
RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000005
RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056c0b0
R13: 00007fff30a5021f R14: 00007fb25febe300 R15: 0000000000022000


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

Dmitry Vyukov

unread,
May 19, 2021, 3:49:07 PM5/19/21
to syzbot, Peter Zijlstra, Tetsuo Handa, Jason A. Donenfeld, David Miller, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list
On Wed, May 19, 2021 at 7:35 PM syzbot
<syzbot+a70a63...@syzkaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec
> git tree: net
> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5
> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+a70a63...@syzkaller.appspotmail.com
>
> BUG: MAX_LOCKDEP_KEYS too low!


What config controls this? I don't see "MAX_LOCKDEP_KEYS too low" in
any of the config descriptions...
Here is what syzbot used:

CONFIG_LOCKDEP=y
CONFIG_LOCKDEP_BITS=16
CONFIG_LOCKDEP_CHAINS_BITS=17
CONFIG_LOCKDEP_STACK_TRACE_BITS=20
CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12

We already bumped most of these.
The log contains dump of the lockdep debug files, is there any offender?

Also looking at the log I noticed a memory safety bug in lockdep implementation:

[ 2023.605505][ T6807]
==================================================================
[ 2023.613589][ T6807] BUG: KASAN: global-out-of-bounds in
print_name+0x1b0/0x1d0
[ 2023.624553][ T6807] Read of size 8 at addr ffffffff90225cb0 by task cat/6807
[ 2023.631765][ T6807]
[ 2023.634096][ T6807] CPU: 1 PID: 6807 Comm: cat Not tainted
5.12.0-syzkaller #0
[ 2023.641488][ T6807] Hardware name: Google Google Compute
Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 2023.651745][ T6807] Call Trace:
[ 2023.655031][ T6807] dump_stack+0x141/0x1d7
[ 2023.659375][ T6807] ? print_name+0x1b0/0x1d0
[ 2023.663890][ T6807] print_address_description.constprop.0.cold+0x5/0x2f8
[ 2023.670895][ T6807] ? print_name+0x1b0/0x1d0
[ 2023.675413][ T6807] ? print_name+0x1b0/0x1d0
[ 2023.679948][ T6807] kasan_report.cold+0x7c/0xd8
[ 2023.684725][ T6807] ? print_name+0x1b0/0x1d0
[ 2023.689248][ T6807] print_name+0x1b0/0x1d0
[ 2023.694196][ T6807] ? lockdep_stats_show+0xa20/0xa20
[ 2023.699940][ T6807] ? seq_file_path+0x30/0x30
[ 2023.704721][ T6807] ? mutex_lock_io_nested+0xf70/0xf70
[ 2023.710118][ T6807] ? lock_acquire+0x58a/0x740
[ 2023.715156][ T6807] ? kasan_unpoison+0x3c/0x60
[ 2023.719843][ T6807] lc_show+0x10a/0x210
[ 2023.723924][ T6807] seq_read_iter+0xb66/0x1220
[ 2023.728617][ T6807] proc_reg_read_iter+0x1fb/0x2d0
[ 2023.733651][ T6807] new_sync_read+0x41e/0x6e0
[ 2023.738272][ T6807] ? ksys_lseek+0x1b0/0x1b0
[ 2023.742784][ T6807] ? lock_acquire+0x58a/0x740
[ 2023.747563][ T6807] vfs_read+0x35c/0x570
[ 2023.751737][ T6807] ksys_read+0x12d/0x250
[ 2023.756003][ T6807] ? vfs_write+0xa30/0xa30
[ 2023.760429][ T6807] ? syscall_enter_from_user_mode+0x27/0x70
[ 2023.766335][ T6807] do_syscall_64+0x3a/0xb0
[ 2023.770764][ T6807] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 2023.776685][ T6807] RIP: 0033:0x7f99856e2910
[ 2023.781104][ T6807] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec
08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00
00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00
48 89 04 24
[ 2023.800719][ T6807] RSP: 002b:00007ffee7328628 EFLAGS: 00000246
ORIG_RAX: 0000000000000000
[ 2023.809169][ T6807] RAX: ffffffffffffffda RBX: 0000000000020000
RCX: 00007f99856e2910
[ 2023.817150][ T6807] RDX: 0000000000020000 RSI: 0000564290b2a000
RDI: 0000000000000003
[ 2023.825123][ T6807] RBP: 0000564290b2a000 R08: 0000000000000003
R09: 0000000000021010
[ 2023.833107][ T6807] R10: 0000000000000002 R11: 0000000000000246
R12: 0000564290b2a000
[ 2023.841091][ T6807] R13: 0000000000000003 R14: 0000000000020000
R15: 0000000000001000
[ 2023.849074][ T6807]
[ 2023.851408][ T6807] The buggy address belongs to the variable:
[ 2023.857388][ T6807] lock_classes_in_use+0x410/0x420
[ 2023.862510][ T6807]
[ 2023.864826][ T6807] Memory state around the buggy address:
[ 2023.870450][ T6807] ffffffff90225b80: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00
[ 2023.878511][ T6807] ffffffff90225c00: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00
[ 2023.886572][ T6807] >ffffffff90225c80: 00 00 00 00 f9 f9 f9 f9 00
00 00 00 00 00 00 00
[ 2023.894628][ T6807] ^
[ 2023.900256][ T6807] ffffffff90225d00: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00
[ 2023.908317][ T6807] ffffffff90225d80: 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00
[ 2023.916377][ T6807]
==================================================================
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/0000000000003687bd05c2b2401d%40google.com.

Randy Dunlap

unread,
May 19, 2021, 3:58:05 PM5/19/21
to Dmitry Vyukov, syzbot, Peter Zijlstra, Tetsuo Handa, Jason A. Donenfeld, David Miller, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list
On 5/19/21 12:48 PM, Dmitry Vyukov wrote:
> On Wed, May 19, 2021 at 7:35 PM syzbot
> <syzbot+a70a63...@syzkaller.appspotmail.com> wrote:
>>
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec
>> git tree: net
>> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5
>> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f
>>
>> Unfortunately, I don't have any reproducer for this issue yet.
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+a70a63...@syzkaller.appspotmail.com
>>
>> BUG: MAX_LOCKDEP_KEYS too low!
>

include/linux/lockdep.h

#define MAX_LOCKDEP_KEYS_BITS 13
#define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS)

Documentation/locking/lockdep-design.rst:

Troubleshooting:
----------------

The validator tracks a maximum of MAX_LOCKDEP_KEYS number of lock classes.
Exceeding this number will trigger the following lockdep warning::

(DEBUG_LOCKS_WARN_ON(id >= MAX_LOCKDEP_KEYS))

By default, MAX_LOCKDEP_KEYS is currently set to 8191, and typical
desktop systems have less than 1,000 lock classes, so this warning
normally results from lock-class leakage or failure to properly
initialize locks. These two problems are illustrated below:

>
> What config controls this? I don't see "MAX_LOCKDEP_KEYS too low" in
> any of the config descriptions...
> Here is what syzbot used:
>
> CONFIG_LOCKDEP=y
> CONFIG_LOCKDEP_BITS=16
> CONFIG_LOCKDEP_CHAINS_BITS=17
> CONFIG_LOCKDEP_STACK_TRACE_BITS=20
> CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
> CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
>
> We already bumped most of these.
> The log contains dump of the lockdep debug files, is there any offender?
>
> Also looking at the log I noticed a memory safety bug in lockdep implementation:

...

--
~Randy

Dmitry Vyukov

unread,
May 19, 2021, 4:09:47 PM5/19/21
to Randy Dunlap, syzbot, Peter Zijlstra, Tetsuo Handa, Jason A. Donenfeld, David Miller, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list
On Wed, May 19, 2021 at 9:58 PM Randy Dunlap <rdu...@infradead.org> wrote:
>
> On 5/19/21 12:48 PM, Dmitry Vyukov wrote:
> > On Wed, May 19, 2021 at 7:35 PM syzbot
> > <syzbot+a70a63...@syzkaller.appspotmail.com> wrote:
> >>
> >> Hello,
> >>
> >> syzbot found the following issue on:
> >>
> >> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec
> >> git tree: net
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f
> >>
> >> Unfortunately, I don't have any reproducer for this issue yet.
> >>
> >> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >> Reported-by: syzbot+a70a63...@syzkaller.appspotmail.com
> >>
> >> BUG: MAX_LOCKDEP_KEYS too low!
> >
>
> include/linux/lockdep.h
>
> #define MAX_LOCKDEP_KEYS_BITS 13
> #define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS)

Ouch, so it's not configurable yet :(
Unless, of course, we identify the offender that produced thousands of
lock classes in the log and fix it.
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/c545268c-fe62-883c-4c46-974b3bb3cea1%40infradead.org.

Tetsuo Handa

unread,
May 20, 2021, 1:02:16 AM5/20/21
to Dmitry Vyukov, Randy Dunlap, David Miller, syzbot, Peter Zijlstra, Jason A. Donenfeld, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list
On 2021/05/20 5:09, Dmitry Vyukov wrote:
> On Wed, May 19, 2021 at 9:58 PM Randy Dunlap <rdu...@infradead.org> wrote:
>>
>> On 5/19/21 12:48 PM, Dmitry Vyukov wrote:
>>> On Wed, May 19, 2021 at 7:35 PM syzbot
>>> <syzbot+a70a63...@syzkaller.appspotmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> syzbot found the following issue on:
>>>>
>>>> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec
>>>> git tree: net
>>>> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000
>>>> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5
>>>> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f
>>>>
>>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>>
>>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>>> Reported-by: syzbot+a70a63...@syzkaller.appspotmail.com
>>>>
>>>> BUG: MAX_LOCKDEP_KEYS too low!
>>>
>>
>> include/linux/lockdep.h
>>
>> #define MAX_LOCKDEP_KEYS_BITS 13
>> #define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS)
>
> Ouch, so it's not configurable yet :(

I didn't try to make this value configurable, for

> Unless, of course, we identify the offender that produced thousands of
> lock classes in the log and fix it.

number of currently active locks should decrease over time.
If this message is printed, increasing this value unlikely helps.

We have https://lkml.kernel.org/r/c099ad52-0c2c-b886...@ozlabs.ru
which seems to be unresolved.

Regarding this report, cleanup of bonding device is too slow to catch up to
creation of bonding device?

We might need to throttle creation of BPF, bonding etc. which involve WQ operation for clean up ?

Dmitry Vyukov

unread,
May 21, 2021, 3:04:52 AM5/21/21
to Tetsuo Handa, Randy Dunlap, David Miller, syzbot, Peter Zijlstra, Jason A. Donenfeld, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list
I see, thanks for digging into it.

Unbounded asynchronous queueing is always a recipe for disaster... I
assume such issues can affect production as well, if some program
creates namespaces/devices in a loop. So I think ideally such things
are throttled/restricted in the kernel, e.g. new namespaces/devices
are not created if some threshold is reached.

Potentially syzkaller could throttle creation of new
namespaces/devices if we find a good and reliable way to monitor
backlog. Something like the length of a particular workqueue. It may
also help with OOMs. But so far I haven't found it.
Reply all
Reply to author
Forward
0 new messages