[v6.1] possible deadlock in __dev_queue_xmit (2)

3 views
Skip to first unread message

syzbot

unread,
Jan 19, 2024, 11:24:23 AMJan 19
to syzkaller...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: fec3b1451d5f Linux 6.1.73
git tree: linux-6.1.y
console output: https://syzkaller.appspot.com/x/log.txt?x=12b133c7e80000
kernel config: https://syzkaller.appspot.com/x/.config?x=4c3a37fc80b0c6e3
dashboard link: https://syzkaller.appspot.com/bug?extid=c08be6149a3f2c25a94f
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/5c9deae28f89/disk-fec3b145.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/5972bfdbff94/vmlinux-fec3b145.xz
kernel image: https://storage.googleapis.com/syzbot-assets/c4cd0621bb07/Image-fec3b145.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c08be6...@syzkaller.appspotmail.com

======================================================
WARNING: possible circular locking dependency detected
6.1.73-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor.5/7008 is trying to acquire lock:
ffff0000c3825218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: spin_lock include/linux/spinlock.h:350 [inline]
ffff0000c3825218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3884 [inline]
ffff0000c3825218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_queue_xmit+0x1e3c/0x38d8 net/core/dev.c:4259

but task is already holding lock:
ffff0000d296d4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:350 [inline]
ffff0000d296d4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4304 [inline]
ffff0000d296d4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x164/0x548 net/sched/sch_generic.c:340

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (_xmit_ETHER#2){+.-.}-{2:2}:
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x54/0x6c kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:350 [inline]
__netif_tx_lock include/linux/netdevice.h:4304 [inline]
sch_direct_xmit+0x164/0x548 net/sched/sch_generic.c:340
__dev_xmit_skb net/core/dev.c:3854 [inline]
__dev_queue_xmit+0x1658/0x38d8 net/core/dev.c:4259
dev_queue_xmit include/linux/netdevice.h:3021 [inline]
lapbeth_data_transmit+0x1e0/0x298 drivers/net/wan/lapbether.c:259
lapb_data_transmit+0x8c/0xb0 net/lapb/lapb_iface.c:447
lapb_transmit_buffer+0x178/0x204 net/lapb/lapb_out.c:149
lapb_send_control+0x220/0x320 net/lapb/lapb_subr.c:251
lapb_establish_data_link+0x94/0xec
lapb_device_event+0x348/0x4e0
notifier_call_chain kernel/notifier.c:87 [inline]
raw_notifier_call_chain+0xd4/0x164 kernel/notifier.c:455
__dev_notify_flags+0x2ac/0x534
dev_change_flags+0xc8/0x154 net/core/dev.c:8658
devinet_ioctl+0x858/0x17e4 net/ipv4/devinet.c:1146
inet_ioctl+0x2ac/0x4d8 net/ipv4/af_inet.c:1003
sock_do_ioctl+0x134/0x2dc net/socket.c:1204
sock_ioctl+0x4ec/0x858 net/socket.c:1321
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:870 [inline]
__se_sys_ioctl fs/ioctl.c:856 [inline]
__arm64_sys_ioctl+0x14c/0x1c8 fs/ioctl.c:856
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585

-> #0 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}:
check_prev_add kernel/locking/lockdep.c:3090 [inline]
check_prevs_add kernel/locking/lockdep.c:3209 [inline]
validate_chain kernel/locking/lockdep.c:3825 [inline]
__lock_acquire+0x3338/0x7680 kernel/locking/lockdep.c:5049
lock_acquire+0x26c/0x7cc kernel/locking/lockdep.c:5662
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x54/0x6c kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:350 [inline]
__dev_xmit_skb net/core/dev.c:3884 [inline]
__dev_queue_xmit+0x1e3c/0x38d8 net/core/dev.c:4259
dev_queue_xmit include/linux/netdevice.h:3021 [inline]
neigh_resolve_output+0x518/0x618 net/core/neighbour.c:1556
neigh_output include/net/neighbour.h:544 [inline]
ip6_finish_output2+0xdac/0x1a88 net/ipv6/ip6_output.c:134
__ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
ip6_finish_output+0x5a4/0x940 net/ipv6/ip6_output.c:212
NF_HOOK_COND include/linux/netfilter.h:291 [inline]
ip6_output+0x270/0x594 net/ipv6/ip6_output.c:233
dst_output include/net/dst.h:444 [inline]
NF_HOOK include/linux/netfilter.h:302 [inline]
ndisc_send_skb+0xc38/0x179c net/ipv6/ndisc.c:509
ndisc_solicit+0x3ac/0x590
neigh_probe+0xc4/0x128 net/core/neighbour.c:1068
__neigh_event_send+0xe50/0x1668 net/core/neighbour.c:1235
neigh_event_send_probe include/net/neighbour.h:468 [inline]
neigh_event_send include/net/neighbour.h:474 [inline]
neigh_resolve_output+0x17c/0x618 net/core/neighbour.c:1540
neigh_output include/net/neighbour.h:544 [inline]
ip6_finish_output2+0xdac/0x1a88 net/ipv6/ip6_output.c:134
__ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
ip6_finish_output+0x5a4/0x940 net/ipv6/ip6_output.c:212
NF_HOOK_COND include/linux/netfilter.h:291 [inline]
ip6_output+0x270/0x594 net/ipv6/ip6_output.c:233
dst_output include/net/dst.h:444 [inline]
ip6_local_out+0x120/0x160 net/ipv6/output_core.c:161
ip6_send_skb+0x118/0x428 net/ipv6/ip6_output.c:1995
ip6_push_pending_frames+0xd0/0x118 net/ipv6/ip6_output.c:2015
icmpv6_push_pending_frames+0x288/0x3f4 net/ipv6/icmp.c:303
icmp6_send+0x11a8/0x1b2c net/ipv6/icmp.c:622
__icmpv6_send include/linux/icmpv6.h:28 [inline]
icmpv6_send include/linux/icmpv6.h:49 [inline]
ip6_link_failure+0x44/0x4a8 net/ipv6/route.c:2789
dst_link_failure+0x11c/0x160 include/net/dst.h:423
ip_tunnel_xmit+0x1e1c/0x28f4 net/ipv4/ip_tunnel.c:819
__gre_xmit net/ipv4/ip_gre.c:469 [inline]
erspan_xmit+0x9c8/0x1498 net/ipv4/ip_gre.c:718
__netdev_start_xmit include/linux/netdevice.h:4853 [inline]
netdev_start_xmit include/linux/netdevice.h:4867 [inline]
xmit_one net/core/dev.c:3627 [inline]
dev_hard_start_xmit+0x25c/0x9a4 net/core/dev.c:3643
sch_direct_xmit+0x234/0x548 net/sched/sch_generic.c:342
__dev_xmit_skb net/core/dev.c:3900 [inline]
__dev_queue_xmit+0x1758/0x38d8 net/core/dev.c:4259
dev_queue_xmit include/linux/netdevice.h:3021 [inline]
neigh_resolve_output+0x518/0x618 net/core/neighbour.c:1556
neigh_output include/net/neighbour.h:544 [inline]
ip6_finish_output2+0xdac/0x1a88 net/ipv6/ip6_output.c:134
__ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
ip6_finish_output+0x5a4/0x940 net/ipv6/ip6_output.c:212
NF_HOOK_COND include/linux/netfilter.h:291 [inline]
ip6_output+0x270/0x594 net/ipv6/ip6_output.c:233
dst_output include/net/dst.h:444 [inline]
NF_HOOK include/linux/netfilter.h:302 [inline]
rawv6_send_hdrinc+0xd74/0x1bfc net/ipv6/raw.c:654
rawv6_sendmsg+0x1114/0x1bc0 net/ipv6/raw.c:912
inet_sendmsg+0x15c/0x290 net/ipv4/af_inet.c:836
sock_sendmsg_nosec net/socket.c:718 [inline]
__sock_sendmsg net/socket.c:730 [inline]
____sys_sendmsg+0x55c/0x848 net/socket.c:2514
___sys_sendmsg net/socket.c:2568 [inline]
__sys_sendmmsg+0x318/0x7d8 net/socket.c:2654
__do_sys_sendmmsg net/socket.c:2683 [inline]
__se_sys_sendmmsg net/socket.c:2680 [inline]
__arm64_sys_sendmmsg+0xa0/0xbc net/socket.c:2680
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585

other info that might help us debug this:

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(_xmit_ETHER#2);
lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);
lock(_xmit_ETHER#2);
lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);

*** DEADLOCK ***

10 locks held by syz-executor.5/7008:
#0: ffff8000159f4ce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x10/0x4c include/linux/rcupdate.h:305
#1: ffff8000159f4ce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x10/0x4c include/linux/rcupdate.h:305
#2: ffff8000159f4d40 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x18/0x54 include/linux/rcupdate.h:305
#3: ffff0000d296d4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:350 [inline]
#3: ffff0000d296d4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4304 [inline]
#3: ffff0000d296d4d8 (_xmit_ETHER#2){+.-.}-{2:2}, at: sch_direct_xmit+0x164/0x548 net/sched/sch_generic.c:340
#4: ffff0000d5e700b0 (k-slock-AF_INET6){+.-.}-{2:2}, at: spin_trylock include/linux/spinlock.h:360 [inline]
#4: ffff0000d5e700b0 (k-slock-AF_INET6){+.-.}-{2:2}, at: icmpv6_xmit_lock+0xec/0x19c net/ipv6/icmp.c:108
#5: ffff8000159f4ce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x10/0x4c include/linux/rcupdate.h:305
#6: ffff8000159f4ce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x10/0x4c include/linux/rcupdate.h:305
#7: ffff8000159f4ce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x10/0x4c include/linux/rcupdate.h:305
#8: ffff8000159f4ce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x10/0x4c include/linux/rcupdate.h:305
#9: ffff8000159f4d40 (rcu_read_lock_bh){....}-{1:2}, at: rcu_lock_acquire+0x18/0x54 include/linux/rcupdate.h:305

stack backtrace:
CPU: 0 PID: 7008 Comm: syz-executor.5 Not tainted 6.1.73-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/17/2023
Call trace:
dump_backtrace+0x1c8/0x1f4 arch/arm64/kernel/stacktrace.c:158
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:165
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x108/0x170 lib/dump_stack.c:106
dump_stack+0x1c/0x58 lib/dump_stack.c:113
print_circular_bug+0x150/0x1b8 kernel/locking/lockdep.c:2048
check_noncircular+0x2cc/0x378 kernel/locking/lockdep.c:2170
check_prev_add kernel/locking/lockdep.c:3090 [inline]
check_prevs_add kernel/locking/lockdep.c:3209 [inline]
validate_chain kernel/locking/lockdep.c:3825 [inline]
__lock_acquire+0x3338/0x7680 kernel/locking/lockdep.c:5049
lock_acquire+0x26c/0x7cc kernel/locking/lockdep.c:5662
__raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
_raw_spin_lock+0x54/0x6c kernel/locking/spinlock.c:154
spin_lock include/linux/spinlock.h:350 [inline]
__dev_xmit_skb net/core/dev.c:3884 [inline]
__dev_queue_xmit+0x1e3c/0x38d8 net/core/dev.c:4259
dev_queue_xmit include/linux/netdevice.h:3021 [inline]
neigh_resolve_output+0x518/0x618 net/core/neighbour.c:1556
neigh_output include/net/neighbour.h:544 [inline]
ip6_finish_output2+0xdac/0x1a88 net/ipv6/ip6_output.c:134
__ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
ip6_finish_output+0x5a4/0x940 net/ipv6/ip6_output.c:212
NF_HOOK_COND include/linux/netfilter.h:291 [inline]
ip6_output+0x270/0x594 net/ipv6/ip6_output.c:233
dst_output include/net/dst.h:444 [inline]
NF_HOOK include/linux/netfilter.h:302 [inline]
ndisc_send_skb+0xc38/0x179c net/ipv6/ndisc.c:509
ndisc_solicit+0x3ac/0x590
neigh_probe+0xc4/0x128 net/core/neighbour.c:1068
__neigh_event_send+0xe50/0x1668 net/core/neighbour.c:1235
neigh_event_send_probe include/net/neighbour.h:468 [inline]
neigh_event_send include/net/neighbour.h:474 [inline]
neigh_resolve_output+0x17c/0x618 net/core/neighbour.c:1540
neigh_output include/net/neighbour.h:544 [inline]
ip6_finish_output2+0xdac/0x1a88 net/ipv6/ip6_output.c:134
__ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
ip6_finish_output+0x5a4/0x940 net/ipv6/ip6_output.c:212
NF_HOOK_COND include/linux/netfilter.h:291 [inline]
ip6_output+0x270/0x594 net/ipv6/ip6_output.c:233
dst_output include/net/dst.h:444 [inline]
ip6_local_out+0x120/0x160 net/ipv6/output_core.c:161
ip6_send_skb+0x118/0x428 net/ipv6/ip6_output.c:1995
ip6_push_pending_frames+0xd0/0x118 net/ipv6/ip6_output.c:2015
icmpv6_push_pending_frames+0x288/0x3f4 net/ipv6/icmp.c:303
icmp6_send+0x11a8/0x1b2c net/ipv6/icmp.c:622
__icmpv6_send include/linux/icmpv6.h:28 [inline]
icmpv6_send include/linux/icmpv6.h:49 [inline]
ip6_link_failure+0x44/0x4a8 net/ipv6/route.c:2789
dst_link_failure+0x11c/0x160 include/net/dst.h:423
ip_tunnel_xmit+0x1e1c/0x28f4 net/ipv4/ip_tunnel.c:819
__gre_xmit net/ipv4/ip_gre.c:469 [inline]
erspan_xmit+0x9c8/0x1498 net/ipv4/ip_gre.c:718
__netdev_start_xmit include/linux/netdevice.h:4853 [inline]
netdev_start_xmit include/linux/netdevice.h:4867 [inline]
xmit_one net/core/dev.c:3627 [inline]
dev_hard_start_xmit+0x25c/0x9a4 net/core/dev.c:3643
sch_direct_xmit+0x234/0x548 net/sched/sch_generic.c:342
__dev_xmit_skb net/core/dev.c:3900 [inline]
__dev_queue_xmit+0x1758/0x38d8 net/core/dev.c:4259
dev_queue_xmit include/linux/netdevice.h:3021 [inline]
neigh_resolve_output+0x518/0x618 net/core/neighbour.c:1556
neigh_output include/net/neighbour.h:544 [inline]
ip6_finish_output2+0xdac/0x1a88 net/ipv6/ip6_output.c:134
__ip6_finish_output net/ipv6/ip6_output.c:201 [inline]
ip6_finish_output+0x5a4/0x940 net/ipv6/ip6_output.c:212
NF_HOOK_COND include/linux/netfilter.h:291 [inline]
ip6_output+0x270/0x594 net/ipv6/ip6_output.c:233
dst_output include/net/dst.h:444 [inline]
NF_HOOK include/linux/netfilter.h:302 [inline]
rawv6_send_hdrinc+0xd74/0x1bfc net/ipv6/raw.c:654
rawv6_sendmsg+0x1114/0x1bc0 net/ipv6/raw.c:912
inet_sendmsg+0x15c/0x290 net/ipv4/af_inet.c:836
sock_sendmsg_nosec net/socket.c:718 [inline]
__sock_sendmsg net/socket.c:730 [inline]
____sys_sendmsg+0x55c/0x848 net/socket.c:2514
___sys_sendmsg net/socket.c:2568 [inline]
__sys_sendmmsg+0x318/0x7d8 net/socket.c:2654
__do_sys_sendmmsg net/socket.c:2683 [inline]
__se_sys_sendmmsg net/socket.c:2680 [inline]
__arm64_sys_sendmmsg+0xa0/0xbc net/socket.c:2680
__invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
el0_svc_common+0x138/0x258 arch/arm64/kernel/syscall.c:142
do_el0_svc+0x64/0x218 arch/arm64/kernel/syscall.c:206
el0_svc+0x58/0x168 arch/arm64/kernel/entry-common.c:637
el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655
el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:585


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup
Reply all
Reply to author
Forward
0 new messages