possible deadlock in __dev_queue_xmit

19 views
Skip to first unread message

syzbot

unread,
Apr 13, 2019, 8:02:24 PM4/13/19
to syzkaller-a...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: 815e34f8 Merge 4.14.90 into android-4.14
git tree: android-4.14
console output: https://syzkaller.appspot.com/x/log.txt?x=1374d4c3400000
kernel config: https://syzkaller.appspot.com/x/.config?x=608dc5a2664d6079
dashboard link: https://syzkaller.appspot.com/bug?extid=4ee152e2fab0df969674
compiler: gcc (GCC) 8.0.1 20180413 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+4ee152...@syzkaller.appspotmail.com

IPv6: ADDRCONF(NETDEV_CHANGE): Y�4��`Ҙ: link becomes ready
ip6_tunnel: ip6tnl2 xmit: Local address not yet configured!
ip6_tunnel: ip6tnl1 xmit: Local address not yet configured!

============================================
WARNING: possible recursive locking detected
4.14.90+ #29 Not tainted
--------------------------------------------
syz-executor5/11042 is trying to acquire lock:
(_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>] spin_lock
include/linux/spinlock.h:317 [inline]
(_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>] __netif_tx_lock
include/linux/netdevice.h:3527 [inline]
(_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>]
__dev_queue_xmit+0x1348/0x1f40 net/core/dev.c:3521

but task is already holding lock:
(_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>] spin_lock
include/linux/spinlock.h:317 [inline]
(_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>] __netif_tx_lock
include/linux/netdevice.h:3527 [inline]
(_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>]
__dev_queue_xmit+0x1348/0x1f40 net/core/dev.c:3521

other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(_xmit_TUNNEL6#2);
lock(_xmit_TUNNEL6#2);

*** DEADLOCK ***

May be due to missing lock nesting notation

6 locks held by syz-executor5/11042:
#0: (rcu_read_lock_bh){....}, at: [<ffffffff9c6c5be5>]
ip_finish_output2+0x215/0x12f0 net/ipv4/ip_output.c:199
#1: (rcu_read_lock_bh){....}, at: [<ffffffff9c521585>]
__dev_queue_xmit+0x1e5/0x1f40 net/core/dev.c:3459
#2: (_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>] spin_lock
include/linux/spinlock.h:317 [inline]
#2: (_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>] __netif_tx_lock
include/linux/netdevice.h:3527 [inline]
#2: (_xmit_TUNNEL6#2){+.-.}, at: [<ffffffff9c5226e8>]
__dev_queue_xmit+0x1348/0x1f40 net/core/dev.c:3521
#3: (k-slock-AF_INET){+...}, at: [<ffffffff9c79b097>] spin_trylock
include/linux/spinlock.h:327 [inline]
#3: (k-slock-AF_INET){+...}, at: [<ffffffff9c79b097>] icmp_xmit_lock
net/ipv4/icmp.c:219 [inline]
#3: (k-slock-AF_INET){+...}, at: [<ffffffff9c79b097>]
icmp_send+0x4f7/0x12a0 net/ipv4/icmp.c:668
#4: (rcu_read_lock_bh){....}, at: [<ffffffff9c6c5be5>]
ip_finish_output2+0x215/0x12f0 net/ipv4/ip_output.c:199
#5: (rcu_read_lock_bh){....}, at: [<ffffffff9c521585>]
__dev_queue_xmit+0x1e5/0x1f40 net/core/dev.c:3459

stack backtrace:
CPU: 1 PID: 11042 Comm: syz-executor5 Not tainted 4.14.90+ #29
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0xb9/0x11b lib/dump_stack.c:53
print_deadlock_bug kernel/locking/lockdep.c:1796 [inline]
check_deadlock kernel/locking/lockdep.c:1843 [inline]
validate_chain kernel/locking/lockdep.c:2444 [inline]
__lock_acquire.cold.52+0x639/0xa99 kernel/locking/lockdep.c:3487
lock_acquire+0x10f/0x380 kernel/locking/lockdep.c:3991
__raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
_raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:152
spin_lock include/linux/spinlock.h:317 [inline]
__netif_tx_lock include/linux/netdevice.h:3527 [inline]
__dev_queue_xmit+0x1348/0x1f40 net/core/dev.c:3521
neigh_output include/net/neighbour.h:500 [inline]
ip_finish_output2+0x9ac/0x12f0 net/ipv4/ip_output.c:229
ip_finish_output+0x3a5/0xc40 net/ipv4/ip_output.c:317
NF_HOOK_COND include/linux/netfilter.h:239 [inline]
ip_output+0x1c9/0x520 net/ipv4/ip_output.c:405
dst_output include/net/dst.h:459 [inline]
ip_local_out+0x94/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x37/0xb0 net/ipv4/ip_output.c:1417
ip_push_pending_frames+0x5e/0x80 net/ipv4/ip_output.c:1437
icmp_push_reply+0x38e/0x510 net/ipv4/icmp.c:394
icmp_send+0x9ef/0x12a0 net/ipv4/icmp.c:741
ipv4_link_failure+0x24/0x1b0 net/ipv4/route.c:1196
dst_link_failure include/net/dst.h:442 [inline]
vti6_xmit net/ipv6/ip6_vti.c:521 [inline]
vti6_tnl_xmit+0xac0/0x1580 net/ipv6/ip6_vti.c:560
ip6_tunnel: ip6tnl2 xmit: Local address not yet configured!
__netdev_start_xmit include/linux/netdevice.h:4030 [inline]
netdev_start_xmit include/linux/netdevice.h:4039 [inline]
xmit_one net/core/dev.c:3009 [inline]
dev_hard_start_xmit+0x191/0x890 net/core/dev.c:3025
__dev_queue_xmit+0x13d9/0x1f40 net/core/dev.c:3525
ip6_tunnel: ip6tnl1 xmit: Local address not yet configured!
neigh_output include/net/neighbour.h:500 [inline]
ip_finish_output2+0x9ac/0x12f0 net/ipv4/ip_output.c:229
ip_finish_output+0x3a5/0xc40 net/ipv4/ip_output.c:317
NF_HOOK_COND include/linux/netfilter.h:239 [inline]
ip_output+0x1c9/0x520 net/ipv4/ip_output.c:405
dst_output include/net/dst.h:459 [inline]
ip_local_out+0x94/0x170 net/ipv4/ip_output.c:124
ip_send_skb+0x37/0xb0 net/ipv4/ip_output.c:1417
udp_send_skb+0x533/0xb90 net/ipv4/udp.c:829
udp_sendmsg+0x16ce/0x1da0 net/ipv4/udp.c:1053
udpv6_sendmsg+0x125f/0x2510 net/ipv6/udp.c:1193
inet_sendmsg+0x168/0x540 net/ipv4/af_inet.c:781
sock_sendmsg_nosec net/socket.c:645 [inline]
sock_sendmsg+0xb5/0x100 net/socket.c:655
sock_write_iter+0x209/0x350 net/socket.c:924
call_write_iter include/linux/fs.h:1784 [inline]
new_sync_write fs/read_write.c:471 [inline]
__vfs_write+0x417/0x5c0 fs/read_write.c:484
ip6_tunnel: ip6tnl1 xmit: Local address not yet configured!
vfs_write+0x17f/0x4d0 fs/read_write.c:546
SYSC_write fs/read_write.c:593 [inline]
SyS_write+0xc2/0x1a0 fs/read_write.c:585
do_syscall_64+0x19b/0x4b0 arch/x86/entry/common.c:289
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x457669
RSP: 002b:00007fe804392c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000457669
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 000000000072bf00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fe8043936d4
R13: 00000000004c6188 R14: 00000000004dada0 R15: 00000000ffffffff
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
ip6_tunnel: ip6tnl2 xmit: Local address not yet configured!
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
SELinux: security_context_str_to_sid(root) failed for (dev bpf, type bpf)
errno=-22
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 60 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 8 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 8 bytes leftover after parsing attributes in process
`syz-executor0'.
netlink: 8 bytes leftover after parsing attributes in process
`syz-executor0'.


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

syzbot

unread,
Apr 14, 2019, 5:30:20 AM4/14/19
to syzkaller-a...@googlegroups.com
Hello,

syzbot found the following crash on:

HEAD commit: 043c92bd UPSTREAM: mm: /proc/pid/smaps_rollup: fix NULL po..
git tree: android-4.9
console output: https://syzkaller.appspot.com/x/log.txt?x=104fcb9b400000
kernel config: https://syzkaller.appspot.com/x/.config?x=55dc746adbeac1dd
dashboard link: https://syzkaller.appspot.com/bug?extid=b2264ee620ec30188e63
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+b2264e...@syzkaller.appspotmail.com

=============================================
[ INFO: possible recursive locking detected ]
4.9.148+ #3 Not tainted
---------------------------------------------
syz-executor0/5021 is trying to acquire lock:
(_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>] spin_lock
include/linux/spinlock.h:302 [inline]
(_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>] __netif_tx_lock
include/linux/netdevice.h:3570 [inline]
(_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>]
__dev_queue_xmit+0x1116/0x1bd0 net/core/dev.c:3469
but task is already holding lock:
(_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>] spin_lock
include/linux/spinlock.h:302 [inline]
(_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>] __netif_tx_lock
include/linux/netdevice.h:3570 [inline]
(_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>]
__dev_queue_xmit+0x1116/0x1bd0 net/core/dev.c:3469
other info that might help us debug this:
Possible unsafe locking scenario:

CPU0
----
lock(_xmit_TUNNEL6#2);
lock(_xmit_TUNNEL6#2);

*** DEADLOCK ***

input: syz1 as /devices/virtual/input/input37
May be due to missing lock nesting notation

6 locks held by syz-executor0/5021:
#0: (rcu_read_lock_bh){......}, at: [<ffffffff824a8d0b>]
ip_finish_output2+0x20b/0x1280 net/ipv4/ip_output.c:198
#1: (rcu_read_lock_bh){......}, at: [<ffffffff82314e54>]
__dev_queue_xmit+0x1d4/0x1bd0 net/core/dev.c:3407
#2: (_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>] spin_lock
include/linux/spinlock.h:302 [inline]
#2: (_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>] __netif_tx_lock
include/linux/netdevice.h:3570 [inline]
#2: (_xmit_TUNNEL6#2){+.-...}, at: [<ffffffff82315d96>]
__dev_queue_xmit+0x1116/0x1bd0 net/core/dev.c:3469
#3: (slock-AF_INET){+.-...}, at: [<ffffffff8257ed54>] spin_trylock
include/linux/spinlock.h:312 [inline]
#3: (slock-AF_INET){+.-...}, at: [<ffffffff8257ed54>] icmp_xmit_lock
net/ipv4/icmp.c:220 [inline]
#3: (slock-AF_INET){+.-...}, at: [<ffffffff8257ed54>]
icmp_send+0x484/0x1410 net/ipv4/icmp.c:655
#4: (rcu_read_lock_bh){......}, at: [<ffffffff824a8d0b>]
ip_finish_output2+0x20b/0x1280 net/ipv4/ip_output.c:198
#5: (rcu_read_lock_bh){......}, at: [<ffffffff82314e54>]
__dev_queue_xmit+0x1d4/0x1bd0 net/core/dev.c:3407

stack backtrace:
CPU: 0 PID: 5021 Comm: syz-executor0 Not tainted 4.9.148+ #3
ffff8801732ee600 ffffffff81b456e1 ffffffff8424cd80 ffffffff83ccc510
ffffffff83ccc510 f5fec735e74a353d ffff88019f4f17c0 ffff8801732ee7a0
ffffffff81400f5c 0000000000000005 ffff88019f4f17c0 ffff8801732ee7c0
Call Trace:
[<ffffffff81b456e1>] __dump_stack lib/dump_stack.c:15 [inline]
[<ffffffff81b456e1>] dump_stack+0xc1/0x120 lib/dump_stack.c:51
[<ffffffff81400f5c>] print_deadlock_bug kernel/locking/lockdep.c:1727
[inline]
[<ffffffff81400f5c>] check_deadlock kernel/locking/lockdep.c:1771 [inline]
[<ffffffff81400f5c>] validate_chain kernel/locking/lockdep.c:2249 [inline]
[<ffffffff81400f5c>] __lock_acquire.cold+0x384/0x734
kernel/locking/lockdep.c:3345
[<ffffffff8120c2a3>] lock_acquire+0x133/0x3d0 kernel/locking/lockdep.c:3756
[<ffffffff82812708>] __raw_spin_lock include/linux/spinlock_api_smp.h:144
[inline]
[<ffffffff82812708>] _raw_spin_lock+0x38/0x50 kernel/locking/spinlock.c:151
[<ffffffff82315d96>] spin_lock include/linux/spinlock.h:302 [inline]
[<ffffffff82315d96>] __netif_tx_lock include/linux/netdevice.h:3570
[inline]
[<ffffffff82315d96>] __dev_queue_xmit+0x1116/0x1bd0 net/core/dev.c:3469
audit: type=1400 audit(2000000266.694:179): avc: denied { ioctl } for
pid=5016 comm="syz-executor5" path="socket:[183630]" dev="sockfs"
ino=183630 ioctlcmd=0x89a0
scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
tclass=netlink_generic_socket permissive=1
[<ffffffff82316868>] dev_queue_xmit+0x18/0x20 net/core/dev.c:3506
[<ffffffff8232e536>] neigh_direct_output+0x16/0x20
net/core/neighbour.c:1365
[<ffffffff824a91a2>] dst_neigh_output include/net/dst.h:470 [inline]
[<ffffffff824a91a2>] ip_finish_output2+0x6a2/0x1280
net/ipv4/ip_output.c:225
[<ffffffff824b0014>] ip_finish_output+0x3c4/0xce0 net/ipv4/ip_output.c:313
[<ffffffff824b44cc>] NF_HOOK_COND include/linux/netfilter.h:246 [inline]
[<ffffffff824b44cc>] ip_output+0x1ec/0x5b0 net/ipv4/ip_output.c:401
[<ffffffff824b123c>] dst_output include/net/dst.h:507 [inline]
[<ffffffff824b123c>] ip_local_out+0x9c/0x180 net/ipv4/ip_output.c:124
[<ffffffff824b71de>] ip_send_skb+0x3e/0xc0 net/ipv4/ip_output.c:1489
[<ffffffff824b72c4>] ip_push_pending_frames+0x64/0x80
net/ipv4/ip_output.c:1509
[<ffffffff8257de6e>] icmp_push_reply+0x39e/0x510 net/ipv4/icmp.c:381
[<ffffffff8257f39f>] icmp_send+0xacf/0x1410 net/ipv4/icmp.c:727
[<ffffffff82482e49>] ipv4_link_failure+0x29/0x1d0 net/ipv4/route.c:1178
[<ffffffff82790e63>] dst_link_failure include/net/dst.h:490 [inline]
[<ffffffff82790e63>] vti6_xmit net/ipv6/ip6_vti.c:521 [inline]
[<ffffffff82790e63>] vti6_tnl_xmit+0xb03/0x17e0 net/ipv6/ip6_vti.c:560
[<ffffffff82314265>] __netdev_start_xmit include/linux/netdevice.h:4069
[inline]
[<ffffffff82314265>] netdev_start_xmit include/linux/netdevice.h:4078
[inline]
[<ffffffff82314265>] xmit_one net/core/dev.c:2977 [inline]
[<ffffffff82314265>] dev_hard_start_xmit+0x195/0x8b0 net/core/dev.c:2993
[<ffffffff82315e23>] __dev_queue_xmit+0x11a3/0x1bd0 net/core/dev.c:3473
[<ffffffff82316868>] dev_queue_xmit+0x18/0x20 net/core/dev.c:3506
[<ffffffff8232e536>] neigh_direct_output+0x16/0x20
net/core/neighbour.c:1365
[<ffffffff824a91a2>] dst_neigh_output include/net/dst.h:470 [inline]
[<ffffffff824a91a2>] ip_finish_output2+0x6a2/0x1280
net/ipv4/ip_output.c:225
[<ffffffff824b0014>] ip_finish_output+0x3c4/0xce0 net/ipv4/ip_output.c:313
[<ffffffff824b44cc>] NF_HOOK_COND include/linux/netfilter.h:246 [inline]
[<ffffffff824b44cc>] ip_output+0x1ec/0x5b0 net/ipv4/ip_output.c:401
[<ffffffff824b123c>] dst_output include/net/dst.h:507 [inline]
[<ffffffff824b123c>] ip_local_out+0x9c/0x180 net/ipv4/ip_output.c:124
[<ffffffff824b71de>] ip_send_skb+0x3e/0xc0 net/ipv4/ip_output.c:1489
[<ffffffff8256142c>] udp_send_skb+0x4fc/0xc60 net/ipv4/udp.c:833
[<ffffffff82569a54>] udp_sendmsg+0x1634/0x1c60 net/ipv4/udp.c:1057
[<ffffffff8270564f>] udpv6_sendmsg+0x12af/0x2430 net/ipv6/udp.c:1086
[<ffffffff82592b72>] inet_sendmsg+0x202/0x4d0 net/ipv4/af_inet.c:770
[<ffffffff822a0ece>] sock_sendmsg_nosec net/socket.c:648 [inline]
[<ffffffff822a0ece>] sock_sendmsg+0xbe/0x110 net/socket.c:658
[<ffffffff822a2887>] ___sys_sendmsg+0x387/0x8b0 net/socket.c:1982
[<ffffffff822a5d04>] __sys_sendmmsg+0x164/0x3d0 net/socket.c:2072
[<ffffffff822a5fa5>] SYSC_sendmmsg net/socket.c:2103 [inline]
[<ffffffff822a5fa5>] SyS_sendmmsg+0x35/0x60 net/socket.c:2098
[<ffffffff810056bd>] do_syscall_64+0x1ad/0x570 arch/x86/entry/common.c:285
[<ffffffff82813393>] entry_SYSCALL_64_after_swapgs+0x5d/0xdb
audit: type=1400 audit(2000000267.448:180): avc: denied { write } for
pid=5062 comm="syz-executor5"
scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023
tclass=netlink_netfilter_socket permissive=1

syzbot

unread,
Oct 21, 2019, 5:31:04 PM10/21/19
to syzkaller-a...@googlegroups.com
Auto-closing this bug as obsolete.
Crashes did not happen for a while, no reproducer and no activity.

syzbot

unread,
Oct 25, 2019, 4:36:09 AM10/25/19
to syzkaller-a...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages