BUG: corrupted list in rxrpc_local_processor

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit: 87b983f5 Add linux-next specific files for 20190809
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=161309c2600000
kernel config: https://syzkaller.appspot.com/x/.config?x=28eea330e11df0eb
dashboard link: https://syzkaller.appspot.com/bug?extid=193e29e9387ea5837f1d
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+193e29...@syzkaller.appspotmail.com

list_del corruption. prev->next should be ffff8880a4570da0, but was
ffff88808c74b6e0
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN
CPU: 1 PID: 22 Comm: kworker/1:1 Not tainted 5.3.0-rc3-next-20190809 #63
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: krxrpcd rxrpc_local_processor
RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51
Code: e8 f9 73 1d fe 0f 0b 48 89 f1 48 c7 c7 c0 6f e6 87 4c 89 e6 e8 e5 73
1d fe 0f 0b 4c 89 f6 48 c7 c7 60 71 e6 87 e8 d4 73 1d fe <0f> 0b 4c 89 ea
4c 89 f6 48 c7 c7 a0 70 e6 87 e8 c0 73 1d fe 0f 0b
RSP: 0018:ffff8880a9a47cc0 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff8880a4570db8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815bb706 RDI: ffffed1015348f8a
RBP: ffff8880a9a47cd8 R08: 0000000000000054 R09: ffffed1015d260d9
R10: ffffed1015d260d8 R11: ffff8880ae9306c7 R12: ffff888074400878
R13: ffff888074400878 R14: ffff8880a4570da0 R15: ffff88809509e580
FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000400200 CR3: 000000006f1ac000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
__list_del_entry include/linux/list.h:131 [inline]
list_del_init include/linux/list.h:190 [inline]
rxrpc_local_destroyer net/rxrpc/local_object.c:427 [inline]
rxrpc_local_processor+0x251/0x830 net/rxrpc/local_object.c:463
process_one_work+0x9af/0x1740 kernel/workqueue.c:2269
worker_thread+0x98/0xe40 kernel/workqueue.c:2415
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Modules linked in:
---[ end trace c8e00778000f001d ]---
RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51
Code: e8 f9 73 1d fe 0f 0b 48 89 f1 48 c7 c7 c0 6f e6 87 4c 89 e6 e8 e5 73
1d fe 0f 0b 4c 89 f6 48 c7 c7 60 71 e6 87 e8 d4 73 1d fe <0f> 0b 4c 89 ea
4c 89 f6 48 c7 c7 a0 70 e6 87 e8 c0 73 1d fe 0f 0b
RSP: 0018:ffff8880a9a47cc0 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff8880a4570db8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815bb706 RDI: ffffed1015348f8a
RBP: ffff8880a9a47cd8 R08: 0000000000000054 R09: ffffed1015d260d9
R10: ffffed1015d260d8 R11: ffff8880ae9306c7 R12: ffff888074400878
R13: ffff888074400878 R14: ffff8880a4570da0 R15: ffff88809509e580
FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b32f22000 CR3: 000000006f1ac000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit: 125b7e09 net: tc35815: Explicitly check NET_IP_ALIGN is no..
git tree: net
console output: https://syzkaller.appspot.com/x/log.txt?x=16fb7bc2600000
kernel config: https://syzkaller.appspot.com/x/.config?x=a4c9e9f08e9e8960

dashboard link: https://syzkaller.appspot.com/bug?extid=193e29e9387ea5837f1d
compiler: gcc (GCC) 9.0.0 20181231 (experimental)

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159d4eba600000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ba194a600000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+193e29...@syzkaller.appspotmail.com

IPv6: ADDRCONF(NETDEV_CHANGE): hsr_slave_1: link becomes ready
list_del corruption. prev->next should be ffff8880996e84e0, but was
ffff8880996e8960

------------[ cut here ]------------
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN

CPU: 1 PID: 17 Comm: kworker/1:0 Not tainted 5.3.0-rc3+ #159

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: krxrpcd rxrpc_local_processor
RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51

Code: e8 e9 03 1e fe 0f 0b 48 89 f1 48 c7 c7 80 25 c6 87 4c 89 e6 e8 d5 03
1e fe 0f 0b 4c 89 f6 48 c7 c7 20 27 c6 87 e8 c4 03 1e fe <0f> 0b 4c 89 ea
4c 89 f6 48 c7 c7 60 26 c6 87 e8 b0 03 1e fe 0f 0b
RSP: 0018:ffff8880a9917cc0 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff8880996e84f8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815c3b96 RDI: ffffed1015322f8a
RBP: ffff8880a9917cd8 R08: 0000000000000054 R09: ffffed1015d260d1
R10: ffffed1015d260d0 R11: ffff8880ae930687 R12: ffff88808f998638
R13: ffff88808f998638 R14: ffff8880996e84e0 R15: ffff8880aa0a8500

FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f2648a08db8 CR3: 00000000981a9000 CR4: 00000000001406e0

Call Trace:
__list_del_entry include/linux/list.h:131 [inline]
list_del_init include/linux/list.h:190 [inline]

rxrpc_local_destroyer net/rxrpc/local_object.c:429 [inline]
rxrpc_local_processor+0x251/0x830 net/rxrpc/local_object.c:465

process_one_work+0x9af/0x1740 kernel/workqueue.c:2269
worker_thread+0x98/0xe40 kernel/workqueue.c:2415
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Modules linked in:

---[ end trace 4d70382ddfcfe2b8 ]---

RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51

Code: e8 e9 03 1e fe 0f 0b 48 89 f1 48 c7 c7 80 25 c6 87 4c 89 e6 e8 d5 03
1e fe 0f 0b 4c 89 f6 48 c7 c7 20 27 c6 87 e8 c4 03 1e fe <0f> 0b 4c 89 ea
4c 89 f6 48 c7 c7 60 26 c6 87 e8 b0 03 1e fe 0f 0b
RSP: 0018:ffff8880a9917cc0 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff8880996e84f8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815c3b96 RDI: ffffed1015322f8a
RBP: ffff8880a9917cd8 R08: 0000000000000054 R09: ffffed1015d260d1
R10: ffffed1015d260d0 R11: ffff8880ae930687 R12: ffff88808f998638
R13: ffff88808f998638 R14: ffff8880996e84e0 R15: ffff8880aa0a8500

FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f2648a08db8 CR3: 00000000981a9000 CR4: 00000000001406e0

[syzbot]

syzbot has bisected this bug to:

commit 427545b3046326cd7b4dbbd7869f08737df2ad2b
Author: Jakub Kicinski <jakub.k...@netronome.com>
Date: Tue Jul 9 02:53:12 2019 +0000

nfp: tls: count TSO segments separately for the TLS offload

bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=11d04eee600000
start commit: 125b7e09 net: tc35815: Explicitly check NET_IP_ALIGN is no..
git tree: net
final crash: https://syzkaller.appspot.com/x/report.txt?x=13d04eee600000
console output: https://syzkaller.appspot.com/x/log.txt?x=15d04eee600000

kernel config: https://syzkaller.appspot.com/x/.config?x=a4c9e9f08e9e8960
dashboard link: https://syzkaller.appspot.com/bug?extid=193e29e9387ea5837f1d

syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159d4eba600000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16ba194a600000

Reported-by: syzbot+193e29...@syzkaller.appspotmail.com
Fixes: 427545b30463 ("nfp: tls: count TSO segments separately for the TLS
offload")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[Jakub Kicinski]

Is there a way perhaps to tell syzbot to discard clearly bogus
bisection results?

[David Howells]

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git 03a62469fffcbd535d85e42ef25ba098262e9d72

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer still triggered
crash:
KASAN: use-after-free Read in rxrpc_queue_local

==================================================================
BUG: KASAN: use-after-free in atomic_read
include/asm-generic/atomic-instrumented.h:26 [inline]
BUG: KASAN: use-after-free in rxrpc_queue_local+0x7c/0x3e0
net/rxrpc/local_object.c:354
Read of size 4 at addr ffff8880a82b56d4 by task syz-executor.0/11829

CPU: 1 PID: 11829 Comm: syz-executor.0 Not tainted 5.3.0-rc3+ #1

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011

Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x172/0x1f0 lib/dump_stack.c:113
print_address_description.cold+0xd4/0x306 mm/kasan/report.c:351
__kasan_report.cold+0x1b/0x36 mm/kasan/report.c:482
kasan_report+0x12/0x17 mm/kasan/common.c:612
check_memory_region_inline mm/kasan/generic.c:185 [inline]
check_memory_region+0x134/0x1a0 mm/kasan/generic.c:192
__kasan_check_read+0x11/0x20 mm/kasan/common.c:92
atomic_read include/asm-generic/atomic-instrumented.h:26 [inline]
rxrpc_queue_local+0x7c/0x3e0 net/rxrpc/local_object.c:354
rxrpc_unuse_local+0x52/0x80 net/rxrpc/local_object.c:408
rxrpc_release_sock net/rxrpc/af_rxrpc.c:904 [inline]
rxrpc_release+0x47d/0x840 net/rxrpc/af_rxrpc.c:930
__sock_release+0xce/0x280 net/socket.c:590
sock_close+0x1e/0x30 net/socket.c:1268
__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113
tracehook_notify_resume include/linux/tracehook.h:188 [inline]
exit_to_usermode_loop+0x316/0x380 arch/x86/entry/common.c:163
prepare_exit_to_usermode arch/x86/entry/common.c:194 [inline]
syscall_return_slowpath arch/x86/entry/common.c:274 [inline]
do_syscall_64+0x5a9/0x6a0 arch/x86/entry/common.c:299
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x413511
Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 04 1b 00 00 c3 48
83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48
89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01
RSP: 002b:00007ffc204e87c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000413511
RDX: 0000001b2e420000 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000001 R08: ffffffffffffffff R09: ffffffffffffffff
R10: 00007ffc204e88a0 R11: 0000000000000293 R12: 000000000075bf20
R13: 000000000001ac29 R14: 0000000000760210 R15: ffffffffffffffff

Allocated by task 11830:
save_stack+0x23/0x90 mm/kasan/common.c:69
set_track mm/kasan/common.c:77 [inline]
__kasan_kmalloc mm/kasan/common.c:487 [inline]
__kasan_kmalloc.constprop.0+0xcf/0xe0 mm/kasan/common.c:460
kasan_kmalloc+0x9/0x10 mm/kasan/common.c:501
kmem_cache_alloc_trace+0x158/0x790 mm/slab.c:3550
kmalloc include/linux/slab.h:552 [inline]
kzalloc include/linux/slab.h:748 [inline]
rxrpc_alloc_local net/rxrpc/local_object.c:79 [inline]
rxrpc_lookup_local+0x562/0x1ba0 net/rxrpc/local_object.c:277
rxrpc_sendmsg+0x379/0x5f0 net/rxrpc/af_rxrpc.c:566
sock_sendmsg_nosec net/socket.c:637 [inline]
sock_sendmsg+0xd7/0x130 net/socket.c:657
___sys_sendmsg+0x3e2/0x920 net/socket.c:2311
__sys_sendmmsg+0x1bf/0x4d0 net/socket.c:2413
__do_sys_sendmmsg net/socket.c:2442 [inline]
__se_sys_sendmmsg net/socket.c:2439 [inline]
__x64_sys_sendmmsg+0x9d/0x100 net/socket.c:2439
do_syscall_64+0xfd/0x6a0 arch/x86/entry/common.c:296
entry_SYSCALL_64_after_hwframe+0x49/0xbe

Freed by task 16:
save_stack+0x23/0x90 mm/kasan/common.c:69
set_track mm/kasan/common.c:77 [inline]
__kasan_slab_free+0x102/0x150 mm/kasan/common.c:449
kasan_slab_free+0xe/0x10 mm/kasan/common.c:457
__cache_free mm/slab.c:3425 [inline]
kfree+0x10a/0x2c0 mm/slab.c:3756
rxrpc_local_rcu+0x62/0x80 net/rxrpc/local_object.c:495
__rcu_reclaim kernel/rcu/rcu.h:222 [inline]
rcu_do_batch kernel/rcu/tree.c:2114 [inline]
rcu_core+0x67f/0x1580 kernel/rcu/tree.c:2314
rcu_core_si+0x9/0x10 kernel/rcu/tree.c:2323
__do_softirq+0x262/0x98c kernel/softirq.c:292

The buggy address belongs to the object at ffff8880a82b56c0
which belongs to the cache kmalloc-1k of size 1024
The buggy address is located 20 bytes inside of
1024-byte region [ffff8880a82b56c0, ffff8880a82b5ac0)
The buggy address belongs to the page:
page:ffffea0002a0ad00 refcount:1 mapcount:0 mapping:ffff8880aa400c40
index:0xffff8880a82b5b40 compound_mapcount: 0
flags: 0x1fffc0000010200(slab|head)
raw: 01fffc0000010200 ffffea0002a1cb08 ffffea00023b0c88 ffff8880aa400c40
raw: ffff8880a82b5b40 ffff8880a82b4040 0000000100000006 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
ffff8880a82b5580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8880a82b5600: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
> ffff8880a82b5680: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
^
ffff8880a82b5700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8880a82b5780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


Tested on:

commit: 03a62469 rxrpc: Fix local endpoint replacement
git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git
console output: https://syzkaller.appspot.com/x/log.txt?x=15f1679a600000
kernel config: https://syzkaller.appspot.com/x/.config?x=a4c9e9f08e9e8960

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer still triggered
crash:

BUG: corrupted list in rxrpc_local_processor

list_del corruption. prev->next should be ffff888099146de0, but was
ffff8880a1b89220

------------[ cut here ]------------
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN

CPU: 1 PID: 3517 Comm: kworker/1:2 Not tainted 5.3.0-rc3+ #1

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011

Workqueue: krxrpcd rxrpc_local_processor
RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51
Code: e8 e9 03 1e fe 0f 0b 48 89 f1 48 c7 c7 80 25 c6 87 4c 89 e6 e8 d5 03
1e fe 0f 0b 4c 89 f6 48 c7 c7 20 27 c6 87 e8 c4 03 1e fe <0f> 0b 4c 89 ea
4c 89 f6 48 c7 c7 60 26 c6 87 e8 b0 03 1e fe 0f 0b

RSP: 0018:ffff88809cb4fcc0 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff888099146df8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815c3b96 RDI: ffffed1013969f8a
RBP: ffff88809cb4fcd8 R08: 0000000000000054 R09: ffffed1015d260d1
R10: ffffed1015d260d0 R11: ffff8880ae930687 R12: ffff888094d78738
R13: ffff888094d78738 R14: ffff888099146de0 R15: ffff88809293b580

FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 0000000020003c40 CR3: 000000009ae10000 CR4: 00000000001406e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Call Trace:
__list_del_entry include/linux/list.h:131 [inline]
list_del_init include/linux/list.h:190 [inline]

rxrpc_local_destroyer net/rxrpc/local_object.c:430 [inline]
rxrpc_local_processor+0x253/0x870 net/rxrpc/local_object.c:466

process_one_work+0x9af/0x1740 kernel/workqueue.c:2269
worker_thread+0x98/0xe40 kernel/workqueue.c:2415
kthread+0x361/0x430 kernel/kthread.c:255
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Modules linked in:

---[ end trace 485fb694b3023b44 ]---

RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51
Code: e8 e9 03 1e fe 0f 0b 48 89 f1 48 c7 c7 80 25 c6 87 4c 89 e6 e8 d5 03
1e fe 0f 0b 4c 89 f6 48 c7 c7 20 27 c6 87 e8 c4 03 1e fe <0f> 0b 4c 89 ea
4c 89 f6 48 c7 c7 60 26 c6 87 e8 b0 03 1e fe 0f 0b

RSP: 0018:ffff88809cb4fcc0 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff888099146df8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815c3b96 RDI: ffffed1013969f8a
RBP: ffff88809cb4fcd8 R08: 0000000000000054 R09: ffffed1015d260d1
R10: ffffed1015d260d0 R11: ffff8880ae930687 R12: ffff888094d78738
R13: ffff888094d78738 R14: ffff888099146de0 R15: ffff88809293b580

FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 0000000020003c40 CR3: 000000009ae10000 CR4: 00000000001406e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Tested on:

commit: 845c5b49 rxrpc: Fix read-after-free in rxrpc_queue_local()
git tree:
git://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git
console output: https://syzkaller.appspot.com/x/log.txt?x=11c0b422600000