x86: warning in unwind_get_return_address

[Andrey Konovalov]

Hi,

I've got the following warning while running the syzkaller fuzzer:

WARNING: unrecognized kernel stack return address ffffffffa0000001 at
ffff88006377fa18 in a.out:4467

By adding a BUG() to unwind_get_return_address() I was able to capture
the stack trace (see below). Looks like unwind_get_return_address()
gets called when KASAN tries to unwind the stack to save the stack
trace.

A reproducer is attached. CONFIG_KASAN=y is most likely needed for it to work.

On commit e93b1cc8a8965da137ffea0b88e5f62fa1d2a9e6 (Dec 19).

------------[ cut here ]------------
kernel BUG at arch/x86/kernel/unwind_frame.c:27!
invalid opcode: 0000 [#1] SMP KASAN
Modules linked in:
CPU: 1 PID: 4467 Comm: a.out Not tainted 4.9.0+ #53
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
task: ffff8800665b9600 task.stack: ffff880063778000
RIP: 0010:unwind_get_return_address+0xcc/0x1b0 arch/x86/kernel/unwind_frame.c:24
RSP: 0018:ffff88006cb06fa0 EFLAGS: 00010286
RAX: 0000000000000064 RBX: ffff8800665b9600 RCX: 0000000000000000
RDX: 0000000000000100 RSI: ffff88006cb15f08 RDI: ffffed000d960de6
RBP: ffff88006cb06fb8 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000009 R11: 0000000000000000 R12: ffffffffa0000001
R13: ffff88006377fa18 R14: ffff8800665b9600 R15: 0000000000000246
FS: 00007f7c347947c0(0000) GS:ffff88006cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020004fc8 CR3: 0000000062668000 CR4: 00000000000006e0
Call Trace:
<IRQ>
__save_stack_trace+0x7e/0xd0 arch/x86/kernel/stacktrace.c:43
save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
save_stack+0x43/0xd0 mm/kasan/kasan.c:502
set_track mm/kasan/kasan.c:514
kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:578
slab_free_hook mm/slub.c:1352
slab_free_freelist_hook mm/slub.c:1374
slab_free mm/slub.c:2951
kmem_cache_free+0xb2/0x2c0 mm/slub.c:2973
file_free_rcu+0x6d/0xa0 fs/file_table.c:49
__rcu_reclaim kernel/rcu/rcu.h:118
rcu_do_batch.isra.67+0x900/0xc50 kernel/rcu/tree.c:2777
invoke_rcu_callbacks kernel/rcu/tree.c:3040
__rcu_process_callbacks kernel/rcu/tree.c:3007
rcu_process_callbacks+0x2b7/0xba0 kernel/rcu/tree.c:3024
__do_softirq+0x2fb/0xb7d kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364
irq_exit+0x19e/0x1d0 kernel/softirq.c:405
exiting_irq ./arch/x86/include/asm/apic.h:658
smp_apic_timer_interrupt+0x76/0xa0 arch/x86/kernel/apic/apic.c:961
apic_timer_interrupt+0x93/0xa0
RIP: 0010:0xffffffffa0000001
RSP: 0018:ffff88006377fa48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
RAX: ffff8800665b9600 RBX: ffff880066471400 RCX: 1ffffffff0bc9005
RDX: 0000000000000000 RSI: ffff88006377fe78 RDI: ffff880066471400
RBP: ffff88006377fb00 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff1000c6eff4f
R13: ffffffff85e47fc0 R14: 0000000000000000 R15: ffff880066471428
</IRQ>
Code: 00 fc ff df 48 c1 ea 03 80 3c 02 00 0f 85 eb 00 00 00 4d 8b 65
00 4c 89 e7 e8 21 75 0c 00 85 c0 75 0e 80 3d 44 fc c9 04 00 74 4f <0f>
0b 45 31 e4 5b 4c 89 e0 41 5c 41 5d 5d c3 48 8d 7b 38 48 b8
RIP: unwind_get_return_address+0xcc/0x1b0 RSP: ffff88006cb06fa0
---[ end trace a4f6a441af47c2dc ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt

[Josh Poimboeuf]

On Tue, Dec 20, 2016 at 03:43:27PM +0100, Andrey Konovalov wrote:
> Hi,
>
> I've got the following warning while running the syzkaller fuzzer:
>
> WARNING: unrecognized kernel stack return address ffffffffa0000001 at
> ffff88006377fa18 in a.out:4467
>
> By adding a BUG() to unwind_get_return_address() I was able to capture
> the stack trace (see below). Looks like unwind_get_return_address()
> gets called when KASAN tries to unwind the stack to save the stack
> trace.
>
> A reproducer is attached. CONFIG_KASAN=y is most likely needed for it to work.

Hi Andrey,

I've tried with your reproducer but it didn't recreate. Can you try
again with the following patch from the tip tree, instead of your BUG()
patch?

http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/patch/?id=8b5e99f02264130782a10ba5c0c759797fb064ee

That will dump the stack data, which should give more clues about what
went wrong.

--
Josh

[Andrey Konovalov]

Hi Josh,

Sure, here it is:

[ 26.106079] WARNING: unrecognized kernel stack return address
ffffffffa0000000 at ffff8800646e7a28 in a.out:4232
[ 26.106086] unwind stack type:1 next_sp: (null) mask:6 graph_idx:0
[ 26.106094] ffff8800646e79a8: ffff8800683e0028 (0xffff8800683e0028)
[ 26.106098] ffff8800646e79b0: 0000000000000000 ...
[ 26.106123] ffff8800646e79b8: ffffffff85e480c0 (dccp_v6_protosw+0x60/0x60)
[ 26.106129] ffff8800646e79c0: 1ffff1000c8dcf4f (0x1ffff1000c8dcf4f)
[ 26.106134] ffff8800646e79c8: ffff8800646e7b00 (0xffff8800646e7b00)
[ 26.106139] ffff8800646e79d0: ffff8800683e0000 (0xffff8800683e0000)
[ 26.106143] ffff8800646e79d8: 0000000000000000 ...
[ 26.106147] ffff8800646e79f0: 0000000000000001 (0x1)
[ 26.106153] ffff8800646e79f8: ffff88006765c200 (0xffff88006765c200)
[ 26.106158] ffff8800646e7a00: 1ffffffff0bc9025 (0x1ffffffff0bc9025)
[ 26.106161] ffff8800646e7a08: 0000000000000000 ...
[ 26.106166] ffff8800646e7a10: ffff8800646e7e78 (0xffff8800646e7e78)
[ 26.106171] ffff8800646e7a18: ffff8800683e0000 (0xffff8800683e0000)
[ 26.106177] ffff8800646e7a20: ffffffffffffff10 (0xffffffffffffff10)
[ 26.106182] ffff8800646e7a28: ffffffffa0000000 (0xffffffffa0000000)
[ 26.106186] ffff8800646e7a30: 0000000000000010 (0x10)
[ 26.106191] ffff8800646e7a38: 0000000000000246 (0x246)
[ 26.106196] ffff8800646e7a40: ffff8800646e7a50 (0xffff8800646e7a50)
[ 26.106201] ffff8800646e7a48: 0000000000000018 (0x18)
[ 26.106212] ffff8800646e7a50: ffffffff83a987a4 (inet_sendmsg+0x164/0x5b0)
[ 26.106222] ffff8800646e7a58: ffffffff81842514 (kasan_check_write+0x14/0x20)
[ 26.106228] ffff8800646e7a60: ffff8800646e7c40 (0xffff8800646e7c40)
[ 26.106237] ffff8800646e7a68: ffffffff8206ac89 (_copy_from_user+0x99/0x120)
[ 26.106242] ffff8800646e7a70: ffff8800646e7e78 (0xffff8800646e7e78)
[ 26.106247] ffff8800646e7a78: 0000000041b58ab3 (0x41b58ab3)
[ 26.106257] ffff8800646e7a80: ffffffff855d9233
(__func__.54982+0x22a033/0x2d46f0)
[ 26.106266] ffff8800646e7a88: ffffffff83a98640 (inet_recvmsg+0x610/0x610)
[ 26.106271] ffff8800646e7a90: ffff8800659b8cc0 (0xffff8800659b8cc0)
[ 26.106276] ffff8800646e7a98: ffff8800646e7e78 (0xffff8800646e7e78)
[ 26.106280] ffff8800646e7aa0: 0000000000000000 ...
[ 26.106285] ffff8800646e7aa8: ffff8800646e7ac0 (0xffff8800646e7ac0)
[ 26.106294] ffff8800646e7ab0: ffffffff81e75f4f
(selinux_socket_sendmsg+0x3f/0x50)
[ 26.106303] ffff8800646e7ab8: ffffffff85a27d00 (selinux_hooks+0x10c0/0x1560)
[ 26.106308] ffff8800646e7ac0: ffff8800646e7b00 (0xffff8800646e7b00)
[ 26.106317] ffff8800646e7ac8: ffffffff81e4e749
(security_socket_sendmsg+0x89/0xb0)
[ 26.106322] ffff8800646e7ad0: 000000008331fbd3 (0x8331fbd3)
[ 26.106337] ffff8800646e7ad8: ffff8800646e7e78 (0xffff8800646e7e78)
[ 26.106343] ffff8800646e7ae0: ffffffff83a98640 (inet_recvmsg+0x610/0x610)
[ 26.106347] ffff8800646e7ae8: ffff8800659b8cc0 (0xffff8800659b8cc0)
[ 26.106350] ffff8800646e7af0: ffff8800646e7e98 (0xffff8800646e7e98)
[ 26.106354] ffff8800646e7af8: ffff8800646e7be0 (0xffff8800646e7be0)
[ 26.106358] ffff8800646e7b00: ffff8800646e7b30 (0xffff8800646e7b30)
[ 26.106365] ffff8800646e7b08: ffffffff8331de8a (sock_sendmsg+0xca/0x110)
[ 26.106369] ffff8800646e7b10: ffff8800646e7c40 (0xffff8800646e7c40)
[ 26.106373] ffff8800646e7b18: ffff8800646e7e78 (0xffff8800646e7e78)
[ 26.106376] ffff8800646e7b20: ffff8800649aca00 (0xffff8800649aca00)
[ 26.106380] ffff8800646e7b28: 0000000000000040 (0x40)
[ 26.106392] ffff8800646e7b30: ffff8800646e7da8 (0xffff8800646e7da8)
[ 26.106399] ffff8800646e7b38: ffffffff833207d2 (___sys_sendmsg+0x9d2/0xae0)
[ 26.106402] ffff8800646e7b40: ffff8800646e7eb8 (0xffff8800646e7eb8)
[ 26.106405] ffff8800646e7b48: 0000000000000000 ...
[ 26.106408] ffff8800646e7b58: ffff8800646e7be0 (0xffff8800646e7be0)
[ 26.106412] ffff8800646e7b60: 1ffff1000c8dcf70 (0x1ffff1000c8dcf70)
[ 26.106416] ffff8800646e7b68: ffff8800659b8cc0 (0xffff8800659b8cc0)
[ 26.106419] ffff8800646e7b70: ffff8800646e7ce0 (0xffff8800646e7ce0)
[ 26.106423] ffff8800646e7b78: ffff8800646e7ba0 (0xffff8800646e7ba0)
[ 26.106427] ffff8800646e7b80: 0000000041b58ab3 (0x41b58ab3)
[ 26.106433] ffff8800646e7b88: ffffffff856a04f0
(_fw_yam_9600_bin_name+0x1cb68/0x5d928)
[ 26.106440] ffff8800646e7b90: ffffffff8331fe00
(copy_msghdr_from_user+0x550/0x550)
[ 26.106447] ffff8800646e7b98: ffffffff8141f810 (lock_acquire+0x580/0x580)
[ 26.106449] ffff8800646e7ba0: 0000000000000000 ...
[ 26.106453] ffff8800646e7ba8: dead000000000100 (0xdead000000000100)
[ 26.106457] ffff8800646e7bb0: 000077ff80000000 (0x77ff80000000)
[ 26.106461] ffff8800646e7bb8: ffff8800646e7bd0 (0xffff8800646e7bd0)
[ 26.106469] ffff8800646e7bc0: ffffffff84c29b32 (_raw_spin_unlock+0x22/0x30)
[ 26.106473] ffff8800646e7bc8: ffff88006bb64bb0 (0xffff88006bb64bb0)
[ 26.106477] ffff8800646e7bd0: ffff88006bb64bb0 (0xffff88006bb64bb0)
[ 26.106480] ffff8800646e7bd8: ffff8800696822e8 (0xffff8800696822e8)
[ 26.106484] ffff8800646e7be0: ffff8800646e7d40 (0xffff8800646e7d40)
[ 26.106488] ffff8800646e7be8: ffff880064974020 (0xffff880064974020)
[ 26.106492] ffff8800646e7bf0: ffff8800696822e8 (0xffff8800696822e8)
[ 26.106495] ffff8800646e7bf8: ffff8800646e7c10 (0xffff8800646e7c10)
[ 26.106502] ffff8800646e7c00: ffffffff84c29b32 (_raw_spin_unlock+0x22/0x30)
[ 26.106505] ffff8800646e7c08: ffff8800646e7cc0 (0xffff8800646e7cc0)
[ 26.106509] ffff8800646e7c10: ffff8800646e7d68 (0xffff8800646e7d68)
[ 26.106515] ffff8800646e7c18: ffffffff817bae1b (handle_mm_fault+0xafb/0x1f30)
[ 26.106519] ffff8800646e7c20: 0000000064974067 (0x64974067)
[ 26.106522] ffff8800646e7c28: ffff8800678f0580 (0xffff8800678f0580)
[ 26.106526] ffff8800646e7c30: 8000000062b56067 (0x8000000062b56067)
[ 26.106533] ffff8800646e7c38: ffffffff818d8a2a (__fget_light+0x2aa/0x3e0)
[ 26.106537] ffff8800646e7c40: ffff880066711e58 (0xffff880066711e58)
[ 26.106540] ffff8800646e7c48: 0000400000000000 (0x400000000000)
[ 26.106544] ffff8800646e7c50: 0000000041b58ab3 (0x41b58ab3)
[ 26.106550] ffff8800646e7c58: ffffffff855e6380
(__func__.54982+0x237180/0x2d46f0)
[ 26.106556] ffff8800646e7c60: ffffffff818d8780 (fget_raw+0x20/0x20)
[ 26.106562] ffff8800646e7c68: ffffffff856104ba
(__func__.54982+0x2612ba/0x2d46f0)
[ 26.106566] ffff8800646e7c70: ffffffff00000001 (0xffffffff00000001)
[ 26.106569] ffff8800646e7c78: 0000000041b58ab3 (0x41b58ab3)
[ 26.106572] ffff8800646e7c80: 0000000000000000 ...
[ 26.106577] ffff8800646e7c88: ffffffff81410f40 (__lock_is_held+0x140/0x140)
[ 26.106581] ffff8800646e7c90: 0000000000000003 (0x3)
[ 26.106585] ffff8800646e7c98: ffff8800678f0620 (0xffff8800678f0620)
[ 26.106588] ffff8800646e7ca0: 1ffff1000c8dcf9c (0x1ffff1000c8dcf9c)
[ 26.106591] ffff8800646e7ca8: 0000000000000001 (0x1)
[ 26.106595] ffff8800646e7cb0: ffff880066711e58 (0xffff880066711e58)
[ 26.106599] ffff8800646e7cb8: ffff8800678f0678 (0xffff8800678f0678)
[ 26.106602] ffff8800646e7cc0: ffff8800696822e8 (0xffff8800696822e8)
[ 26.106606] ffff8800646e7cc8: 024000c000000055 (0x24000c000000055)
[ 26.106610] ffff8800646e7cd0: 0000000000020004 (0x20004)
[ 26.106613] ffff8800646e7cd8: 0000000020004000 (0x20004000)
[ 26.106617] ffff8800646e7ce0: ffff8800662d0000 (0xffff8800662d0000)
[ 26.106619] ffff8800646e7ce8: 0000000000000000 ...
[ 26.106623] ffff8800646e7cf0: ffff8800649aca00 (0xffff8800649aca00)
[ 26.106625] ffff8800646e7cf8: 0000000000000000 ...
[ 26.106629] ffff8800646e7d08: ffff880064974020 (0xffff880064974020)
[ 26.106633] ffff8800646e7d10: ffff88006bb64bb0 (0xffff88006bb64bb0)
[ 26.106635] ffff8800646e7d18: 0000000000000000 ...
[ 26.106638] ffff8800646e7d20: 0000000020004fc8 (0x20004fc8)
[ 26.106642] ffff8800646e7d28: ffff8800678f0620 (0xffff8800678f0620)
[ 26.106645] ffff8800646e7d30: 0000000000000003 (0x3)
[ 26.106649] ffff8800646e7d38: ffff8800646e7e38 (0xffff8800646e7e38)
[ 26.106653] ffff8800646e7d40: ffff8800646e7e38 (0xffff8800646e7e38)
[ 26.106656] ffff8800646e7d48: ffff8800646e7df8 (0xffff8800646e7df8)
[ 26.106660] ffff8800646e7d50: 1ffff1000c8dcfbb (0x1ffff1000c8dcfbb)
[ 26.106664] ffff8800646e7d58: ffff8800646e7d70 (0xffff8800646e7d70)
[ 26.106670] ffff8800646e7d60: ffffffff818d8b78 (__fdget+0x18/0x20)
[ 26.106673] ffff8800646e7d68: 0000000000000003 (0x3)
[ 26.106677] ffff8800646e7d70: ffff8800646e7da8 (0xffff8800646e7da8)
[ 26.106683] ffff8800646e7d78: ffffffff8331ac04
(sockfd_lookup_light+0x104/0x150)
[ 26.106687] ffff8800646e7d80: ffff8800646e7ef8 (0xffff8800646e7ef8)
[ 26.106690] ffff8800646e7d88: ffffed000c8dcfc7 (0xffffed000c8dcfc7)
[ 26.106694] ffff8800646e7d90: ffff8800646e7e38 (0xffff8800646e7e38)
[ 26.106698] ffff8800646e7d98: ffff8800659b8cc0 (0xffff8800659b8cc0)
[ 26.106702] ffff8800646e7da0: 1ffff1000c8dcfbb (0x1ffff1000c8dcfbb)
[ 26.106705] ffff8800646e7da8: ffff8800646e7f20 (0xffff8800646e7f20)
[ 26.106712] ffff8800646e7db0: ffffffff83323428 (__sys_sendmsg+0x138/0x320)
[ 26.106715] ffff8800646e7db8: ffff8800646e7df8 (0xffff8800646e7df8)
[ 26.106719] ffff8800646e7dc0: 0000000041b58ab3 (0x41b58ab3)
[ 26.106722] ffff8800646e7dc8: 0000000020004fc8 (0x20004fc8)
[ 26.106726] ffff8800646e7dd0: ffffffff00000003 (0xffffffff00000003)
[ 26.106729] ffff8800646e7dd8: 0000000041b58ab3 (0x41b58ab3)
[ 26.106736] ffff8800646e7de0: ffffffff856a0598
(_fw_yam_9600_bin_name+0x1cc10/0x5d928)
[ 26.106742] ffff8800646e7de8: ffffffff833232f0 (SyS_shutdown+0x2f0/0x2f0)
[ 26.106745] ffff8800646e7df0: 0000000041b58ab3 (0x41b58ab3)
[ 26.106749] ffff8800646e7df8: ffffffff00000000 (0xffffffff00000000)
[ 26.106756] ffff8800646e7e00: ffffffff8128f720 (do_page_fault+0x30/0x30)
[ 26.106759] ffff8800646e7e08: ffff88006bc74a20 (0xffff88006bc74a20)
[ 26.106763] ffff8800646e7e10: ffff88006c03adc0 (0xffff88006c03adc0)
[ 26.106767] ffff8800646e7e18: ffff8800646e7e60 (0xffff8800646e7e60)
[ 26.106771] ffff8800646e7e20: ffff88006765c200 (0xffff88006765c200)
[ 26.106774] ffff8800646e7e28: 0000000000000003 (0x3)
[ 26.106777] ffff8800646e7e30: ffff8800649aca00 (0xffff8800649aca00)
[ 26.106781] ffff8800646e7e38: ffff8800fffffff7 (0xffff8800fffffff7)
[ 26.106784] ffff8800646e7e40: 0000000000000000 ...
[ 26.106787] ffff8800646e7e48: 0000000000000802 (0x802)
[ 26.106790] ffff8800646e7e50: 0000000041b58ab3 (0x41b58ab3)
[ 26.106796] ffff8800646e7e58: ffffffff855e2f80
(__func__.54982+0x233d80/0x2d46f0)
[ 26.106802] ffff8800646e7e60: ffffffff8140d720
(trace_raw_output_lock+0x190/0x190)
[ 26.106806] ffff8800646e7e68: ffff8800659b8cc0 (0xffff8800659b8cc0)
[ 26.106810] ffff8800646e7e70: ffff8800646e7e98 (0xffff8800646e7e98)
[ 26.106814] ffff8800646e7e78: ffff8800646e7ce0 (0xffff8800646e7ce0)
[ 26.106817] ffff8800646e7e80: ffff880000000002 (0xffff880000000002)
[ 26.106821] ffff8800646e7e88: 1ffff10000000001 (0x1ffff10000000001)
[ 26.106823] ffff8800646e7e90: 0000000000000000 ...
[ 26.106827] ffff8800646e7ea0: ffff8800646e7c40 (0xffff8800646e7c40)
[ 26.106829] ffff8800646e7ea8: 0000000000000000 ...
[ 26.106833] ffff8800646e7eb0: 0000000020005000 (0x20005000)
[ 26.106835] ffff8800646e7eb8: 0000000000000000 ...
[ 26.106838] ffff8800646e7ec0: 0000000000000040 (0x40)
[ 26.106840] ffff8800646e7ec8: 0000000000000000 ...
[ 26.106844] ffff8800646e7ed0: 0000000000400ff0 (0x400ff0)
[ 26.106847] ffff8800646e7ed8: 00007ffcbbc601b0 (0x7ffcbbc601b0)
[ 26.106850] ffff8800646e7ee0: 0000000000000000 ...
[ 26.106853] ffff8800646e7ef0: ffff8800646e7f48 (0xffff8800646e7f48)
[ 26.106856] ffff8800646e7ef8: 0000000000000000 ...
[ 26.106859] ffff8800646e7f00: 0000000000000003 (0x3)
[ 26.106864] ffff8800646e7f08: 0000000020004fc8 (0x20004fc8)
[ 26.106867] ffff8800646e7f10: 0000000000000000 ...
[ 26.106870] ffff8800646e7f20: ffff8800646e7f48 (0xffff8800646e7f48)
[ 26.106877] ffff8800646e7f28: ffffffff8332363d (SyS_sendmsg+0x2d/0x50)
[ 26.106879] ffff8800646e7f30: 0000000000000000 ...
[ 26.106882] ffff8800646e7f38: 0000000000400ff0 (0x400ff0)
[ 26.106887] ffff8800646e7f40: 00007ffcbbc601b0 (0x7ffcbbc601b0)
[ 26.106893] ffff8800646e7f48: 00007ffcbbc5ff30 (0x7ffcbbc5ff30)
[ 26.106903] ffff8800646e7f50: ffffffff84c2a881
(entry_SYSCALL_64_fastpath+0x1f/0xc2)
[ 26.106907] ffff8800646e7f58: 0000000000000000 ...
[ 26.106913] ffff8800646e7f68: 00007ffcbbc601b0 (0x7ffcbbc601b0)
[ 26.106917] ffff8800646e7f70: 0000000000000000 ...
[ 26.106923] ffff8800646e7f78: 0000000000400ff0 (0x400ff0)
[ 26.106930] ffff8800646e7f80: 00007ffcbbc601b0 (0x7ffcbbc601b0)
[ 26.106936] ffff8800646e7f88: 0000000000000206 (0x206)
[ 26.106940] ffff8800646e7f90: 0000000000000000 ...
[ 26.106946] ffff8800646e7fa8: ffffffffffffffda (0xffffffffffffffda)
[ 26.106953] ffff8800646e7fb0: 00007f6ca65f4b79 (0x7f6ca65f4b79)
[ 26.106957] ffff8800646e7fb8: 0000000000000000 ...
[ 26.106963] ffff8800646e7fc0: 0000000020004fc8 (0x20004fc8)
[ 26.106968] ffff8800646e7fc8: 0000000000000003 (0x3)
[ 26.106974] ffff8800646e7fd0: 000000000000002e (0x2e)
[ 26.106980] ffff8800646e7fd8: 00007f6ca65f4b79 (0x7f6ca65f4b79)
[ 26.106986] ffff8800646e7fe0: 0000000000000033 (0x33)
[ 26.106989] ffff8800646e7fe8: 0000000000000206 (0x206)
[ 26.106992] ffff8800646e7ff0: 00007ffcbbc5fee8 (0x7ffcbbc5fee8)
[ 26.106996] ffff8800646e7ff8: 000000000000002b (0x2b)

Thanks!

>
> --
> Josh
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To post to this group, send email to kasa...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20161220210144.u47znzx6qniecuvv%40treble.
> For more options, visit https://groups.google.com/d/optout.

[Josh Poimboeuf]

On Tue, Dec 20, 2016 at 11:35:23PM +0100, Andrey Konovalov wrote:
> On Tue, Dec 20, 2016 at 10:01 PM, Josh Poimboeuf <jpoi...@redhat.com> wrote:
> > On Tue, Dec 20, 2016 at 03:43:27PM +0100, Andrey Konovalov wrote:
> >> Hi,
> >>
> >> I've got the following warning while running the syzkaller fuzzer:
> >>
> >> WARNING: unrecognized kernel stack return address ffffffffa0000001 at
> >> ffff88006377fa18 in a.out:4467
> >>
> >> By adding a BUG() to unwind_get_return_address() I was able to capture
> >> the stack trace (see below). Looks like unwind_get_return_address()
> >> gets called when KASAN tries to unwind the stack to save the stack
> >> trace.
> >>
> >> A reproducer is attached. CONFIG_KASAN=y is most likely needed for it to work.
> >
> > Hi Andrey,
> >
> > I've tried with your reproducer but it didn't recreate. Can you try
> > again with the following patch from the tip tree, instead of your BUG()
> > patch?
> >
> > http://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/patch/?id=8b5e99f02264130782a10ba5c0c759797fb064ee
> >
> > That will dump the stack data, which should give more clues about what
> > went wrong.
>
> Hi Josh,
>
> Sure, here it is:

Thanks. Looking at the stack trace, my guess is that an interrupt hit
while running in generated BPF code, and the unwinder got confused
because regs->ip points to the generated code. I may need to disable
that warning until we figure out a better solution.

Can you share your .config file?

--
Josh

[Andrey Konovalov]

On Wed, Dec 21, 2016 at 12:36 AM, Josh Poimboeuf <jpoi...@redhat.com> wrote:
>
> Thanks. Looking at the stack trace, my guess is that an interrupt hit
> while running in generated BPF code, and the unwinder got confused
> because regs->ip points to the generated code. I may need to disable
> that warning until we figure out a better solution.
>
> Can you share your .config file?

Sure, attached.

>
> --
> Josh
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.

[Josh Poimboeuf]

On Wed, Dec 21, 2016 at 01:46:36PM +0100, Andrey Konovalov wrote:
> On Wed, Dec 21, 2016 at 12:36 AM, Josh Poimboeuf <jpoi...@redhat.com> wrote:
> >
> > Thanks. Looking at the stack trace, my guess is that an interrupt hit
> > while running in generated BPF code, and the unwinder got confused
> > because regs->ip points to the generated code. I may need to disable
> > that warning until we figure out a better solution.
> >
> > Can you share your .config file?
>
> Sure, attached.

Ok, I was able to recreate with your config. The culprit was generated
code, as I suspected, though it wasn't BPF, it was a kprobe (created by
dccpprobe_init()).

I'll make a patch to disable the warning.

--
Josh

[Dmitry Vyukov]

Hi,

I am also seeing the following warnings:

[ 281.889259] WARNING: kernel stack regs at ffff8801c29a7ea8 in
syz-executor8:1302 has bad 'bp' value ffff8801c29a7f28
[ 833.994878] WARNING: kernel stack regs at ffff8801c4e77ea8 in
syz-executor1:13094 has bad 'bp' value ffff8801c4e77f28

Can it also be caused by bpf/kprobe?

[Josh Poimboeuf]

This is a different warning. I suspect it's due to unwinding the stack
of another CPU while it's running, which is still possible in a few
places. I'm going to have to disable all these warnings for now.

--
Josh

[Dmitry Vyukov]

I also have the following diff locally. These loads trigger episodic
KASAN warnings about stack-of-bounds reads on rcu stall warnings when
it does backtrace of all cpus.
If it looks correct to you, can you please also incorporate it into your patch?


diff --git a/arch/x86/include/asm/stacktrace.h
b/arch/x86/include/asm/stacktrace.h
index a3269c897ec5..d8d4fc66ffec 100644
--- a/arch/x86/include/asm/stacktrace.h
+++ b/arch/x86/include/asm/stacktrace.h
@@ -58,7 +58,7 @@ get_frame_pointer(struct task_struct *task, struct
pt_regs *regs)
if (task == current)
return __builtin_frame_address(0);

- return (unsigned long *)((struct inactive_task_frame
*)task->thread.sp)->bp;
+ return (unsigned long *)READ_ONCE_NOCHECK(((struct
inactive_task_frame *)task->thread.sp)->bp);
}
#else
static inline unsigned long *
diff --git a/arch/x86/kernel/unwind_frame.c b/arch/x86/kernel/unwind_frame.c
index 4443e499f279..f3a225ffa231 100644
--- a/arch/x86/kernel/unwind_frame.c
+++ b/arch/x86/kernel/unwind_frame.c
@@ -162,7 +162,7 @@ bool unwind_next_frame(struct unwind_state *state)
if (state->regs)
next_bp = (unsigned long *)state->regs->bp;
else
- next_bp = (unsigned long *)*state->bp;
+ next_bp = (unsigned long *)READ_ONCE_NOCHECK(*state->bp);

/* is the next frame pointer an encoded pointer to pt_regs? */
regs = decode_frame_pointer(next_bp);

[Josh Poimboeuf]

Ok, will do.

BTW, I think there's an issue with your mail client. Your last two
replies to me didn't have me on To/Cc.

--
Josh

[Josh Poimboeuf]

Would you mind testing if the following patch fixes it? It's based on
the latest linus/master.


diff --git a/arch/x86/kernel/unwind_frame.c b/arch/x86/kernel/unwind_frame.c
index 4443e49..05adf9a 100644
--- a/arch/x86/kernel/unwind_frame.c
+++ b/arch/x86/kernel/unwind_frame.c
@@ -6,6 +6,21 @@

#define FRAME_HEADER_SIZE (sizeof(long) * 2)

+/*
+ * This disables KASAN checking when reading a value from another task's stack,
+ * since the other task could be running on another CPU and could have poisoned
+ * the stack in the meantime.
+ */
+#define UNWIND_READ_ONCE(state, x) \
+({ \
+ unsigned long val; \
+ if (state->task == current) \
+ val = READ_ONCE(x); \
+ else \
+ val = READ_ONCE_NOCHECK(x); \
+ val; \
+})
+
static void unwind_dump(struct unwind_state *state, unsigned long *sp)
{
static bool dumped_before = false;
@@ -48,7 +63,8 @@ unsigned long unwind_get_return_address(struct unwind_state *state)
if (state->regs && user_mode(state->regs))
return 0;

- addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, *addr_p,
+ addr = UNWIND_READ_ONCE(state, *addr_p);
+ addr = ftrace_graph_ret_addr(state->task, &state->graph_idx, addr,
addr_p);

return __kernel_text_address(addr) ? addr : 0;
@@ -162,7 +178,7 @@ bool unwind_next_frame(struct unwind_state *state)

if (state->regs)
next_bp = (unsigned long *)state->regs->bp;
else
- next_bp = (unsigned long *)*state->bp;

+ next_bp = (unsigned long *)UNWIND_READ_ONCE(state, *state->bp);

/* is the next frame pointer an encoded pointer to pt_regs? */
regs = decode_frame_pointer(next_bp);

@@ -207,6 +223,16 @@ bool unwind_next_frame(struct unwind_state *state)
return true;

bad_address:
+ /*
+ * When dumping a task other than current, the task might actually be
+ * running on another CPU, in which case it could be modifying its
+ * stack while we're reading it. This is generally not a problem and
+ * can be ignored as long as the caller understands that unwinding
+ * another task will not always succeed.
+ */
+ if (state->task != current)
+ goto the_end;
+
if (state->regs) {
printk_deferred_once(KERN_WARNING
"WARNING: kernel stack regs at %p in %s:%d has bad 'bp' value %p\n",

[Dmitry Vyukov]

Applied locally for testing.


What about this part?

diff --git a/arch/x86/include/asm/stacktrace.h
b/arch/x86/include/asm/stacktrace.h
index a3269c897ec5..d8d4fc66ffec 100644
--- a/arch/x86/include/asm/stacktrace.h
+++ b/arch/x86/include/asm/stacktrace.h
@@ -58,7 +58,7 @@ get_frame_pointer(struct task_struct *task, struct
pt_regs *regs)
if (task == current)
return __builtin_frame_address(0);

- return (unsigned long *)((struct

inactive_task_frame*)task->thread.sp)->bp;

+ return (unsigned long *)READ_ONCE_NOCHECK(((struct
inactive_task_frame *)task->thread.sp)->bp);
}
#else
static inline unsigned long *

I am hitting it as welL:

[< inline >] kasan_report_error mm/kasan/report.c:213
[< none >] kasan_report+0x42d/0x460 mm/kasan/report.c:307
[< none >] __asan_report_load8_noabort+0x14/0x20
mm/kasan/report.c:333
[< inline >] get_frame_pointer ./arch/x86/include/asm/stacktrace.h:61
[< none >] __unwind_start+0x38d/0x3a0
arch/x86/kernel/unwind_frame.c:247
[< inline >] unwind_start ./arch/x86/include/asm/unwind.h:40
[< none >] show_trace_log_lvl+0x1ed/0x2c0
arch/x86/kernel/dumpstack.c:61
[< none >] show_stack+0x2f/0x40 arch/x86/kernel/dumpstack.c:174
[< none >] sched_show_task+0x3e4/0x540 kernel/sched/core.c:5217

[Josh Poimboeuf]

Oops, I missed that part. That's needed too.

BTW, I'm still not on your email To: list.

--
Josh