KASAN use-after-free not showing freed stacktrace?

[Vegard Nossum]

Hi again,

I am seeing some KASAN use-after-free bugs now but there is no
stacktrace for where they were freed anymore:

BUG: KASAN: use-after-free in acct_collect+0x7d5/0x830 at addr
ffff88010e129b08
Read of size 8 by task trinity-c0/13609
CPU: 0 PID: 13609 Comm: trinity-c0 Not tainted 4.7.0+ #65
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
Ubuntu-1.8.2-1ubuntu1 04/01/2014
ffff88010e129b00 ffff88011482f8f0 ffffffff81d701c1 ffff88011482f980
ffff88010d4d5c00 ffff88011482f970 ffffffff81477d5e 0000000000000001
0000000000000000 0000000000000296 0000000000000246 ffffffff8126347d
Call Trace:
[<ffffffff81d701c1>] dump_stack+0x65/0x84
[<ffffffff81477d5e>] kasan_report_error+0x22e/0x5e0
[<ffffffff8126347d>] ? acct_collect+0x12d/0x830
[<ffffffff8147824e>] __asan_report_load8_noabort+0x3e/0x40
[<ffffffff81263b25>] ? acct_collect+0x7d5/0x830
[<ffffffff81263b25>] acct_collect+0x7d5/0x830
[<ffffffff81263350>] ? acct_exit_ns+0x70/0x70
[<ffffffff812c9ba0>] ? xacct_add_tsk+0x670/0x670
[<ffffffff81231b80>] ? hrtimer_active+0x340/0x340
[<ffffffff8112bf40>] ? get_signal+0x1120/0x1120
[<ffffffff8115d1e1>] ? creds_are_invalid.part.1+0x11/0xb0
[<ffffffff8115f5f2>] ? __validate_process_creds+0x242/0x3e0
[<ffffffff81109421>] do_exit+0xca1/0x2c90
[<ffffffff81367984>] ? ___perf_sw_event+0x284/0x330
[<ffffffff813677f4>] ? ___perf_sw_event+0xf4/0x330
[<ffffffff81367700>] ? perf_swevent_put_recursion_context+0x90/0x90
[<ffffffff81108780>] ? mm_update_next_owner+0x720/0x720
[<ffffffff8105a026>] ? print_context_stack+0x76/0xe0
[<ffffffff8112afc2>] ? get_signal+0x1a2/0x1120
[<ffffffff8110b544>] do_group_exit+0xf4/0x2f0
[<ffffffff8112b35d>] get_signal+0x53d/0x1120
[<ffffffff811e21f2>] ? __lock_acquire.isra.32+0xc2/0x1a30
[<ffffffff81051673>] do_signal+0x83/0x1f10
[<ffffffff81dcf247>] ? debug_smp_processor_id+0x17/0x20
[<ffffffff810515f0>] ? setup_sigcontext+0x7d0/0x7d0
[<ffffffff810ce68b>] ? __do_page_fault+0x53b/0x8f0
[<ffffffff8134dcc7>] ? perf_iterate_sb+0x97/0x6d0
[<ffffffff810cec7d>] ? trace_do_page_fault+0x18d/0x310
[<ffffffff81308d40>] ? ftrace_syscall_exit+0x550/0x550
[<ffffffff838a1258>] ? async_page_fault+0x28/0x30
[<ffffffff81002aa2>] exit_to_usermode_loop+0xa2/0x120
[<ffffffff81005224>] syscall_return_slowpath+0x144/0x170
[<ffffffff8389f56f>] ret_from_fork+0x2f/0x40
Object at ffff88010e129b00, in cache vm_area_struct
Object allocated with size 192 bytes.
Allocation:
PID = 1334
[<ffffffff81077ed6>] save_stack_trace+0x26/0x50
[<ffffffff814769d6>] save_stack+0x46/0xd0
[<ffffffff814771ca>] kasan_kmalloc+0xda/0x100
[<ffffffff81477202>] kasan_slab_alloc+0x12/0x20
[<ffffffff81472909>] kmem_cache_alloc+0xe9/0x290
[<ffffffff810f7e57>] copy_process.part.39+0x1e07/0x5390
[<ffffffff810fb87a>] _do_fork+0x17a/0xa70
[<ffffffff810fc1f4>] SyS_clone+0x14/0x20
[<ffffffff810053f1>] do_syscall_64+0x1a1/0x460
[<ffffffff8389f3ea>] return_from_SYSCALL_64+0x0/0x6a
Memory state around the buggy address:
ffff88010e129a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff88010e129a80: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
>ffff88010e129b00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff88010e129b80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
ffff88010e129c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================
Disabling lock debugging due to kernel taint
==================================================================

That seems like a regression, maybe related to memory quarantine
for SLUB? Or is there something else going on?


Vegard

[Dmitry Vyukov]

Do you use SLAB or SLUB? Is CONFIG_STACKDEPOT enabled? Kernel revision?

> --
> You received this message because you are subscribed to the Google Groups
> "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kasan-dev+...@googlegroups.com.
> To post to this group, send email to kasa...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kasan-dev/579BB9F0.8080700%40oracle.com.
> For more options, visit https://groups.google.com/d/optout.

[Vegard Nossum]

On 07/29/2016 11:27 PM, Dmitry Vyukov wrote:
> On Fri, Jul 29, 2016 at 10:17 PM, Vegard Nossum
> <vegard...@oracle.com> wrote:
>> Hi again,
>>
>> I am seeing some KASAN use-after-free bugs now but there is no
>> stacktrace for where they were freed anymore:

[...]

>> That seems like a regression, maybe related to memory quarantine
>> for SLUB? Or is there something else going on?

> Do you use SLAB or SLUB? Is CONFIG_STACKDEPOT enabled? Kernel revision?

CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_SLABINFO=y
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_FAILSLAB=y

CONFIG_STACKDEPOT=y

git version c624c86615fb8aa61fa76ed8c935446d06c80e77


Vegard

[Alexander Potapenko]

I'm seeing problems like this with syzkaller from time to time, but I never managed to reproduce those. Do you have a reliable reproducer?

Please note that according to the report the allocated chunk is in the KASAN_STATE_ALLOC state (see kasan_object_err() in mm/kasan/report.c)

On the other hand, the shadow memory is in the KASAN_KMALLOC_FREE state, because the bug has been classified as a use-after-free.

 

That said, for some reason the shadow memory value and the chunk state are inconsistent.

Since we're on x86, it's quite unlikely that the assignment to the chunk state has been reordered with the assignment to the shadow byte.

Dmitry's guess is that in kasan_slab_free(), between quarantine_put() and kasan_poison_slab_free(), the quarantine was drained, and another thread took our memory chunk.

This sounds really probable, as quarantine isn't really that big (1/32 of the available physical memory, minus 1Mb for each CPU).

Vegard