HW-KASAN and CONFIG_SLUB_DEBUG_ON=y screams about redzone corruption

[Will Deacon]

Hi memory tagging folks,

While debugging something else, I ended up running v6.4 on an arm64 (v9)
fastmodel with both CONFIG_SLUB_DEBUG_ON=y and CONFIG_KASAN_HW_TAGS=y.
This makes the system pretty unusable, as I see a tonne of kmalloc
Redzone corruption messages pretty much straight out of startup (example
below).

Please can you take a look?

Cheers,

Will

--->8

[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.000000] =============================================================================
[ 0.000000] BUG kmalloc-128 (Not tainted): kmalloc Redzone overwritten
[ 0.000000] -----------------------------------------------------------------------------
[ 0.000000]
[ 0.000000] 0xffff00080001a9b0-0xf1ff00080001a9ff @offset=2480. First byte 0x0 instead of 0xcc
[ 0.000000] Allocated in apply_wqattrs_prepare+0x90/0x2a4 age=0 cpu=0 pid=0
[ 0.000000] kmalloc_trace+0x34/0x6c
[ 0.000000] apply_wqattrs_prepare+0x90/0x2a4
[ 0.000000] apply_workqueue_attrs+0x5c/0xb4
[ 0.000000] alloc_workqueue+0x368/0x4f8
[ 0.000000] workqueue_init_early+0x2e8/0x3ac
[ 0.000000] start_kernel+0x168/0x394
[ 0.000000] __primary_switched+0xbc/0xc4
[ 0.000000] Slab 0xfffffc0020000680 objects=21 used=8 fp=0xffff00080001ac80 flags=0xbfffc0000010200(slab|head|node=0|zone=2|lastcpupid=0xffff|kasantag=0x0)
[ 0.000000] Object 0xf1ff00080001a980 @offset=17437937757178562944 fp=0x0000000000000000
[ 0.000000]
[ 0.000000] Redzone ffff00080001a900: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a910: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a920: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a930: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a940: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a950: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a960: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Redzone ffff00080001a970: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................
[ 0.000000] Object ffff00080001a980: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Object ffff00080001a9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 0.000000] Redzone ffff00080001aa00: cc cc cc cc cc cc cc cc ........
[ 0.000000] Padding ffff00080001aa54: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 0.000000] Padding ffff00080001aa64: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[ 0.000000] Padding ffff00080001aa74: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0-00001-g56e11237836c #1
[ 0.000000] Hardware name: FVP Base RevC (DT)
[ 0.000000] Call trace:
[ 0.000000] dump_backtrace+0xec/0x108
[ 0.000000] show_stack+0x18/0x2c
[ 0.000000] dump_stack_lvl+0x50/0x68
[ 0.000000] dump_stack+0x18/0x24
[ 0.000000] print_trailer+0x1ec/0x230
[ 0.000000] check_bytes_and_report+0x110/0x154
[ 0.000000] check_object+0x31c/0x360
[ 0.000000] free_to_partial_list+0x174/0x5d8
[ 0.000000] __slab_free+0x220/0x28c
[ 0.000000] __kmem_cache_free+0x364/0x3dc
[ 0.000000] kfree+0x50/0x70
[ 0.000000] apply_wqattrs_prepare+0x244/0x2a4
[ 0.000000] apply_workqueue_attrs+0x5c/0xb4
[ 0.000000] alloc_workqueue+0x368/0x4f8
[ 0.000000] workqueue_init_early+0x2e8/0x3ac
[ 0.000000] start_kernel+0x168/0x394
[ 0.000000] __primary_switched+0xbc/0xc4
[ 0.000000] Disabling lock debugging due to kernel taint
[ 0.000000] FIX kmalloc-128: Restoring kmalloc Redzone 0xffff00080001a9b0-0xf1ff00080001a9ff=0xcc
[ 0.000000] FIX kmalloc-128: Object at 0xf1ff00080001a980 not freed

[Alexander Potapenko]

On Wed, Jun 28, 2023 at 5:47 PM Will Deacon <wi...@kernel.org> wrote:
>
> Hi memory tagging folks,
>
> While debugging something else, I ended up running v6.4 on an arm64 (v9)
> fastmodel with both CONFIG_SLUB_DEBUG_ON=y and CONFIG_KASAN_HW_TAGS=y.
> This makes the system pretty unusable, as I see a tonne of kmalloc
> Redzone corruption messages pretty much straight out of startup (example
> below).
>
> Please can you take a look?
>
> Cheers,

Does the problem reproduce with CONFIG_KASAN_SW_TAGS?
Also, any chance you could share the file:line info for the stack trace below?

I myself haven't expected KASAN to work together with SLUB_DEBUG...

> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20230628154714.GB22090%40willie-the-truck.



--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Andrey Konovalov]

On Tue, Jul 4, 2023 at 9:42 AM Alexander Potapenko <gli...@google.com> wrote:
>
> > While debugging something else, I ended up running v6.4 on an arm64 (v9)
> > fastmodel with both CONFIG_SLUB_DEBUG_ON=y and CONFIG_KASAN_HW_TAGS=y.
> > This makes the system pretty unusable, as I see a tonne of kmalloc
> > Redzone corruption messages pretty much straight out of startup (example
> > below).

I've reproduced the issue, looking into the root cause.

> Does the problem reproduce with CONFIG_KASAN_SW_TAGS?

Looks like SW_TAGS is not affected.

> Also, any chance you could share the file:line info for the stack trace below?
>
> I myself haven't expected KASAN to work together with SLUB_DEBUG...

This was implemented at some point, so it should work.

Thanks!