[syzbot] [arm?] upstream test error: KASAN: invalid-access Write in setup_arch

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 33faa93bc856 Merge branch kvmarm-master/next into kvmarm-m..
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fuzzme
console output: https://syzkaller.appspot.com/x/log.txt?x=1398420b980000
kernel config: https://syzkaller.appspot.com/x/.config?x=2b7b31c9aa1397ca
dashboard link: https://syzkaller.appspot.com/bug?extid=908886656a02769af987
compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/384ffdcca292/non_bootable_disk-33faa93b.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/9093742fcee9/vmlinux-33faa93b.xz
kernel image: https://storage.googleapis.com/syzbot-assets/b1f599907931/Image-33faa93b.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+908886...@syzkaller.appspotmail.com

Booting Linux on physical CPU 0x0000000000 [0x000f0510]
Linux version 6.11.0-rc5-syzkaller-g33faa93bc856 (syzkaller@syzkaller) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #0 SMP PREEMPT now
random: crng init done
Machine model: linux,dummy-virt
efi: UEFI not found.
NUMA: No NUMA configuration found
NUMA: Faking a node at [mem 0x0000000040000000-0x00000000bfffffff]
NUMA: NODE_DATA [mem 0xbfc1d340-0xbfc20fff]
Zone ranges:
DMA [mem 0x0000000040000000-0x00000000bfffffff]
DMA32 empty
Normal empty
Device empty
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000040000000-0x00000000bfffffff]
Initmem setup node 0 [mem 0x0000000040000000-0x00000000bfffffff]
cma: Reserved 32 MiB at 0x00000000bba00000 on node -1
psci: probing for conduit method from DT.
psci: PSCIv1.1 detected in firmware.
psci: Using standard PSCI v0.2 function IDs
psci: Trusted OS migration not required
psci: SMC Calling Convention v1.0
==================================================================
BUG: KASAN: invalid-access in smp_build_mpidr_hash arch/arm64/kernel/setup.c:133 [inline]
BUG: KASAN: invalid-access in setup_arch+0x984/0xd60 arch/arm64/kernel/setup.c:356
Write of size 4 at addr 03ff800086867e00 by task swapper/0
Pointer tag: [03], memory tag: [fe]

CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.11.0-rc5-syzkaller-g33faa93bc856 #0
Hardware name: linux,dummy-virt (DT)
Call trace:
dump_backtrace+0x204/0x3b8 arch/arm64/kernel/stacktrace.c:317
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:324
__dump_stack lib/dump_stack.c:93 [inline]
dump_stack_lvl+0x260/0x3b4 lib/dump_stack.c:119
print_address_description mm/kasan/report.c:377 [inline]
print_report+0x118/0x5ac mm/kasan/report.c:488
kasan_report+0xc8/0x108 mm/kasan/report.c:601
kasan_check_range+0x94/0xb8 mm/kasan/sw_tags.c:84
__hwasan_store4_noabort+0x20/0x2c mm/kasan/sw_tags.c:149
smp_build_mpidr_hash arch/arm64/kernel/setup.c:133 [inline]
setup_arch+0x984/0xd60 arch/arm64/kernel/setup.c:356
start_kernel+0xe0/0xff0 init/main.c:926
__primary_switched+0x84/0x8c arch/arm64/kernel/head.S:243

The buggy address belongs to stack of task swapper/0

Memory state around the buggy address:
ffff800086867c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff800086867d00: 00 fe fe 00 00 00 fe fe fe fe fe fe fe fe fe fe
>ffff800086867e00: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
^
ffff800086867f00: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
ffff800086868000: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
==================================================================
percpu: Embedded 35 pages/cpu s104840 r8192 d30328 u143360
Detected PIPT I-cache on CPU0
CPU features: detected: GIC system register CPU interface
CPU features: detected: HCRX_EL2 register
CPU features: detected: 52-bit Virtual Addressing (LPA2)
CPU features: detected: Virtualization Host Extensions
CPU features: detected: Spectre-v4
alternatives: applying boot alternatives
kasan: KernelAddressSanitizer initialized (sw-tags, stacktrace=on)
Kernel command line: root=/dev/vda console=ttyAMA0
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
Fallback order for Node 0: 0
Built 1 zonelists, mobility grouping on. Total pages: 524288
Policy zone: DMA
mem auto-init: stack:all(zero), heap alloc:on, heap free:off
stackdepot: allocating hash table via alloc_large_system_hash
stackdepot hash table entries: 1048576 (order: 12, 16777216 bytes, linear)
software IO TLB: SWIOTLB bounce buffer size adjusted to 2MB
software IO TLB: area num 1.
software IO TLB: mapped [mem 0x00000000b1a29000-0x00000000b1c29000] (2MB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
allocated 4194304 bytes of page_ext
trace event string verifier disabled
Running RCU self tests
Running RCU synchronous self tests
rcu: Preemptible hierarchical RCU implementation.
rcu: RCU lockdep checking is enabled.
rcu: RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
rcu: RCU callback double-/use-after-free debug is enabled.
rcu: RCU debug extended QS entry/exit.
Trampoline variant of Tasks RCU enabled.
Tracing variant of Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
Running RCU synchronous self tests
RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
GICv3: GIC: Using split EOI/Deactivate mode
GICv3: 256 SPIs implemented
GICv3: 0 Extended SPIs implemented
Root IRQ handler: gic_handle_irq
GICv3: GICv3 features: 16 PPIs
GICv3: GICv4 features:
GICv3: GICD_CTRL.DS=1, SCR_EL3.FIQ=0
GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000
ITS [mem 0x08080000-0x0809ffff]
ITS@0x0000000008080000: Single VMOVP capable
ITS@0x0000000008080000: allocated 8192 Devices @4a230000 (indirect, esz 8, psz 64K, shr 1)
ITS@0x0000000008080000: allocated 8192 Interrupt Collections @4a240000 (flat, esz 8, psz 64K, shr 1)
ITS@0x0000000008080000: allocated 8192 Virtual CPUs @4a250000 (indirect, esz 8, psz 64K, shr 1)
GICv3: using LPI property table @0x000000004a260000
ITS: Allocated DevID ffff as GICv4 proxy device (2 slots)
ITS: Enabling GICv4 support
GICv3: CPU0: using allocated LPI pending table @0x000000004a270000
rcu: srcu_init: Setting srcu_struct sizes based on contention.
arch_timer: cp15 timer(s) running at 62.50MHz (phys).
clocksource: arch_sys_counter: mask: 0x1ffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
sched_clock: 57 bits at 63MHz, resolution 16ns, wraps every 4398046511096ns
Console: colour dummy device 80x25
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES: 8
... MAX_LOCK_DEPTH: 48
... MAX_LOCKDEP_KEYS: 8192
... CLASSHASH_SIZE: 4096
... MAX_LOCKDEP_ENTRIES: 131072
... MAX_LOCKDEP_CHAINS: 65536
... CHAINHASH_SIZE: 32768
memory used by lock dependency info: 11817 kB
memory used for stack traces: 8320 kB
per task-struct memory footprint: 1920 bytes
Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=625000)
pid_max: default: 32768 minimum: 301
LSM: initializing lsm=lockdown,capability,landlock,yama,safesetid,tomoyo,selinux,ima,evm
landlock: Up and running.
Yama: becoming mindful.
TOMOYO Linux initialized
SELinux: Initializing.
Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
Running RCU synchronous self tests
Running RCU synchronous self tests


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Will Deacon]

On Fri, Aug 30, 2024 at 01:35:24AM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 33faa93bc856 Merge branch kvmarm-master/next into kvmarm-m..
> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fuzzme

+Marc, as this is his branch.

I can't spot the issue here. We have a couple of fixed-length
(4 element) arrays on the stack and they're indexed by a simple loop
counter that runs from 0-3.

Will

[Marc Zyngier]

On Fri, 30 Aug 2024 10:52:54 +0100,
Will Deacon <wi...@kernel.org> wrote:
>
> On Fri, Aug 30, 2024 at 01:35:24AM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: 33faa93bc856 Merge branch kvmarm-master/next into kvmarm-m..
> > git tree: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fuzzme
>
> +Marc, as this is his branch.
>
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1398420b980000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=2b7b31c9aa1397ca
> > dashboard link: https://syzkaller.appspot.com/bug?extid=908886656a02769af987
> > compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> > userspace arch: arm64

As it turns out, this isn't specific to this branch. I can reproduce
it with this config on a vanilla 6.10 as a KVM guest. Even worse,
compiling with clang results in an unbootable kernel (without any
output at all).

Mind you, the binary is absolutely massive (130MB with gcc, 156MB with
clang), and I wouldn't be surprised if we were hitting some kind of
odd limit.

Having trimmed the config to the extreme, I can only trigger the
warning with CONFIG_KASAN_SW_TAGS (CONFIG_KASAN_GENERIC does not
scream). Same thing if I use gcc 14.2.0.

However, compiling with clang 14 (Debian clang version 14.0.6) does
*not* result in a screaming kernel, even with KASAN_SW_TAGS.

So I can see two possibilities here:

- either gcc is incompatible with KASAN_SW_TAGS and the generic
version is the only one that works

- or we have a compiler bug on our hands.

Frankly, I can't believe the later, as the code is so daft that I
can't imagine gcc getting it *that* wrong.

Who knows enough about KASAN to dig into this?

M.

--
Without deviation from the norm, progress is not possible.

[Aleksandr Nogikh]

+kasan-dev

[Alexander Potapenko]

This looks related to Samuel's "arm64: Fix KASAN random tag seed
initialization" patch that landed in August.

I am a bit surprised the bug is reported before the
"KernelAddressSanitizer initialized" banner is printed - I thought we
shouldn't be reporting anything until the tool is fully initialized.

[Marc Zyngier]

f75c235565f9 arm64: Fix KASAN random tag seed initialization

$ git describe --contains f75c235565f9 --match=v\*
v6.11-rc4~15^2

So while this is in -rc4, -rc6 still has the same issue (with GCC --
clang is OK).

> I am a bit surprised the bug is reported before the
> "KernelAddressSanitizer initialized" banner is printed - I thought we
> shouldn't be reporting anything until the tool is fully initialized.

Specially if this can report false positives...

Thanks,

[Samuel Holland]

I wouldn't expect it to be related to my patch. smp_build_mpidr_hash() gets
called before kasan_init_sw_tags() both before and after applying my patch.

Since the variable in question is a stack variable, the random tag is generated
by GCC, not the kernel function.

Since smp_build_mpidr_hash() is inlined into setup_arch(), which also calls
kasan_init(), maybe the issue is that GCC tries to allocate the local variable
and write the tag to shadow memory before kasan_init() actually sets up the
shadow memory?

Regards,
Samuel

[Alexander Potapenko]

> >>>> Who knows enough about KASAN to dig into this?
> >>
> >> This looks related to Samuel's "arm64: Fix KASAN random tag seed
> >> initialization" patch that landed in August.
> >
> > f75c235565f9 arm64: Fix KASAN random tag seed initialization
> >
> > $ git describe --contains f75c235565f9 --match=v\*
> > v6.11-rc4~15^2
> >
> > So while this is in -rc4, -rc6 still has the same issue (with GCC --
> > clang is OK).
>
> I wouldn't expect it to be related to my patch. smp_build_mpidr_hash() gets
> called before kasan_init_sw_tags() both before and after applying my patch.

Hm, you are right, this problem indeed dates back to v6.9 or earlier.

> Since the variable in question is a stack variable, the random tag is generated
> by GCC, not the kernel function.
>
> Since smp_build_mpidr_hash() is inlined into setup_arch(), which also calls
> kasan_init(), maybe the issue is that GCC tries to allocate the local variable
> and write the tag to shadow memory before kasan_init() actually sets up the
> shadow memory?

Should it be inlined at all?
setup_arch() is a __no_sanitize_address function, and
smp_build_mpidr_hash() is an instrumented one.
The latter is not supposed to be inlined into the former, unless the
latter is always_inline
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67368,
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89124).

The report seems to go away if I mark smp_build_mpidr_hash() as noinline.
This doesn't explain, though, why Clang build doesn't work at all...

>
> Regards,
> Samuel
>
> >> I am a bit surprised the bug is reported before the
> >> "KernelAddressSanitizer initialized" banner is printed - I thought we
> >> shouldn't be reporting anything until the tool is fully initialized.
> >
> > Specially if this can report false positives...
> >
> > Thanks,
> >
> > M.
> >
>

--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Mark Rutland]

Likewise.

> However, compiling with clang 14 (Debian clang version 14.0.6) does
> *not* result in a screaming kernel, even with KASAN_SW_TAGS.
>
> So I can see two possibilities here:
>
> - either gcc is incompatible with KASAN_SW_TAGS and the generic
> version is the only one that works
>
> - or we have a compiler bug on our hands.

Looking at this there seem to be a bunch of problems here. I'll try to
write this up better tomorrow, but as a holding reply for now, there's
definitely a compiler issue and mahybe a kernel issue.

Mark.

[Mark Rutland]

[adding Ard and LLVM folk; there's a question right at the end after
some context]

On Sat, Aug 31, 2024 at 06:52:52PM +0100, Marc Zyngier wrote:

> On Fri, 30 Aug 2024 10:52:54 +0100,
> Will Deacon <wi...@kernel.org> wrote:
> >
> > On Fri, Aug 30, 2024 at 01:35:24AM -0700, syzbot wrote:
> > > Hello,
> > >
> > > syzbot found the following issue on:
> > >
> > > HEAD commit: 33faa93bc856 Merge branch kvmarm-master/next into kvmarm-m..
> > > git tree: git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git fuzzme
> >
> > +Marc, as this is his branch.
> >
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=1398420b980000
> > > kernel config: https://syzkaller.appspot.com/x/.config?x=2b7b31c9aa1397ca
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=908886656a02769af987
> > > compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> > > userspace arch: arm64
>
> As it turns out, this isn't specific to this branch. I can reproduce
> it with this config on a vanilla 6.10 as a KVM guest. Even worse,
> compiling with clang results in an unbootable kernel (without any
> output at all).
>
> Mind you, the binary is absolutely massive (130MB with gcc, 156MB with
> clang), and I wouldn't be surprised if we were hitting some kind of
> odd limit.

Putting the KASAN issue aside (which I'll handle in a separate thread),
I think there is a real issue here with LLVM.

What's going on here is that .idmap.text ends up more than 128M away
from .head.text, so the 'b primary_entry' at the start of the Image
isn't in range:

| [mark@lakrids:~/src/linux]% usekorg 14.1.0 aarch64-linux-objdump -t vmlinux | grep -w _text
| ffff800080000000 g .head.text 0000000000000000 _text
| [mark@lakrids:~/src/linux]% usekorg 14.1.0 aarch64-linux-objdump -t vmlinux | grep -w primary_entry
| ffff8000889df0e0 g .rodata.text 000000000000006c primary_entry

... as those are ~128MiB apart.

When building with GCC those end up ~101MiB apart:

| [mark@lakrids:~/src/linux]% usekorg 14.1.0 aarch64-linux-objdump -t vmlinux | grep -w _text
| ffff800080000000 g .head.text 0000000000000000 _text
| [mark@lakrids:~/src/linux]% usekorg 14.1.0 aarch64-linux-objdump -t vmlinux | grep -w primary_entry
| ffff8000865ae0e0 g .rodata.text 000000000000006c primary_entry

When that happens, LLD makes the header branch to a veneer/thunk:

| ffff800080000000 <_text>:
| ffff800080000000: fa405a4d ccmp x18, #0x0, #0xd, pl // pl = nfrst
| ffff800080000004: 14003fff b ffff800080010000 <__AArch64AbsLongThunk_primary_entry>

... and unfortunately, that veneer/thunk uses a literal with the
statically-linked TTBR1 address of primary_entry:

| ffff800080010000 <__AArch64AbsLongThunk_primary_entry>:
| ffff800080010000: 58000050 ldr x16, ffff800080010008 <__AArch64AbsLongThunk_primary_entry+0x8>
| ffff800080010004: d61f0200 br x16
| ffff800080010008: 889df0e0 .word 0x889df0e0
| ffff80008001000c: ffff8000 .word 0xffff8000

... so as soon as the CPU tries to branch there it'll take a synchronous
exception since either:

(a) The MMU is off, and that's larger than the physical address size.

(b) The MMU is on, but there's no TTBR1 mapping.

We can bodge around this instance by manually open-coding a veneer with
ADRP+ADD+BR after the header, and having the header branch to that, but
AFAICT we have no guarantee that other early asm or PI C code won't hit
the same problem.

It'd be good if we could convince LLD to use ADRP+ADD, since we already
rely on the entire kernel image falling within 2GiB for data
relocations. I'm not sure if it doesn't support using ADRP+ADD in
veneers or if we're doing something that prevents it from using ADRP+ADD
in the veneer.

By comparison, if I force the branch range to be longer, GCC 14.1.0 and
GNU LD 2.4.20 use ADRP+ADD for the veneer, and the resulting kernel
boots successfully.

I tested that by hacking some .rodata between .head.text and .idmap.text
with:

| char hack_force_veneer[SZ_128M] __ro_after_init;

... which forces a ~230MiB branch range using the config above:

| [mark@lakrids:~/src/linux]% usekorg 14.1.0 aarch64-linux-objdump -t vmlinux | grep -w _text
| ffff800080000000 g .head.text 0000000000000000 _text
| [mark@lakrids:~/src/linux]% usekorg 14.1.0 aarch64-linux-objdump -t vmlinux | grep -w primary_entry
| ffff80008e5be0e0 g .rodata.text 000000000000006c primary_entry

... with the generated code being:

| ffff800080000000 <__efistub__text>:
| ffff800080000000: fa405a4d ccmp x18, #0x0, #0xd, pl // pl = nfrst
| ffff800080000004: 14004001 b ffff800080010008 <__primary_entry_veneer>
...
| ffff800080010008 <__primary_entry_veneer>:
| ffff800080010008: d0072d70 adrp x16, ffff80008e5be000 <__idmap_text_start>
| ffff80008001000c: 91038210 add x16, x16, #0xe0
| ffff800080010010: d61f0200 br x16

LLVM folk, is there any existing option to ask LLD to use ADRP+ADD for
the veneer/thunk? ... and if not, would it be possible to add an option
for that?

I realise it shouldn't matter for most users, but it'd be nice to avoid
the boobytrap for anyone building test kernels.

Mark.

[Ard Biesheuvel]

> LLVM folk, is there any existing option to ask LLD to use ADRP+ADD for
> the veneer/thunk? ... and if not, would it be possible to add an option
> for that?
>

ld.lld takes --pic-veneer, which (from looking at the llvm sources)
appears to do what we need here.

[Mark Rutland]

Ah; now I take another look I see that's in the man page and is also
supported by GNU LD, so I'll spin a patch to use that.

Thanks for the pointer!

Mark.

[Mark Rutland]

[adding KASAN folk]

There appears to be a GCC bug here, analysis below.

The issues with clang are unrelated, and I will follow up with a
separate mail for those.

On Sat, Aug 31, 2024 at 06:52:52PM +0100, Marc Zyngier wrote:

It looks like what's happening here is:

(1) With CONFIG_KASAN_SW_TAGS=y we pass the compiler
`-fsanitize=kernel-hwaddress`.

(2) When GCC is passed `-fsanitize=hwaddress` or
`-fsanitize=kernel-hwaddress` it ignores
`__attribute__((no_sanitize_address))`, and instruments functions we
require are not instrumented.

I believe this is a compiler bug, as there doesn't seem to be a
separate attribute to prevent instrumentation in this mode.

(3) In this config, smp_build_mpidr_hash() gets inlined into
setup_arch(), and as setup_arch() is instrumented, all of the stack
variables for smp_build_mpidr_hash() are initialized at the start of
setup_arch(), with calls to __hwasan_tag_memory().

At this point, we are using the early shadow (where a single page of
shadow is used for all memory).

(4) In setup_arch(), we call kasan_init() to transition from the early
shadow to the runtime shadow. This replaces the early shadow memory
with new shadow memory initialized to KASAN_SHADOW_INIT (0xFE AKA
KASAN_TAG_INVALID), including the shadow for the stack.

(5) Once the CPU returns back into setup_arch(), it's using the new
shadow initialized to 0xFE. Subsequent stack accesses which check
the shadow see 0xFE in the shadow, and fault. Note that in the dump
of the shadow above, the shadow around ffff800086867d80 and above is
all 0xFE, while below that functions have managed to clear the
shadow.

Compiler test case below. Note that this demonstrates the compiler
ignores `__attribute__((no_sanitize_address))` regardless of
KASAN_STACK, so KASAN_SW_TAGS is generally broken with GCC. All versions
I tried were broken, from 11.3.0 to 14.2.0 inclusive.

I think we have to disable KASAN_SW_TAGS with GCC until this is fixed.

| [mark@lakrids:/mnt/data/tests/kasan-tags]% cat test.c
| #define __nsa __attribute__((no_sanitize_address))
|
| long __nsa load_long(long *ptr)
| {
| return *ptr;
| }
|
| void __nsa store_long(long *ptr, long val)
| {
| *ptr = val;
| }
|
| void extern_func(void);
|
| long __nsa stack_func(void)
| {
| volatile long val = 0;
| extern_func();
| return val;
| }
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 12.1.0 aarch64-linux-gcc -c test.c -O2 -fsanitize=kernel-hwaddress
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 14.2.0 aarch64-linux-objdump -d test.o
|
| test.o: file format elf64-littleaarch64
|
|
| Disassembly of section .text:
|
| 0000000000000000 <load_long>:
| 0: a9be7bfd stp x29, x30, [sp, #-32]!
| 4: 910003fd mov x29, sp
| 8: f9000bf3 str x19, [sp, #16]
| c: aa0003f3 mov x19, x0
| 10: 94000000 bl 0 <__hwasan_load8_noabort>
| 14: f9400260 ldr x0, [x19]
| 18: f9400bf3 ldr x19, [sp, #16]
| 1c: a8c27bfd ldp x29, x30, [sp], #32
| 20: d65f03c0 ret
|
| 0000000000000024 <store_long>:
| 24: a9be7bfd stp x29, x30, [sp, #-32]!
| 28: 910003fd mov x29, sp
| 2c: a90153f3 stp x19, x20, [sp, #16]
| 30: aa0003f3 mov x19, x0
| 34: aa0103f4 mov x20, x1
| 38: 94000000 bl 0 <__hwasan_store8_noabort>
| 3c: f9000274 str x20, [x19]
| 40: a94153f3 ldp x19, x20, [sp, #16]
| 44: a8c27bfd ldp x29, x30, [sp], #32
| 48: d65f03c0 ret
| 4c: d503201f nop
|
| 0000000000000050 <stack_func>:
| 50: a9be7bfd stp x29, x30, [sp, #-32]!
| 54: 910003fd mov x29, sp
| 58: f9000fff str xzr, [sp, #24]
| 5c: 94000000 bl 0 <extern_func>
| 60: f9400fe0 ldr x0, [sp, #24]
| 64: a8c27bfd ldp x29, x30, [sp], #32
| 68: d65f03c0 ret
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 12.1.0 aarch64-linux-gcc -c test.c -O2 -fsanitize=kernel-hwaddress --param hwasan-instrument-stack=1
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 14.2.0 aarch64-linux-objdump -d test.o
|
| test.o: file format elf64-littleaarch64
|
|
| Disassembly of section .text:
|
| 0000000000000000 <load_long>:
| 0: a9be7bfd stp x29, x30, [sp, #-32]!
| 4: 910003fd mov x29, sp
| 8: f9000bf3 str x19, [sp, #16]
| c: aa0003f3 mov x19, x0
| 10: 94000000 bl 0 <__hwasan_load8_noabort>
| 14: f9400260 ldr x0, [x19]
| 18: f9400bf3 ldr x19, [sp, #16]
| 1c: a8c27bfd ldp x29, x30, [sp], #32
| 20: d65f03c0 ret
|
| 0000000000000024 <store_long>:
| 24: a9be7bfd stp x29, x30, [sp, #-32]!
| 28: 910003fd mov x29, sp
| 2c: a90153f3 stp x19, x20, [sp, #16]
| 30: aa0003f3 mov x19, x0
| 34: aa0103f4 mov x20, x1
| 38: 94000000 bl 0 <__hwasan_store8_noabort>
| 3c: f9000274 str x20, [x19]
| 40: a94153f3 ldp x19, x20, [sp, #16]
| 44: a8c27bfd ldp x29, x30, [sp], #32
| 48: d65f03c0 ret
| 4c: d503201f nop
|
| 0000000000000050 <stack_func>:
| 50: a9bd7bfd stp x29, x30, [sp, #-48]!
| 54: d2800202 mov x2, #0x10 // #16
| 58: 9100c3e0 add x0, sp, #0x30
| 5c: 910003fd mov x29, sp
| 60: d378fc01 lsr x1, x0, #56
| 64: 910083e0 add x0, sp, #0x20
| 68: 11000821 add w1, w1, #0x2
| 6c: f9000bf3 str x19, [sp, #16]
| 70: 94000000 bl 0 <__hwasan_tag_memory>
| 74: d2e04000 mov x0, #0x200000000000000 // #144115188075855872
| 78: 8b2063e0 add x0, sp, x0
| 7c: f900101f str xzr, [x0, #32]
| 80: 94000000 bl 0 <extern_func>
| 84: d2e04000 mov x0, #0x200000000000000 // #144115188075855872
| 88: 8b2063e0 add x0, sp, x0
| 8c: d2800202 mov x2, #0x10 // #16
| 90: 52800001 mov w1, #0x0 // #0
| 94: f9401013 ldr x19, [x0, #32]
| 98: 910083e0 add x0, sp, #0x20
| 9c: 94000000 bl 0 <__hwasan_tag_memory>
| a0: aa1303e0 mov x0, x19
| a4: f9400bf3 ldr x19, [sp, #16]
| a8: a8c37bfd ldp x29, x30, [sp], #48
| ac: d65f03c0 ret
| [mark@lakrids:/mnt/data/tests/kasan-tags]%
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 12.1.0 aarch64-linux-gcc -c test.c -O2 -fsanitize=hwaddress
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 14.2.0 aarch64-linux-objdump -d test.o
|
| test.o: file format elf64-littleaarch64
|
|
| Disassembly of section .text:
|
| 0000000000000000 <load_long>:
| 0: a9be7bfd stp x29, x30, [sp, #-32]!
| 4: 910003fd mov x29, sp
| 8: f9000bf3 str x19, [sp, #16]
| c: aa0003f3 mov x19, x0
| 10: 94000000 bl 0 <__hwasan_load8>
| 14: f9400260 ldr x0, [x19]
| 18: f9400bf3 ldr x19, [sp, #16]
| 1c: a8c27bfd ldp x29, x30, [sp], #32
| 20: d65f03c0 ret
|
| 0000000000000024 <store_long>:
| 24: a9be7bfd stp x29, x30, [sp, #-32]!
| 28: 910003fd mov x29, sp
| 2c: a90153f3 stp x19, x20, [sp, #16]
| 30: aa0003f3 mov x19, x0
| 34: aa0103f4 mov x20, x1
| 38: 94000000 bl 0 <__hwasan_store8>
| 3c: f9000274 str x20, [x19]
| 40: a94153f3 ldp x19, x20, [sp, #16]
| 44: a8c27bfd ldp x29, x30, [sp], #32
| 48: d65f03c0 ret
| 4c: d503201f nop
|
| 0000000000000050 <stack_func>:
| 50: a9bd7bfd stp x29, x30, [sp, #-48]!
| 54: 910003fd mov x29, sp
| 58: f9000bf3 str x19, [sp, #16]
| 5c: 94000000 bl 0 <__hwasan_generate_tag>
| 60: 9100c3e1 add x1, sp, #0x30
| 64: d2800202 mov x2, #0x10 // #16
| 68: aa00e033 orr x19, x1, x0, lsl #56
| 6c: 910083e0 add x0, sp, #0x20
| 70: d378fe61 lsr x1, x19, #56
| 74: 94000000 bl 0 <__hwasan_tag_memory>
| 78: f81f027f stur xzr, [x19, #-16]
| 7c: 94000000 bl 0 <extern_func>
| 80: f85f0273 ldur x19, [x19, #-16]
| 84: 910083e0 add x0, sp, #0x20
| 88: d2800202 mov x2, #0x10 // #16
| 8c: 52800001 mov w1, #0x0 // #0
| 90: 94000000 bl 0 <__hwasan_tag_memory>
| 94: aa1303e0 mov x0, x19
| 98: f9400bf3 ldr x19, [sp, #16]
| 9c: a8c37bfd ldp x29, x30, [sp], #48
| a0: d65f03c0 ret
|
| Disassembly of section .text.startup:
|
| 0000000000000000 <_sub_I_00099_0>:
| 0: 14000000 b 0 <__hwasan_init>
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 12.1.0 aarch64-linux-gcc -c test.c -O2 -fsanitize=hwaddress --param hwasan-instrument-stack=1
| [mark@lakrids:/mnt/data/tests/kasan-tags]% usekorg 14.2.0 aarch64-linux-objdump -d test.o
|
| test.o: file format elf64-littleaarch64
|
|
| Disassembly of section .text:
|
| 0000000000000000 <load_long>:
| 0: a9be7bfd stp x29, x30, [sp, #-32]!
| 4: 910003fd mov x29, sp
| 8: f9000bf3 str x19, [sp, #16]
| c: aa0003f3 mov x19, x0
| 10: 94000000 bl 0 <__hwasan_load8>
| 14: f9400260 ldr x0, [x19]
| 18: f9400bf3 ldr x19, [sp, #16]
| 1c: a8c27bfd ldp x29, x30, [sp], #32
| 20: d65f03c0 ret
|
| 0000000000000024 <store_long>:
| 24: a9be7bfd stp x29, x30, [sp, #-32]!
| 28: 910003fd mov x29, sp
| 2c: a90153f3 stp x19, x20, [sp, #16]
| 30: aa0003f3 mov x19, x0
| 34: aa0103f4 mov x20, x1
| 38: 94000000 bl 0 <__hwasan_store8>
| 3c: f9000274 str x20, [x19]
| 40: a94153f3 ldp x19, x20, [sp, #16]
| 44: a8c27bfd ldp x29, x30, [sp], #32
| 48: d65f03c0 ret
| 4c: d503201f nop
|
| 0000000000000050 <stack_func>:
| 50: a9bd7bfd stp x29, x30, [sp, #-48]!
| 54: 910003fd mov x29, sp
| 58: f9000bf3 str x19, [sp, #16]
| 5c: 94000000 bl 0 <__hwasan_generate_tag>
| 60: 9100c3e1 add x1, sp, #0x30
| 64: d2800202 mov x2, #0x10 // #16
| 68: aa00e033 orr x19, x1, x0, lsl #56
| 6c: 910083e0 add x0, sp, #0x20
| 70: d378fe61 lsr x1, x19, #56
| 74: 94000000 bl 0 <__hwasan_tag_memory>
| 78: f81f027f stur xzr, [x19, #-16]
| 7c: 94000000 bl 0 <extern_func>
| 80: f85f0273 ldur x19, [x19, #-16]
| 84: 910083e0 add x0, sp, #0x20
| 88: d2800202 mov x2, #0x10 // #16
| 8c: 52800001 mov w1, #0x0 // #0
| 90: 94000000 bl 0 <__hwasan_tag_memory>
| 94: aa1303e0 mov x0, x19
| 98: f9400bf3 ldr x19, [sp, #16]
| 9c: a8c37bfd ldp x29, x30, [sp], #48
| a0: d65f03c0 ret
|
| Disassembly of section .text.startup:
|
| 0000000000000000 <_sub_I_00099_0>:
| 0: 14000000 b 0 <__hwasan_init>

Mark.

[Andrey Konovalov]

Yeah, this is #1 from https://bugzilla.kernel.org/show_bug.cgi?id=218854.

Thank you for the detailed investigation report!

> I think we have to disable KASAN_SW_TAGS with GCC until this is fixed.

Sounds good to me.

Please reference https://bugzilla.kernel.org/show_bug.cgi?id=218854 if
you end up sending a patch for this.

Also the syzbot's kvm instance should probably be switched to Clang
(@Alexander).

Thank you!