[PATCH v7 00/15] kasan: x86: arm64: KASAN tag-based mode for x86

[Maciej Wieczor-Retman]

======= Introduction
The patchset aims to add a KASAN tag-based mode for the x86 architecture
with the help of the new CPU feature called Linear Address Masking
(LAM). Main improvement introduced by the series is 2x lower memory
usage compared to KASAN's generic mode, the only currently available
mode on x86. The tag based mode may also find errors that the generic
mode couldn't because of differences in how these modes operate.

======= How does KASAN' tag-based mode work?
When enabled, memory accesses and allocations are augmented by the
compiler during kernel compilation. Instrumentation functions are added
to each memory allocation and each pointer dereference.

The allocation related functions generate a random tag and save it in
two places: in shadow memory that maps to the allocated memory, and in
the top bits of the pointer that points to the allocated memory. Storing
the tag in the top of the pointer is possible because of Top-Byte Ignore
(TBI) on arm64 architecture and LAM on x86.

The access related functions are performing a comparison between the tag
stored in the pointer and the one stored in shadow memory. If the tags
don't match an out of bounds error must have occurred and so an error
report is generated.

The general idea for the tag-based mode is very well explained in the
series with the original implementation [1].

[1] https://lore.kernel.org/all/cover.154409902...@google.com/

======= Differences summary compared to the arm64 tag-based mode
- Tag width:
- Tag width influences the chance of a tag mismatch due to two
tags from different allocations having the same value. The
bigger the possible range of tag values the lower the chance
of that happening.
- Shortening the tag width from 8 bits to 4, while it can help
with memory usage, it also increases the chance of not
reporting an error. 4 bit tags have a ~7% chance of a tag
mismatch.

- Address masking mechanism
- TBI in arm64 allows for storing metadata in the top 8 bits of
the virtual address.
- LAM in x86 allows storing tags in bits [62:57] of the pointer.
To maximize memory savings the tag width is reduced to bits
[60:57].

- Inline mode mismatch reporting
- Arm64 inserts a BRK instruction to pass metadata about a tag
mismatch to the KASAN report.
- Right now on x86 the INT3 instruction is used for the same
purpose. This series attempts to move it over to use UD1 in
the future so it's consistent with UBSan. Also using INT3 in
on the kernel side causes other issues that need to be patched
over.

======= Testing
Checked all the kunits for both software tags and generic KASAN after
making changes.

In generic mode (both with these patches and without) the results were:

kasan: pass:61 fail:1 skip:14 total:76
Totals: pass:61 fail:1 skip:14 total:76
not ok 1 kasan

and for software tags:

kasan: pass:65 fail:1 skip:10 total:76
Totals: pass:65 fail:1 skip:10 total:76
not ok 1 kasan

At the time of testing the one failing case is also present on generic
mode without this patchset applied. This seems to point to something
else being at fault for the one case not passing. The test case in
question concerns strscpy() out of bounds error not getting caught.

======= Benchmarks [1]
All tests were ran on a Sierra Forest server platform. The only
differences between the tests were kernel options:
- CONFIG_KASAN
- CONFIG_KASAN_GENERIC
- CONFIG_KASAN_SW_TAGS
- CONFIG_KASAN_INLINE [1]
- CONFIG_KASAN_OUTLINE

Boot time (until login prompt):
* 02:55 for clean kernel
* 05:42 / 06:32 for generic KASAN (inline/outline)
* 05:58 for tag-based KASAN (outline) [2]

Total memory usage (512GB present on the system - MemAvailable just
after boot):
* 12.56 GB for clean kernel
* 81.74 GB for generic KASAN
* 44.39 GB for tag-based KASAN

Kernel size:
* 14 MB for clean kernel
* 24.7 MB / 19.5 MB for generic KASAN (inline/outline)
* 27.1 MB / 18.1 MB for tag-based KASAN (inline/outline)

Work under load time comparison (compiling the mainline kernel) (200 cores):
* 62s for clean kernel
* 171s / 125s for generic KASAN (outline/inline)
* 145s for tag-based KASAN (outline) [2]

[1] Currently inline mode doesn't work on x86 due to things missing in
the compiler. I have written a patch for clang that seems to fix the
inline mode and I was able to boot and check that all patches regarding
the inline mode work as expected. My hope is to post the patch to LLVM
once this series is completed, and then make inline mode available in
the kernel config.

[2] While I was able to boot the inline tag-based kernel with my
compiler changes in a simulated environment, due to toolchain
difficulties I couldn't get it to boot on the machine I had access to.
Also boot time results from the simulation seem too good to be true, and
they're much too worse for the generic case to be believable. Therefore
I'm posting only results from the physical server platform.

======= Compilation
Clang was used to compile the series (make LLVM=1) since gcc doesn't
seem to have support for KASAN tag-based compiler instrumentation on
x86.

======= Dependencies
The series is based on this series [1] by Peter Zijlstra. Also for the
series to work on bigger systems (with more than 1 NUMA node and more
than 128 cores from my experience) the vmalloc fix patchset [2] is
needed. If you don't want vmalloc support it should run without it.

[1] https://lore.kernel.org/all/202511101146...@infradead.org/
[2] https://lore.kernel.org/all/cover.1764945396.g...@pm.me/

======= Previous versions
v6: https://lore.kernel.org/all/cover.1761763681.g...@pm.me/
v5: https://lore.kernel.org/all/cover.1756151769.git....@intel.com/
v4: https://lore.kernel.org/all/cover.1755004923.git....@intel.com/
v3: https://lore.kernel.org/all/cover.1743772053.git....@intel.com/
v2: https://lore.kernel.org/all/cover.1739866028.git....@intel.com/
v1: https://lore.kernel.org/all/cover.1738686764.git....@intel.com/

=== (two fixes patches were split off after v6) (merged into mm-unstable)
v1: https://lore.kernel.org/all/cover.1762267022.g...@pm.me/
v2: https://lore.kernel.org/all/cover.1764685296.g...@pm.me/
v3: https://lore.kernel.org/all/cover.1764874575.g...@pm.me/
v4: https://lore.kernel.org/all/cover.1764945396.g...@pm.me/

Changes v7:
- Rebased the series onto Peter Zijlstra's "WARN() hackery" v2 patchset.
- Fix flipped memset arguments in "x86/kasan: KASAN raw shadow memory
PTE init".
- Reorder tag width defines on arm64 to avoid redefinition warnings.
- Split off the pcpu unpoison patches into a separate fix oriented
series.
- Redid the canonicality checks so it works for KVM too (didn't change
the __canonical_address() function previously).
- A lot of fixes pointed out by Alexander in his great review:
- Fixed "x86/mm: Physical address comparisons in fill_p*d/pte"
- Merged "Support tag widths less than 8 bits" and "Make special
tags arch specific".
- Added comments and extended patch messages for patches
"x86/kasan: Make software tag-based kasan available" and
"mm/execmem: Untag addresses in EXECMEM_ROX related pointer arithmetic",
- Fixed KASAN_TAG_MASK definition order so all patches compile
individually.
- Renamed kasan_inline.c to kasan_sw_tags.c.

Changes v6:
- Initialize sw-tags only when LAM is available.
- Move inline mode to use UD1 instead of INT3
- Remove inline multishot patch.
- Fix the canonical check to work for user addresses too.
- Revise patch names and messages to align to tip tree rules.
- Fix vdso compilation issue.

Changes v5:
- Fix a bunch of arm64 compilation errors I didn't catch earlier.
Thank You Ada for testing the series!
- Simplify the usage of the tag handling x86 functions (virt_to_page,
phys_addr etc.).
- Remove within() and within_range() from the EXECMEM_ROX patch.

Changes v4:
- Revert x86 kasan_mem_to_shadow() scheme to the same on used in generic
KASAN. Keep the arithmetic shift idea for the KASAN in general since
it makes more sense for arm64 and in risc-v.
- Fix inline mode but leave it unavailable until a complementary
compiler patch can be merged.
- Apply Dave Hansen's comments on series formatting, patch style and
code simplifications.

Changes v3:
- Remove the runtime_const patch and setup a unified offset for both 5
and 4 paging levels.
- Add a fix for inline mode on x86 tag-based KASAN. Add a handler for
int3 that is generated on inline tag mismatches.
- Fix scripts/gdb/linux/kasan.py so the new signed mem_to_shadow() is
reflected there.
- Fix Documentation/arch/arm64/kasan-offsets.sh to take new offsets into
account.
- Made changes to the kasan_non_canonical_hook() according to upstream
discussion.
- Remove patches 2 and 3 since they related to risc-v and this series
adds only x86 related things.
- Reorder __tag_*() functions so they're before arch_kasan_*(). Remove
CONFIG_KASAN condition from __tag_set().

Changes v2:
- Split the series into one adding KASAN tag-based mode (this one) and
another one that adds the dense mode to KASAN (will post later).
- Removed exporting kasan_poison() and used a wrapper instead in
kasan_init_64.c
- Prepended series with 4 patches from the risc-v series and applied
review comments to the first patch as the rest already are reviewed.

Maciej Wieczor-Retman (13):
kasan: Fix inline mode for x86 tag-based mode
x86/kasan: Add arch specific kasan functions
x86/mm: Reset tag for virtual to physical address conversions
mm/execmem: Untag addresses in EXECMEM_ROX related pointer arithmetic
x86/mm: Physical address comparisons in fill_p*d/pte
x86/kasan: KASAN raw shadow memory PTE init
x86/mm: LAM compatible non-canonical definition
x86/mm: LAM initialization
x86: Minimal SLAB alignment
x86/kasan: Handle UD1 for inline KASAN reports
arm64: Unify software tag-based KASAN inline recovery path
x86/kasan: Logical bit shift for kasan_mem_to_shadow
x86/kasan: Make software tag-based kasan available

Samuel Holland (2):
kasan: sw_tags: Use arithmetic shift for shadow computation
kasan: arm64: x86: Make special tags arch specific

Documentation/arch/arm64/kasan-offsets.sh | 8 ++-
Documentation/arch/x86/x86_64/mm.rst | 6 +-
MAINTAINERS | 4 +-
arch/arm64/Kconfig | 10 +--
arch/arm64/include/asm/kasan-tags.h | 14 ++++
arch/arm64/include/asm/kasan.h | 2 -
arch/arm64/include/asm/memory.h | 14 +++-
arch/arm64/include/asm/uaccess.h | 1 +
arch/arm64/kernel/traps.c | 17 +----
arch/arm64/mm/kasan_init.c | 7 +-
arch/x86/Kconfig | 4 ++
arch/x86/boot/compressed/misc.h | 1 +
arch/x86/include/asm/bug.h | 1 +
arch/x86/include/asm/cache.h | 4 ++
arch/x86/include/asm/kasan-tags.h | 9 +++
arch/x86/include/asm/kasan.h | 81 ++++++++++++++++++++++-
arch/x86/include/asm/page.h | 23 ++++++-
arch/x86/include/asm/page_64.h | 1 +
arch/x86/kernel/head_64.S | 3 +
arch/x86/kernel/traps.c | 13 +++-
arch/x86/mm/Makefile | 2 +
arch/x86/mm/init.c | 3 +
arch/x86/mm/init_64.c | 11 +--
arch/x86/mm/kasan_init_64.c | 25 ++++++-
arch/x86/mm/kasan_sw_tags.c | 19 ++++++
arch/x86/mm/physaddr.c | 2 +
include/linux/kasan-tags.h | 21 ++++--
include/linux/kasan.h | 36 ++++++++--
include/linux/mm.h | 6 +-
include/linux/mmzone.h | 2 +-
include/linux/page-flags-layout.h | 9 +--
lib/Kconfig.kasan | 3 +-
mm/execmem.c | 9 ++-
mm/kasan/report.c | 37 +++++++++--
mm/vmalloc.c | 7 +-
scripts/Makefile.kasan | 3 +
scripts/gdb/linux/kasan.py | 5 +-
scripts/gdb/linux/mm.py | 5 +-
38 files changed, 354 insertions(+), 74 deletions(-)
create mode 100644 arch/arm64/include/asm/kasan-tags.h
create mode 100644 arch/x86/include/asm/kasan-tags.h
create mode 100644 arch/x86/mm/kasan_sw_tags.c

--
2.52.0

[Maciej Wieczor-Retman]

From: Samuel Holland <samuel....@sifive.com>

Currently, kasan_mem_to_shadow() uses a logical right shift, which turns
canonical kernel addresses into non-canonical addresses by clearing the
high KASAN_SHADOW_SCALE_SHIFT bits. The value of KASAN_SHADOW_OFFSET is
then chosen so that the addition results in a canonical address for the
shadow memory.

For KASAN_GENERIC, this shift/add combination is ABI with the compiler,
because KASAN_SHADOW_OFFSET is used in compiler-generated inline tag
checks[1], which must only attempt to dereference canonical addresses.

However, for KASAN_SW_TAGS there is some freedom to change the algorithm
without breaking the ABI. Because TBI is enabled for kernel addresses,
the top bits of shadow memory addresses computed during tag checks are
irrelevant, and so likewise are the top bits of KASAN_SHADOW_OFFSET.
This is demonstrated by the fact that LLVM uses a logical right shift in
the tag check fast path[2] but a sbfx (signed bitfield extract)
instruction in the slow path[3] without causing any issues.

Using an arithmetic shift in kasan_mem_to_shadow() provides a number of
benefits:

1) The memory layout doesn't change but is easier to understand.
KASAN_SHADOW_OFFSET becomes a canonical memory address, and the shifted
pointer becomes a negative offset, so KASAN_SHADOW_OFFSET ==
KASAN_SHADOW_END regardless of the shift amount or the size of the
virtual address space.

2) KASAN_SHADOW_OFFSET becomes a simpler constant, requiring only one
instruction to load instead of two. Since it must be loaded in each
function with a tag check, this decreases kernel text size by 0.5%.

3) This shift and the sign extension from kasan_reset_tag() can be
combined into a single sbfx instruction. When this same algorithm change
is applied to the compiler, it removes an instruction from each inline
tag check, further reducing kernel text size by an additional 4.6%.

These benefits extend to other architectures as well. On RISC-V, where
the baseline ISA does not shifted addition or have an equivalent to the
sbfx instruction, loading KASAN_SHADOW_OFFSET is reduced from 3 to 2
instructions, and kasan_mem_to_shadow(kasan_reset_tag(addr)) similarly
combines two consecutive right shifts.

Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp#L1316 [1]
Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp#L895 [2]
Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp#L669 [3]
Signed-off-by: Samuel Holland <samuel....@sifive.com>
Co-developed-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
Acked-by: Catalin Marinas <catalin...@arm.com>
---
Changelog v7: (Maciej)
- Change UL to ULL in report.c to fix some compilation warnings.

Changelog v6: (Maciej)
- Add Catalin's acked-by.
- Move x86 gdb snippet here from the last patch.

Changelog v5: (Maciej)
- (u64) -> (unsigned long) in report.c

Changelog v4: (Maciej)
- Revert x86 to signed mem_to_shadow mapping.
- Remove last two paragraphs since they were just poorer duplication of
the comments in kasan_non_canonical_hook().

Changelog v3: (Maciej)

- Fix scripts/gdb/linux/kasan.py so the new signed mem_to_shadow() is
reflected there.
- Fix Documentation/arch/arm64/kasan-offsets.sh to take new offsets into
account.
- Made changes to the kasan_non_canonical_hook() according to upstream

discussion. Settled on overflow on both ranges and separate checks for
x86 and arm.

Changelog v2: (Maciej)
- Correct address range that's checked in kasan_non_canonical_hook().
Adjust the comment inside.
- Remove part of comment from arch/arm64/include/asm/memory.h.
- Append patch message paragraph about the overflow in
kasan_non_canonical_hook().

Documentation/arch/arm64/kasan-offsets.sh | 8 +++--
arch/arm64/Kconfig | 10 +++----
arch/arm64/include/asm/memory.h | 14 ++++++++-
arch/arm64/mm/kasan_init.c | 7 +++--
include/linux/kasan.h | 10 +++++--
mm/kasan/report.c | 36 ++++++++++++++++++++---
scripts/gdb/linux/kasan.py | 5 +++-
scripts/gdb/linux/mm.py | 5 ++--
8 files changed, 76 insertions(+), 19 deletions(-)

diff --git a/Documentation/arch/arm64/kasan-offsets.sh b/Documentation/arch/arm64/kasan-offsets.sh
index 2dc5f9e18039..ce777c7c7804 100644
--- a/Documentation/arch/arm64/kasan-offsets.sh
+++ b/Documentation/arch/arm64/kasan-offsets.sh
@@ -5,8 +5,12 @@

print_kasan_offset () {
printf "%02d\t" $1
- printf "0x%08x00000000\n" $(( (0xffffffff & (-1 << ($1 - 1 - 32))) \
- - (1 << (64 - 32 - $2)) ))
+ if [[ $2 -ne 4 ]] then
+ printf "0x%08x00000000\n" $(( (0xffffffff & (-1 << ($1 - 1 - 32))) \
+ - (1 << (64 - 32 - $2)) ))
+ else
+ printf "0x%08x00000000\n" $(( (0xffffffff & (-1 << ($1 - 1 - 32))) ))
+ fi
}

echo KASAN_SHADOW_SCALE_SHIFT = 3
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6663ffd23f25..ac50ba2d760b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -433,11 +433,11 @@ config KASAN_SHADOW_OFFSET
default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS
default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS
default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS
- default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS
- default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS
- default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS
- default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS
- default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS
+ default 0xffff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS
+ default 0xffffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS
+ default 0xfffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS
+ default 0xffffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS
+ default 0xfffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS
default 0xffffffffffffffff

config UNWIND_TABLES
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index f1505c4acb38..7bbebde59a75 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -89,7 +89,15 @@
*
* KASAN_SHADOW_END is defined first as the shadow address that corresponds to
* the upper bound of possible virtual kernel memory addresses UL(1) << 64
- * according to the mapping formula.
+ * according to the mapping formula. For Generic KASAN, the address in the
+ * mapping formula is treated as unsigned (part of the compiler's ABI), so the
+ * end of the shadow memory region is at a large positive offset from
+ * KASAN_SHADOW_OFFSET. For Software Tag-Based KASAN, the address in the
+ * formula is treated as signed. Since all kernel addresses are negative, they
+ * map to shadow memory below KASAN_SHADOW_OFFSET, making KASAN_SHADOW_OFFSET
+ * itself the end of the shadow memory region. (User pointers are positive and
+ * would map to shadow memory above KASAN_SHADOW_OFFSET, but shadow memory is
+ * not allocated for them.)
*
* KASAN_SHADOW_START is defined second based on KASAN_SHADOW_END. The shadow
* memory start must map to the lowest possible kernel virtual memory address
@@ -100,7 +108,11 @@
*/
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
+#ifdef CONFIG_KASAN_GENERIC
#define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) + KASAN_SHADOW_OFFSET)
+#else
+#define KASAN_SHADOW_END KASAN_SHADOW_OFFSET
+#endif
#define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (UL(1) << ((va) - KASAN_SHADOW_SCALE_SHIFT)))
#define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual)
#define PAGE_END KASAN_SHADOW_START
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index abeb81bf6ebd..937f6eb8115b 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -198,8 +198,11 @@ static bool __init root_level_aligned(u64 addr)
/* The early shadow maps everything to a single page of zeroes */
asmlinkage void __init kasan_early_init(void)
{
- BUILD_BUG_ON(KASAN_SHADOW_OFFSET !=
- KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC))
+ BUILD_BUG_ON(KASAN_SHADOW_OFFSET !=
+ KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));
+ else
+ BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END);
BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN));
BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN));
BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN));
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index d12e1a5f5a9a..670de5427c32 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -61,8 +61,14 @@ int kasan_populate_early_shadow(const void *shadow_start,
#ifndef kasan_mem_to_shadow
static inline void *kasan_mem_to_shadow(const void *addr)
{
- return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT)
- + KASAN_SHADOW_OFFSET;
+ void *scaled;
+
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC))
+ scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT);
+ else
+ scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT);
+
+ return KASAN_SHADOW_OFFSET + scaled;
}
#endif

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 62c01b4527eb..b5beb1b10bd2 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -642,11 +642,39 @@ void kasan_non_canonical_hook(unsigned long addr)
const char *bug_type;

/*
- * All addresses that came as a result of the memory-to-shadow mapping
- * (even for bogus pointers) must be >= KASAN_SHADOW_OFFSET.
+ * For Generic KASAN, kasan_mem_to_shadow() uses the logical right shift
+ * and never overflows with the chosen KASAN_SHADOW_OFFSET values (on
+ * both x86 and arm64). Thus, the possible shadow addresses (even for
+ * bogus pointers) belong to a single contiguous region that is the
+ * result of kasan_mem_to_shadow() applied to the whole address space.
*/
- if (addr < KASAN_SHADOW_OFFSET)
- return;
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC)) {
+ if (addr < (unsigned long)kasan_mem_to_shadow((void *)(0ULL)) ||
+ addr > (unsigned long)kasan_mem_to_shadow((void *)(~0ULL)))
+ return;
+ }
+
+ /*
+ * For Software Tag-Based KASAN, kasan_mem_to_shadow() uses the
+ * arithmetic shift. Normally, this would make checking for a possible
+ * shadow address complicated, as the shadow address computation
+ * operation would overflow only for some memory addresses. However, due
+ * to the chosen KASAN_SHADOW_OFFSET values and the fact the
+ * kasan_mem_to_shadow() only operates on pointers with the tag reset,
+ * the overflow always happens.
+ *
+ * For arm64, the top byte of the pointer gets reset to 0xFF. Thus, the
+ * possible shadow addresses belong to a region that is the result of
+ * kasan_mem_to_shadow() applied to the memory range
+ * [0xFF000000000000, 0xFFFFFFFFFFFFFFFF]. Despite the overflow, the
+ * resulting possible shadow region is contiguous, as the overflow
+ * happens for both 0xFF000000000000 and 0xFFFFFFFFFFFFFFFF.
+ */
+ if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) && IS_ENABLED(CONFIG_ARM64)) {
+ if (addr < (unsigned long)kasan_mem_to_shadow((void *)(0xFFULL << 56)) ||
+ addr > (unsigned long)kasan_mem_to_shadow((void *)(~0ULL)))
+ return;
+ }

orig_addr = (unsigned long)kasan_shadow_to_mem((void *)addr);

diff --git a/scripts/gdb/linux/kasan.py b/scripts/gdb/linux/kasan.py
index 56730b3fde0b..4b86202b155f 100644
--- a/scripts/gdb/linux/kasan.py
+++ b/scripts/gdb/linux/kasan.py
@@ -7,7 +7,8 @@
#

import gdb
-from linux import constants, mm
+from linux import constants, utils, mm
+from ctypes import c_int64 as s64

def help():
t = """Usage: lx-kasan_mem_to_shadow [Hex memory addr]
@@ -39,6 +40,8 @@ class KasanMemToShadow(gdb.Command):
else:
help()
def kasan_mem_to_shadow(self, addr):
+ if constants.CONFIG_KASAN_SW_TAGS and not utils.is_target_arch('x86'):
+ addr = s64(addr)
return (addr >> self.p_ops.KASAN_SHADOW_SCALE_SHIFT) + self.p_ops.KASAN_SHADOW_OFFSET

KasanMemToShadow()
diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py
index 7571aebbe650..2e63f3dedd53 100644
--- a/scripts/gdb/linux/mm.py
+++ b/scripts/gdb/linux/mm.py
@@ -110,12 +110,13 @@ class aarch64_page_ops():
self.KERNEL_END = gdb.parse_and_eval("_end")

if constants.LX_CONFIG_KASAN_GENERIC or constants.LX_CONFIG_KASAN_SW_TAGS:
+ self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET
if constants.LX_CONFIG_KASAN_GENERIC:
self.KASAN_SHADOW_SCALE_SHIFT = 3
+ self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET
else:
self.KASAN_SHADOW_SCALE_SHIFT = 4
- self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET
- self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET
+ self.KASAN_SHADOW_END = self.KASAN_SHADOW_OFFSET
self.PAGE_END = self.KASAN_SHADOW_END - (1 << (self.vabits_actual - self.KASAN_SHADOW_SCALE_SHIFT))
else:
self.PAGE_END = self._PAGE_END(self.VA_BITS_MIN)
--
2.52.0

[Maciej Wieczor-Retman]

From: Samuel Holland <samuel....@sifive.com>

KASAN's tag-based mode defines multiple special tag values. They're
reserved for:
- Native kernel value. On arm64 it's 0xFF and it causes an early return
in the tag checking function.
- Invalid value. 0xFE marks an area as freed / unallocated. It's also
the value that is used to initialize regions of shadow memory.
- Min and max values. 0xFD is the highest value that can be randomly
generated for a new tag. 0 is the minimal value with the exception of
arm64's hardware mode where it is equal to 0xF0.

Metadata macro is also defined:
- Tag width equal to 8.

Tag-based mode on x86 is going to use 4 bit wide tags so all the above
values need to be changed accordingly.

Make tag width and native kernel tag arch specific for x86 and arm64.

Base the invalid tag value and the max value on the native kernel tag
since they follow the same pattern on both mentioned architectures.

Also generalize KASAN_SHADOW_INIT and 0xff used in various
page_kasan_tag* helpers.

Give KASAN_TAG_MIN the default value of zero, and move the special value
for hw_tags arm64 to its arch specific kasan-tags.h.

Signed-off-by: Samuel Holland <samuel....@sifive.com>
Co-developed-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>

---
Changelog v7:
- Reorder defines of arm64 tag width to prevent redefinition warnings.
- Remove KASAN_TAG_MASK so it's only defined in mmzone.h (Andrey
Konovalov)
- Merge the 'support tag widths less than 8 bits' with this patch since
they do similar things and overwrite each other. (Alexander)

Changelog v6:
- Add hardware tags KASAN_TAG_WIDTH value to the arm64 arch file.
- Keep KASAN_TAG_MASK in the mmzone.h.
- Remove ifndef from KASAN_SHADOW_INIT.

Changelog v5:
- Move KASAN_TAG_MIN to the arm64 kasan-tags.h for the hardware KASAN
mode case.

Changelog v4:
- Move KASAN_TAG_MASK to kasan-tags.h.

Changelog v2:
- Remove risc-v from the patch.

MAINTAINERS | 2 +-
arch/arm64/include/asm/kasan-tags.h | 14 ++++++++++++++
arch/arm64/include/asm/kasan.h | 2 --
arch/arm64/include/asm/uaccess.h | 1 +
arch/x86/include/asm/kasan-tags.h | 9 +++++++++
include/linux/kasan-tags.h | 19 ++++++++++++++-----
include/linux/kasan.h | 3 +--
include/linux/mm.h | 6 +++---
include/linux/page-flags-layout.h | 9 +--------
9 files changed, 44 insertions(+), 21 deletions(-)

create mode 100644 arch/arm64/include/asm/kasan-tags.h
create mode 100644 arch/x86/include/asm/kasan-tags.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 7bf6385efe04..a591598cc4b5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13420,7 +13420,7 @@ L: kasa...@googlegroups.com
S: Maintained
B: https://bugzilla.kernel.org/buglist.cgi?component=Sanitizers&product=Memory%20Management
F: Documentation/dev-tools/kasan.rst
-F: arch/*/include/asm/*kasan.h
+F: arch/*/include/asm/*kasan*.h
F: arch/*/mm/kasan_init*
F: include/linux/kasan*.h
F: lib/Kconfig.kasan
diff --git a/arch/arm64/include/asm/kasan-tags.h b/arch/arm64/include/asm/kasan-tags.h
new file mode 100644
index 000000000000..259952677443
--- /dev/null
+++ b/arch/arm64/include/asm/kasan-tags.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_KASAN_TAGS_H
+#define __ASM_KASAN_TAGS_H
+
+#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
+
+#ifdef CONFIG_KASAN_HW_TAGS
+#define KASAN_TAG_MIN 0xF0 /* minimum value for random tags */
+#define KASAN_TAG_WIDTH 4
+#else
+#define KASAN_TAG_WIDTH 8
+#endif
+
+#endif /* ASM_KASAN_TAGS_H */
diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h
index e1b57c13f8a4..d2841e0fb908 100644
--- a/arch/arm64/include/asm/kasan.h
+++ b/arch/arm64/include/asm/kasan.h
@@ -6,8 +6,6 @@

#include <linux/linkage.h>
#include <asm/memory.h>
-#include <asm/mte-kasan.h>
-#include <asm/pgtable-types.h>

#define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag)
#define arch_kasan_reset_tag(addr) __tag_reset(addr)
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index 6490930deef8..ccd41a39e3a1 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -22,6 +22,7 @@
#include <asm/cpufeature.h>
#include <asm/mmu.h>
#include <asm/mte.h>
+#include <asm/mte-kasan.h>
#include <asm/ptrace.h>
#include <asm/memory.h>
#include <asm/extable.h>
diff --git a/arch/x86/include/asm/kasan-tags.h b/arch/x86/include/asm/kasan-tags.h
new file mode 100644
index 000000000000..68ba385bc75c
--- /dev/null
+++ b/arch/x86/include/asm/kasan-tags.h
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_KASAN_TAGS_H
+#define __ASM_KASAN_TAGS_H
+
+#define KASAN_TAG_KERNEL 0xF /* native kernel pointers tag */
+
+#define KASAN_TAG_WIDTH 4
+
+#endif /* ASM_KASAN_TAGS_H */
diff --git a/include/linux/kasan-tags.h b/include/linux/kasan-tags.h
index 4f85f562512c..ad5c11950233 100644
--- a/include/linux/kasan-tags.h
+++ b/include/linux/kasan-tags.h
@@ -2,13 +2,22 @@
#ifndef _LINUX_KASAN_TAGS_H
#define _LINUX_KASAN_TAGS_H

+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+#include <asm/kasan-tags.h>
+#endif
+
+#ifndef KASAN_TAG_WIDTH
+#define KASAN_TAG_WIDTH 0
+#endif
+
+#ifndef KASAN_TAG_KERNEL
#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */
-#define KASAN_TAG_INVALID 0xFE /* inaccessible memory tag */
-#define KASAN_TAG_MAX 0xFD /* maximum value for random tags */
+#endif
+
+#define KASAN_TAG_INVALID (KASAN_TAG_KERNEL - 1) /* inaccessible memory tag */
+#define KASAN_TAG_MAX (KASAN_TAG_KERNEL - 2) /* maximum value for random tags */

-#ifdef CONFIG_KASAN_HW_TAGS
-#define KASAN_TAG_MIN 0xF0 /* minimum value for random tags */
-#else
+#ifndef KASAN_TAG_MIN
#define KASAN_TAG_MIN 0x00 /* minimum value for random tags */
#endif

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 670de5427c32..5cb21b90a2ec 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -39,8 +39,7 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;
/* Software KASAN implementations use shadow memory. */

#ifdef CONFIG_KASAN_SW_TAGS
-/* This matches KASAN_TAG_INVALID. */
-#define KASAN_SHADOW_INIT 0xFE
+#define KASAN_SHADOW_INIT KASAN_TAG_INVALID
#else
#define KASAN_SHADOW_INIT 0
#endif
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 8e9268cf929e..b61090a80e3f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1762,7 +1762,7 @@ static inline u8 page_kasan_tag(const struct page *page)

if (kasan_enabled()) {
tag = (page->flags.f >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
- tag ^= 0xff;
+ tag ^= KASAN_TAG_KERNEL;
}

return tag;
@@ -1775,7 +1775,7 @@ static inline void page_kasan_tag_set(struct page *page, u8 tag)
if (!kasan_enabled())
return;

- tag ^= 0xff;
+ tag ^= KASAN_TAG_KERNEL;
old_flags = READ_ONCE(page->flags.f);
do {
flags = old_flags;
@@ -1794,7 +1794,7 @@ static inline void page_kasan_tag_reset(struct page *page)

static inline u8 page_kasan_tag(const struct page *page)
{
- return 0xff;
+ return KASAN_TAG_KERNEL;
}

static inline void page_kasan_tag_set(struct page *page, u8 tag) { }
diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h
index 760006b1c480..b2cc4cb870e0 100644
--- a/include/linux/page-flags-layout.h
+++ b/include/linux/page-flags-layout.h
@@ -3,6 +3,7 @@
#define PAGE_FLAGS_LAYOUT_H

#include <linux/numa.h>
+#include <linux/kasan-tags.h>
#include <generated/bounds.h>

/*
@@ -72,14 +73,6 @@
#define NODE_NOT_IN_PAGE_FLAGS 1
#endif

-#if defined(CONFIG_KASAN_SW_TAGS)
-#define KASAN_TAG_WIDTH 8
-#elif defined(CONFIG_KASAN_HW_TAGS)
-#define KASAN_TAG_WIDTH 4
-#else
-#define KASAN_TAG_WIDTH 0
-#endif
-
#ifdef CONFIG_NUMA_BALANCING
#define LAST__PID_SHIFT 8
#define LAST__PID_MASK ((1 << LAST__PID_SHIFT)-1)
--
2.52.0

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wie...@intel.com>

The LLVM compiler uses hwasan-instrument-with-calls parameter to setup
inline or outline mode in tag-based KASAN. If zeroed, it means the
instrumentation implementation will be pasted into each relevant
location along with KASAN related constants during compilation. If set
to one all function instrumentation will be done with function calls
instead.

The default hwasan-instrument-with-calls value for the x86 architecture
in the compiler is "1", which is not true for other architectures.
Because of this, enabling inline mode in software tag-based KASAN
doesn't work on x86 as the kernel script doesn't zero out the parameter
and always sets up the outline mode.

Explicitly zero out hwasan-instrument-with-calls when enabling inline
mode in tag-based KASAN.

Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
Reviewed-by: Andrey Konovalov <andre...@gmail.com>
Reviewed-by: Alexander Potapenko <gli...@google.com>
---
Changelog v7:
- Add Alexander's Reviewed-by tag.

Changelog v6:
- Add Andrey's Reviewed-by tag.

Changelog v3:
- Add this patch to the series.

scripts/Makefile.kasan | 3 +++
1 file changed, 3 insertions(+)

diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan
index 0ba2aac3b8dc..e485814df3e9 100644
--- a/scripts/Makefile.kasan
+++ b/scripts/Makefile.kasan
@@ -76,8 +76,11 @@ CFLAGS_KASAN := -fsanitize=kernel-hwaddress
RUSTFLAGS_KASAN := -Zsanitizer=kernel-hwaddress \
-Zsanitizer-recover=kernel-hwaddress

+# LLVM sets hwasan-instrument-with-calls to 1 on x86 by default. Set it to 0
+# when inline mode is enabled.
ifdef CONFIG_KASAN_INLINE
kasan_params += hwasan-mapping-offset=$(KASAN_SHADOW_OFFSET)
+ kasan_params += hwasan-instrument-with-calls=0
else
kasan_params += hwasan-instrument-with-calls=1
endif
--
2.52.0

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wie...@intel.com>

KASAN's software tag-based mode needs multiple macros/functions to
handle tag and pointer interactions - to set, retrieve and reset tags
from the top bits of a pointer.

Mimic functions currently used by arm64 but change the tag's position to
bits [60:57] in the pointer.

Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
---
Changelog v7:
- Add KASAN_TAG_BYTE_MASK to avoid circular includes and avoid removing
KASAN_TAG_MASK from mmzone.h.
- Remove Andrey's Acked-by tag.

Changelog v6:
- Remove empty line after ifdef CONFIG_KASAN_SW_TAGS
- Add ifdef 64 bit to avoid problems in vdso32.
- Add Andrey's Acked-by tag.

Changelog v4:
- Rewrite __tag_set() without pointless casts and make it more readable.

Changelog v3:
- Reorder functions so that __tag_*() etc are above the
arch_kasan_*() ones.
- Remove CONFIG_KASAN condition from __tag_set()

arch/x86/include/asm/kasan.h | 42 ++++++++++++++++++++++++++++++++++--
include/linux/kasan-tags.h | 2 ++
include/linux/mmzone.h | 2 +-
3 files changed, 43 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h
index d7e33c7f096b..eab12527ed7f 100644
--- a/arch/x86/include/asm/kasan.h
+++ b/arch/x86/include/asm/kasan.h
@@ -3,6 +3,8 @@
#define _ASM_X86_KASAN_H

#include <linux/const.h>
+#include <linux/kasan-tags.h>
+#include <linux/types.h>
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
#define KASAN_SHADOW_SCALE_SHIFT 3

@@ -24,8 +26,43 @@
KASAN_SHADOW_SCALE_SHIFT)))

#ifndef __ASSEMBLER__
+#include <linux/bitops.h>
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+
+#ifdef CONFIG_KASAN_SW_TAGS
+#define __tag_shifted(tag) FIELD_PREP(GENMASK_ULL(60, 57), tag)
+#define __tag_reset(addr) (sign_extend64((u64)(addr), 56))
+#define __tag_get(addr) ((u8)FIELD_GET(GENMASK_ULL(60, 57), (u64)addr))
+#else
+#define __tag_shifted(tag) 0UL
+#define __tag_reset(addr) (addr)
+#define __tag_get(addr) 0
+#endif /* CONFIG_KASAN_SW_TAGS */
+
+#ifdef CONFIG_64BIT
+static inline void *__tag_set(const void *__addr, u8 tag)
+{
+ u64 addr = (u64)__addr;
+
+ addr &= ~__tag_shifted(KASAN_TAG_BYTE_MASK);
+ addr |= __tag_shifted(tag & KASAN_TAG_BYTE_MASK);
+
+ return (void *)addr;
+}
+#else
+static inline void *__tag_set(void *__addr, u8 tag)
+{
+ return __addr;
+}
+#endif
+
+#define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag)
+#define arch_kasan_reset_tag(addr) __tag_reset(addr)
+#define arch_kasan_get_tag(addr) __tag_get(addr)

#ifdef CONFIG_KASAN
+
void __init kasan_early_init(void);
void __init kasan_init(void);
void __init kasan_populate_shadow_for_vaddr(void *va, size_t size, int nid);
@@ -34,8 +71,9 @@ static inline void kasan_early_init(void) { }
static inline void kasan_init(void) { }
static inline void kasan_populate_shadow_for_vaddr(void *va, size_t size,
int nid) { }
-#endif

-#endif
+#endif /* CONFIG_KASAN */
+
+#endif /* __ASSEMBLER__ */

#endif
diff --git a/include/linux/kasan-tags.h b/include/linux/kasan-tags.h
index ad5c11950233..e4f26bec3673 100644
--- a/include/linux/kasan-tags.h
+++ b/include/linux/kasan-tags.h
@@ -10,6 +10,8 @@
#define KASAN_TAG_WIDTH 0
#endif

+#define KASAN_TAG_BYTE_MASK ((1UL << KASAN_TAG_WIDTH) - 1)
+

#ifndef KASAN_TAG_KERNEL
#define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */

#endif
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 7fb7331c5725..aa35f8331a4b 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1181,7 +1181,7 @@ static inline bool zone_is_empty(const struct zone *zone)
#define NODES_MASK ((1UL << NODES_WIDTH) - 1)
#define SECTIONS_MASK ((1UL << SECTIONS_WIDTH) - 1)
#define LAST_CPUPID_MASK ((1UL << LAST_CPUPID_SHIFT) - 1)
-#define KASAN_TAG_MASK ((1UL << KASAN_TAG_WIDTH) - 1)
+#define KASAN_TAG_MASK KASAN_TAG_BYTE_MASK
#define ZONEID_MASK ((1UL << ZONEID_SHIFT) - 1)

static inline enum zone_type memdesc_zonenum(memdesc_flags_t flags)
--
2.52.0

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wie...@intel.com>

In KASAN's generic mode the default value in shadow memory is zero.
During initialization of shadow memory pages they are allocated and
zeroed.

In KASAN's tag-based mode the default tag for the arm64 architecture is
0xFE which corresponds to any memory that should not be accessed. On x86
(where tags are 4-bit wide instead of 8-bit wide) that tag is 0xE so
during the initializations all the bytes in shadow memory pages should
be filled with it.

Use memblock_alloc_try_nid_raw() instead of memblock_alloc_try_nid() to
avoid zeroing out the memory so it can be set with the KASAN invalid
tag.

Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>

Reviewed-by: Alexander Potapenko <gli...@google.com>
---
Changelog v7:

- Fix flipped arguments in memset().
- Add Alexander's reviewed-by tag.

Changelog v2:
- Remove dense mode references, use memset() instead of kasan_poison().

arch/x86/mm/kasan_init_64.c | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 998b6010d6d3..7f5c11328ec1 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -34,6 +34,18 @@ static __init void *early_alloc(size_t size, int nid, bool should_panic)
return ptr;
}

+static __init void *early_raw_alloc(size_t size, int nid, bool should_panic)
+{
+ void *ptr = memblock_alloc_try_nid_raw(size, size,
+ __pa(MAX_DMA_ADDRESS), MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+
+ if (!ptr && should_panic)
+ panic("%pS: Failed to allocate page, nid=%d from=%lx\n",
+ (void *)_RET_IP_, nid, __pa(MAX_DMA_ADDRESS));
+
+ return ptr;
+}
+
static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr,
unsigned long end, int nid)
{
@@ -63,8 +75,9 @@ static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr,
if (!pte_none(*pte))
continue;

- p = early_alloc(PAGE_SIZE, nid, true);
- entry = pfn_pte(PFN_DOWN(__pa(p)), PAGE_KERNEL);
+ p = early_raw_alloc(PAGE_SIZE, nid, true);
+ memset(p, KASAN_SHADOW_INIT, PAGE_SIZE);
+ entry = pfn_pte(PFN_DOWN(__pa_nodebug(p)), PAGE_KERNEL);
set_pte_at(&init_mm, addr, pte, entry);
} while (pte++, addr += PAGE_SIZE, addr != end);
}
@@ -436,7 +449,7 @@ void __init kasan_init(void)
* it may contain some garbage. Now we can clear and write protect it,
* since after the TLB flush no one should write to it.
*/
- memset(kasan_early_shadow_page, 0, PAGE_SIZE);
+ memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
for (i = 0; i < PTRS_PER_PTE; i++) {
pte_t pte;
pgprot_t prot;
--
2.52.0

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wie...@intel.com>

Inline KASAN on x86 should do tag mismatch reports by passing the
metadata through the UD1 instruction and the faulty address through RDI,
a scheme that's already used by UBSan and is easy to extend.

The current LLVM way of passing KASAN software tag mode metadata is done
using the INT3 instruction. However that should be changed because it
doesn't align to how the kernel already handles UD1 for similar use
cases. Since inline software tag-based KASAN doesn't work on x86 due to
missing compiler support it can be fixed and the INT3 can be changed to
UD1 at the same time.

Add a kasan component to the #UD decoding and handling functions.

Make part of that hook - which decides whether to die or recover from a
tag mismatch - arch independent to avoid duplicating a long comment on
both x86 and arm64 architectures.

Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
---
Changelog v7:

- Redo the #UD handling that's based on Peter Zijlstra WARN() patches.
- Rename kasan_inline.c -> kasan_sw_tags.c (Alexander)

Changelog v6:
- Change the whole patch from using INT3 to UD1.

Changelog v5:
- Add die to argument list of kasan_inline_recover() in
arch/arm64/kernel/traps.c.

Changelog v4:
- Make kasan_handler() a stub in a header file. Remove #ifdef from
traps.c.
- Consolidate the "recover" comment into one place.
- Make small changes to the patch message.

MAINTAINERS | 2 +-
arch/x86/include/asm/bug.h | 1 +
arch/x86/include/asm/kasan.h | 20 ++++++++++++++++++++
arch/x86/kernel/traps.c | 13 ++++++++++++-
arch/x86/mm/Makefile | 2 ++
arch/x86/mm/kasan_sw_tags.c | 19 +++++++++++++++++++
include/linux/kasan.h | 23 +++++++++++++++++++++++
7 files changed, 78 insertions(+), 2 deletions(-)
create mode 100644 arch/x86/mm/kasan_sw_tags.c

diff --git a/MAINTAINERS b/MAINTAINERS
index a591598cc4b5..ff1c036ae39f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13421,7 +13421,7 @@ S: Maintained

B: https://bugzilla.kernel.org/buglist.cgi?component=Sanitizers&product=Memory%20Management
F: Documentation/dev-tools/kasan.rst

F: arch/*/include/asm/*kasan*.h
-F: arch/*/mm/kasan_init*
+F: arch/*/mm/kasan_*
F: include/linux/kasan*.h
F: lib/Kconfig.kasan
F: mm/kasan/
diff --git a/arch/x86/include/asm/bug.h b/arch/x86/include/asm/bug.h
index 83b0fb38732d..eb733ac14598 100644
--- a/arch/x86/include/asm/bug.h
+++ b/arch/x86/include/asm/bug.h
@@ -32,6 +32,7 @@
#define BUG_UD1 0xfffd
#define BUG_UD1_UBSAN 0xfffc
#define BUG_UD1_WARN 0xfffb
+#define BUG_UD1_KASAN 0xfffa
#define BUG_UDB 0xffd6
#define BUG_LOCK 0xfff0

diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h
index eab12527ed7f..6e083d45770d 100644
--- a/arch/x86/include/asm/kasan.h
+++ b/arch/x86/include/asm/kasan.h
@@ -6,6 +6,24 @@
#include <linux/kasan-tags.h>

#include <linux/types.h>
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)

+
+/*
+ * LLVM ABI for reporting tag mismatches in inline KASAN mode.
+ * On x86 the UD1 instruction is used to carry metadata in the ECX register
+ * to the KASAN report. ECX is used to differentiate KASAN from UBSan when
+ * decoding the UD1 instruction.
+ *
+ * SIZE refers to how many bytes the faulty memory access
+ * requested.
+ * WRITE bit, when set, indicates the access was a write, otherwise
+ * it was a read.
+ * RECOVER bit, when set, should allow the kernel to carry on after
+ * a tag mismatch. Otherwise die() is called.
+ */
+#define KASAN_ECX_RECOVER 0x20
+#define KASAN_ECX_WRITE 0x10
+#define KASAN_ECX_SIZE_MASK 0x0f
+#define KASAN_ECX_SIZE(ecx) (1 << ((ecx) & KASAN_ECX_SIZE_MASK))
#define KASAN_SHADOW_SCALE_SHIFT 3

/*
@@ -34,10 +52,12 @@
#define __tag_shifted(tag) FIELD_PREP(GENMASK_ULL(60, 57), tag)
#define __tag_reset(addr) (sign_extend64((u64)(addr), 56))
#define __tag_get(addr) ((u8)FIELD_GET(GENMASK_ULL(60, 57), (u64)addr))
+void kasan_inline_handler(struct pt_regs *regs, unsigned int metadata, u64 addr);
#else
#define __tag_shifted(tag) 0UL
#define __tag_reset(addr) (addr)
#define __tag_get(addr) 0
+static inline void kasan_inline_handler(struct pt_regs *regs, unsigned int metadata, u64 addr) { }
#endif /* CONFIG_KASAN_SW_TAGS */

#ifdef CONFIG_64BIT
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index cb324cc1fd99..e55e5441fc83 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -102,6 +102,7 @@ __always_inline int is_valid_bugaddr(unsigned long addr)
* FineIBT: f0 75 f9 lock jne . - 6
* UBSan{0}: 67 0f b9 00 ud1 (%eax),%eax
* UBSan{10}: 67 0f b9 40 10 ud1 0x10(%eax),%eax
+ * KASAN: 48 0f b9 41 XX ud1 0xXX(%rcx),%reg
* static_call: 0f b9 cc ud1 %esp,%ecx
* __WARN_trap: 67 48 0f b9 3a ud1 (%edx),%reg
*
@@ -190,6 +191,10 @@ __always_inline int decode_bug(unsigned long addr, s32 *imm, int *len)
addr += 1;
if (rm == 0) /* (%eax) */
type = BUG_UD1_UBSAN;
+ if (rm == 1) { /* (%ecx) */
+ type = BUG_UD1_KASAN;
+ *imm += reg << 8;
+ }
break;

case 2: *imm = *(s32 *)addr;
@@ -399,7 +404,7 @@ static inline void handle_invalid_op(struct pt_regs *regs)

static noinstr bool handle_bug(struct pt_regs *regs)
{
- unsigned long addr = regs->ip;
+ unsigned long kasan_addr, addr = regs->ip;
bool handled = false;
int ud_type, ud_len;
s32 ud_imm;
@@ -454,6 +459,12 @@ static noinstr bool handle_bug(struct pt_regs *regs)
}
break;

+ case BUG_UD1_KASAN:
+ kasan_addr = (u64)pt_regs_val(regs, ud_imm >> 8);
+ kasan_inline_handler(regs, ud_imm, kasan_addr);
+ handled = true;
+ break;
+
default:
break;
}
diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
index 5b9908f13dcf..b562963a866e 100644
--- a/arch/x86/mm/Makefile
+++ b/arch/x86/mm/Makefile
@@ -36,7 +36,9 @@ obj-$(CONFIG_PTDUMP) += dump_pagetables.o
obj-$(CONFIG_PTDUMP_DEBUGFS) += debug_pagetables.o

KASAN_SANITIZE_kasan_init_$(BITS).o := n
+KASAN_SANITIZE_kasan_sw_tags.o := n
obj-$(CONFIG_KASAN) += kasan_init_$(BITS).o
+obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o

KMSAN_SANITIZE_kmsan_shadow.o := n
obj-$(CONFIG_KMSAN) += kmsan_shadow.o
diff --git a/arch/x86/mm/kasan_sw_tags.c b/arch/x86/mm/kasan_sw_tags.c
new file mode 100644
index 000000000000..93b63be584fd
--- /dev/null
+++ b/arch/x86/mm/kasan_sw_tags.c
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/kasan.h>
+#include <linux/kdebug.h>
+
+void kasan_inline_handler(struct pt_regs *regs, unsigned int metadata, u64 addr)
+{
+ u64 pc = regs->ip;
+ bool recover = metadata & KASAN_ECX_RECOVER;
+ bool write = metadata & KASAN_ECX_WRITE;
+ size_t size = KASAN_ECX_SIZE(metadata);
+
+ if (user_mode(regs))
+ return;
+
+ if (!kasan_report((void *)addr, size, write, pc))
+ return;
+
+ kasan_die_unless_recover(recover, "Oops - KASAN", regs, metadata, die);
+}
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 5cb21b90a2ec..03e263fb9fa1 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -669,4 +669,27 @@ void kasan_non_canonical_hook(unsigned long addr);
static inline void kasan_non_canonical_hook(unsigned long addr) { }
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

+#ifdef CONFIG_KASAN_SW_TAGS
+/*
+ * The instrumentation allows to control whether we can proceed after
+ * a crash was detected. This is done by passing the -recover flag to
+ * the compiler. Disabling recovery allows to generate more compact
+ * code.
+ *
+ * Unfortunately disabling recovery doesn't work for the kernel right
+ * now. KASAN reporting is disabled in some contexts (for example when
+ * the allocator accesses slab object metadata; this is controlled by
+ * current->kasan_depth). All these accesses are detected by the tool,
+ * even though the reports for them are not printed.
+ *
+ * This is something that might be fixed at some point in the future.
+ */
+static inline void kasan_die_unless_recover(bool recover, char *msg, struct pt_regs *regs,
+ unsigned long err, void die_fn(const char *str, struct pt_regs *regs, long err))
+{
+ if (!recover)
+ die_fn(msg, regs, err);
+}
+#endif
+
#endif /* LINUX_KASAN_H */
--
2.52.0

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wie...@intel.com>

The tag-based KASAN adopts an arithemitc bit shift to convert a memory
address to a shadow memory address. While it makes a lot of sense on
arm64, it doesn't work well for all cases on x86 - either the
non-canonical hook becomes quite complex for different paging levels, or
the inline mode would need a lot more adjustments. Thus the best working
scheme is the logical bit shift and non-canonical shadow offset that x86
uses for generic KASAN, of course adjusted for the increased granularity
from 8 to 16 bytes.

Add an arch specific implementation of kasan_mem_to_shadow() that uses
the logical bit shift.

The non-canonical hook tries to calculate whether an address came from
kasan_mem_to_shadow(). First it checks whether this address fits into
the legal set of values possible to output from the mem to shadow
function.

Tie both generic and tag-based x86 KASAN modes to the address range
check associated with generic KASAN.

Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
---
Changelog v7:

- Redo the patch message and add a comment to __kasan_mem_to_shadow() to
provide better explanation on why x86 doesn't work well with the
arithemitc bit shift approach (Marco).

Changelog v4:

- Add this patch to the series.

arch/x86/include/asm/kasan.h | 15 +++++++++++++++
mm/kasan/report.c | 5 +++--
2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h
index 6e083d45770d..395e133d551d 100644
--- a/arch/x86/include/asm/kasan.h
+++ b/arch/x86/include/asm/kasan.h
@@ -49,6 +49,21 @@
#include <linux/bits.h>

#ifdef CONFIG_KASAN_SW_TAGS
+/*
+ * Using the non-arch specific implementation of __kasan_mem_to_shadow() with a
+ * arithmetic bit shift can cause high code complexity in KASAN's non-canonical
+ * hook for x86 or might not work for some paging level and KASAN mode
+ * combinations. The inline mode compiler support could also suffer from higher
+ * complexity for no specific benefit. Therefore the generic mode's logical
+ * shift implementation is used.
+ */
+static inline void *__kasan_mem_to_shadow(const void *addr)
+{
+ return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT)
+ + KASAN_SHADOW_OFFSET;
+}
+
+#define kasan_mem_to_shadow(addr) __kasan_mem_to_shadow(addr)

#define __tag_shifted(tag) FIELD_PREP(GENMASK_ULL(60, 57), tag)
#define __tag_reset(addr) (sign_extend64((u64)(addr), 56))
#define __tag_get(addr) ((u8)FIELD_GET(GENMASK_ULL(60, 57), (u64)addr))

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index b5beb1b10bd2..db6a9a3d01b2 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -642,13 +642,14 @@ void kasan_non_canonical_hook(unsigned long addr)
const char *bug_type;

/*
- * For Generic KASAN, kasan_mem_to_shadow() uses the logical right shift
+ * For Generic KASAN and Software Tag-Based mode on the x86
+ * architecture, kasan_mem_to_shadow() uses the logical right shift

* and never overflows with the chosen KASAN_SHADOW_OFFSET values (on

* both x86 and arm64). Thus, the possible shadow addresses (even for

* bogus pointers) belong to a single contiguous region that is the

* result of kasan_mem_to_shadow() applied to the whole address space.
*/

- if (IS_ENABLED(CONFIG_KASAN_GENERIC)) {
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC) || IS_ENABLED(CONFIG_X86_64)) {

if (addr < (unsigned long)kasan_mem_to_shadow((void *)(0ULL)) ||

addr > (unsigned long)kasan_mem_to_shadow((void *)(~0ULL)))
return;
--
2.52.0

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wie...@intel.com>

Make CONFIG_KASAN_SW_TAGS available for x86 machines if they have
ADDRESS_MASKING enabled (LAM) as that works similarly to Top-Byte Ignore
(TBI) that allows the software tag-based mode on arm64 platform.

The value for sw_tags KASAN_SHADOW_OFFSET was calculated by rearranging
the formulas for KASAN_SHADOW_START and KASAN_SHADOW_END from
arch/x86/include/asm/kasan.h - the only prerequisites being
KASAN_SHADOW_SCALE_SHIFT of 4, and KASAN_SHADOW_END equal to the
one from KASAN generic mode.

Set scale macro based on KASAN mode: in software tag-based mode 16 bytes
of memory map to one shadow byte and 8 in generic mode.

Disable CONFIG_KASAN_INLINE and CONFIG_KASAN_STACK when
CONFIG_KASAN_SW_TAGS is enabled on x86 until the appropriate compiler
support is available.

Signed-off-by: Maciej Wieczor-Retman <maciej.wie...@intel.com>
---
Changelog v7:

- Add a paragraph to the patch message explaining how the various
addresses and the KASAN_SHADOW_OFFSET were calculated.

Changelog v6:
- Don't enable KASAN if LAM is not supported.
- Move kasan_init_tags() to kasan_init_64.c to not clutter the setup.c
file.
- Move the #ifdef for the KASAN scale shift here.
- Move the gdb code to patch "Use arithmetic shift for shadow
computation".
- Return "depends on KASAN" line to Kconfig.
- Add the defer kasan config option so KASAN can be disabled on hardware
that doesn't have LAM.

Changelog v4:
- Add x86 specific kasan_mem_to_shadow().
- Revert x86 to the older unsigned KASAN_SHADOW_OFFSET. Do the same to
KASAN_SHADOW_START/END.
- Modify scripts/gdb/linux/kasan.py to keep x86 using unsigned offset.
- Disable inline and stack support when software tags are enabled on
x86.

Changelog v3:
- Remove runtime_const from previous patch and merge the rest here.
- Move scale shift definition back to header file.
- Add new kasan offset for software tag based mode.
- Fix patch message typo 32 -> 16, and 16 -> 8.
- Update lib/Kconfig.kasan with x86 now having software tag-based
support.

Changelog v2:
- Remove KASAN dense code.

Documentation/arch/x86/x86_64/mm.rst | 6 ++++--
arch/x86/Kconfig | 4 ++++
arch/x86/boot/compressed/misc.h | 1 +
arch/x86/include/asm/kasan.h | 4 ++++
arch/x86/mm/kasan_init_64.c | 6 ++++++
lib/Kconfig.kasan | 3 ++-
6 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/arch/x86/x86_64/mm.rst b/Documentation/arch/x86/x86_64/mm.rst
index a6cf05d51bd8..ccbdbb4cda36 100644
--- a/Documentation/arch/x86/x86_64/mm.rst
+++ b/Documentation/arch/x86/x86_64/mm.rst
@@ -60,7 +60,8 @@ Complete virtual memory map with 4-level page tables
ffffe90000000000 | -23 TB | ffffe9ffffffffff | 1 TB | ... unused hole
ffffea0000000000 | -22 TB | ffffeaffffffffff | 1 TB | virtual memory map (vmemmap_base)
ffffeb0000000000 | -21 TB | ffffebffffffffff | 1 TB | ... unused hole
- ffffec0000000000 | -20 TB | fffffbffffffffff | 16 TB | KASAN shadow memory
+ ffffec0000000000 | -20 TB | fffffbffffffffff | 16 TB | KASAN shadow memory (generic mode)
+ fffff40000000000 | -8 TB | fffffbffffffffff | 8 TB | KASAN shadow memory (software tag-based mode)
__________________|____________|__________________|_________|____________________________________________________________
|
| Identical layout to the 56-bit one from here on:
@@ -130,7 +131,8 @@ Complete virtual memory map with 5-level page tables
ffd2000000000000 | -11.5 PB | ffd3ffffffffffff | 0.5 PB | ... unused hole
ffd4000000000000 | -11 PB | ffd5ffffffffffff | 0.5 PB | virtual memory map (vmemmap_base)
ffd6000000000000 | -10.5 PB | ffdeffffffffffff | 2.25 PB | ... unused hole
- ffdf000000000000 | -8.25 PB | fffffbffffffffff | ~8 PB | KASAN shadow memory
+ ffdf000000000000 | -8.25 PB | fffffbffffffffff | ~8 PB | KASAN shadow memory (generic mode)
+ ffeffc0000000000 | -6 PB | fffffbffffffffff | 4 PB | KASAN shadow memory (software tag-based mode)
__________________|____________|__________________|_________|____________________________________________________________
|
| Identical layout to the 47-bit one from here on:
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index a26dc3bad804..b5275e322061 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -67,6 +67,7 @@ config X86
select ARCH_CLOCKSOURCE_INIT
select ARCH_CONFIGURES_CPU_MITIGATIONS
select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE
+ select ARCH_DISABLE_KASAN_INLINE if X86_64 && KASAN_SW_TAGS
select ARCH_ENABLE_HUGEPAGE_MIGRATION if X86_64 && HUGETLB_PAGE && MIGRATION
select ARCH_ENABLE_MEMORY_HOTPLUG if X86_64
select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
@@ -196,6 +197,8 @@ config X86
select HAVE_ARCH_JUMP_LABEL_RELATIVE
select HAVE_ARCH_KASAN if X86_64
select HAVE_ARCH_KASAN_VMALLOC if X86_64
+ select HAVE_ARCH_KASAN_SW_TAGS if ADDRESS_MASKING
+ select ARCH_NEEDS_DEFER_KASAN if ADDRESS_MASKING
select HAVE_ARCH_KFENCE
select HAVE_ARCH_KMSAN if X86_64
select HAVE_ARCH_KGDB
@@ -408,6 +411,7 @@ config AUDIT_ARCH
config KASAN_SHADOW_OFFSET
hex
depends on KASAN
+ default 0xeffffc0000000000 if KASAN_SW_TAGS
default 0xdffffc0000000000

config HAVE_INTEL_TXT
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index db1048621ea2..ded92b439ada 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -13,6 +13,7 @@
#undef CONFIG_PARAVIRT_SPINLOCKS
#undef CONFIG_KASAN
#undef CONFIG_KASAN_GENERIC
+#undef CONFIG_KASAN_SW_TAGS

#define __NO_FORTIFY

diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h
index 395e133d551d..3fa63036c93c 100644
--- a/arch/x86/include/asm/kasan.h
+++ b/arch/x86/include/asm/kasan.h
@@ -7,6 +7,7 @@

#include <linux/types.h>
#define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL)

+#ifdef CONFIG_KASAN_SW_TAGS
/*

* LLVM ABI for reporting tag mismatches in inline KASAN mode.

* On x86 the UD1 instruction is used to carry metadata in the ECX register

@@ -24,7 +25,10 @@
#define KASAN_ECX_WRITE 0x10
#define KASAN_ECX_SIZE_MASK 0x0f

#define KASAN_ECX_SIZE(ecx) (1 << ((ecx) & KASAN_ECX_SIZE_MASK))

+#define KASAN_SHADOW_SCALE_SHIFT 4
+#else
#define KASAN_SHADOW_SCALE_SHIFT 3
+#endif

/*
* Compiler uses shadow offset assuming that addresses start
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 7f5c11328ec1..3a5577341805 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -465,4 +465,10 @@ void __init kasan_init(void)

init_task.kasan_depth = 0;
kasan_init_generic();
+ pr_info("KernelAddressSanitizer initialized\n");
+
+ if (boot_cpu_has(X86_FEATURE_LAM))
+ kasan_init_sw_tags();
+ else
+ pr_info("KernelAddressSanitizer not initialized (sw-tags): hardware doesn't support LAM\n");
}
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index a4bb610a7a6f..d13ea8da7bfd 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -112,7 +112,8 @@ config KASAN_SW_TAGS

Requires GCC 11+ or Clang.

- Supported only on arm64 CPUs and relies on Top Byte Ignore.
+ Supported on arm64 CPUs that support Top Byte Ignore and on x86 CPUs
+ that support Linear Address Masking.

Consumes about 1/16th of available memory at kernel start and
add an overhead of ~20% for dynamic allocations.
--
2.52.0