[PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning

[Marco Elver]

Rework the general infrastructure around RANDOM_KMALLOC_CACHES into more
flexible PARTITION_KMALLOC_CACHES, with the former being a partitioning
mode of the latter.

Introduce a new mode, TYPED_KMALLOC_CACHES, which leverages a feature
available in Clang 22 and later, called "allocation tokens" via
__builtin_infer_alloc_token [1]. Unlike RANDOM_KMALLOC_CACHES, this mode
deterministically assigns a slab cache to an allocation of type T,
regardless of allocation site.

The builtin __builtin_infer_alloc_token(<malloc-args>, ...) instructs
the compiler to infer an allocation type from arguments commonly passed
to memory-allocating functions and returns a type-derived token ID. The
implementation passes kmalloc-args to the builtin: the compiler performs
best-effort type inference, and then recognizes common patterns such as
`kmalloc(sizeof(T), ...)`, `kmalloc(sizeof(T) * n, ...)`, but also
`(T *)kmalloc(...)`. Where the compiler fails to infer a type the
fallback token (default: 0) is chosen.

Note: kmalloc_obj(..) APIs fix the pattern how size and result type are
expressed, and therefore ensures there's not much drift in which
patterns the compiler needs to recognize. Specifically, kmalloc_obj()
and friends expand to `(TYPE *)KMALLOC(__obj_size, GFP)`, which the
compiler recognizes via the cast to TYPE*.

Clang's default token ID calculation is described as [1]:

typehashpointersplit: This mode assigns a token ID based on the hash
of the allocated type's name, where the top half ID-space is reserved
for types that contain pointers and the bottom half for types that do
not contain pointers.

Separating pointer-containing objects from pointerless objects and data
allocations can help mitigate certain classes of memory corruption
exploits [2]: attackers who gains a buffer overflow on a primitive
buffer cannot use it to directly corrupt pointers or other critical
metadata in an object residing in a different, isolated heap region.

It is important to note that heap isolation strategies offer a
best-effort approach, and do not provide a 100% security guarantee,
albeit achievable at relatively low performance cost. Note that this
also does not prevent cross-cache attacks, and SLAB_VIRTUAL [3] should
be used as a complementary mitigation (once available).

With all that, my kernel (x86 defconfig) shows me a histogram of slab
cache object distribution per /proc/slabinfo (after boot):

<slab cache> <objs> <hist>
kmalloc-part-15 1537 +++++++++++++++
kmalloc-part-14 2996 +++++++++++++++++++++++++++++
kmalloc-part-13 1555 +++++++++++++++
kmalloc-part-12 1045 ++++++++++
kmalloc-part-11 1717 +++++++++++++++++
kmalloc-part-10 1489 ++++++++++++++
kmalloc-part-09 851 ++++++++
kmalloc-part-08 710 +++++++
kmalloc-part-07 100 +
kmalloc-part-06 217 ++
kmalloc-part-05 105 +
kmalloc-part-04 4047 ++++++++++++++++++++++++++++++++++++++++
kmalloc-part-03 276 ++
kmalloc-part-02 283 ++
kmalloc-part-01 316 +++
kmalloc 1599 +++++++++++++++

The above /proc/slabinfo snapshot shows me there are 6943 allocated
objects (slabs 00 - 07) that the compiler claims contain no pointers or
it was unable to infer the type of, and 11900 objects that contain
pointers (slabs 08 - 15). On a whole, this looks relatively sane.

Additionally, when I compile my kernel with -Rpass=alloc-token, which
provides diagnostics where (after dead-code elimination) type inference
failed, I see 179 allocation sites where the compiler failed to identify
a type (down from 966 when I sent the RFC [4]). Some initial review
confirms these are mostly variable sized buffers, but also include
structs with trailing flexible length arrays.

Link: https://clang.llvm.org/docs/AllocToken.html [1]
Link: https://blog.dfsec.com/ios/2025/05/30/blasting-past-ios-18/ [2]
Link: https://lwn.net/Articles/944647/ [3]
Link: https://lore.kernel.org/all/20250825154505....@google.com/ [4]
Link: https://discourse.llvm.org/t/rfc-a-framework-for-allocator-partitioning-hints/87434
Signed-off-by: Marco Elver <el...@google.com>
---
Changelog:
v1:
* Rebase and switch to builtin name that was released in Clang 22.
* Keep RANDOM_KMALLOC_CACHES the default.

RFC: https://lore.kernel.org/all/20250825154505....@google.com/
---
Makefile | 5 ++
include/linux/percpu.h | 2 +-
include/linux/slab.h | 94 ++++++++++++++++++++-------------
kernel/configs/hardening.config | 2 +-
mm/Kconfig | 45 ++++++++++++----
mm/kfence/kfence_test.c | 4 +-
mm/slab.h | 4 +-
mm/slab_common.c | 48 ++++++++---------
mm/slub.c | 31 +++++------
9 files changed, 144 insertions(+), 91 deletions(-)

diff --git a/Makefile b/Makefile
index 2294decf0afc..93bb704bbf0e 100644
--- a/Makefile
+++ b/Makefile
@@ -957,6 +957,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)
endif
endif

+ifdef CONFIG_TYPED_KMALLOC_CACHES
+# PARTITION_KMALLOC_CACHES_NR + 1
+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 85bf8dd9f087..271b41be314d 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..c0bf00ee6025 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -612,10 +612,10 @@ static inline unsigned int arch_slab_minalign(void)
#define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \
(KMALLOC_MIN_SIZE) : 16)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define PARTITION_KMALLOC_CACHES_NR 15 // # of cache copies
#else
-#define RANDOM_KMALLOC_CACHES_NR 0
+#define PARTITION_KMALLOC_CACHES_NR 0
#endif

/*
@@ -634,8 +634,8 @@ enum kmalloc_cache_type {
#ifndef CONFIG_MEMCG
KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
- KMALLOC_RANDOM_START = KMALLOC_NORMAL,
- KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
+ KMALLOC_PARTITION_START = KMALLOC_NORMAL,
+ KMALLOC_PARTITION_END = KMALLOC_PARTITION_START + PARTITION_KMALLOC_CACHES_NR,
#ifdef CONFIG_SLUB_TINY
KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else
@@ -662,9 +662,20 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];
(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))

+#ifdef CONFIG_RANDOM_KMALLOC_CACHES
extern unsigned long random_kmalloc_seed;
+typedef struct { unsigned long ip; } kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t) { .ip = _RET_IP_ })
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+typedef struct { unsigned long v; } kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })
+#else
+/* no-op */
+typedef struct {} kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){})
+#endif

-static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
+static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
{
/*
* The most common case is KMALLOC_NORMAL, so test for it
@@ -672,9 +683,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
- return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
- ilog2(RANDOM_KMALLOC_CACHES_NR + 1));
+ /* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
+ return KMALLOC_PARTITION_START + hash_64(token.ip ^ random_kmalloc_seed,
+ ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+ return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;
#endif
@@ -864,10 +877,10 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)
@@ -938,7 +951,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;
@@ -948,14 +961,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}
- return __kmalloc_noprof(size, flags);
+ return __kmalloc_noprof(size, flags, token);
}
+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_token_t token);
+#define kmalloc_nolock_noprof(...) _kmalloc_nolock_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1075,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))
+ alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, __kmalloc_token(_size)))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_, __kmalloc_token(_size)))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;
@@ -1075,11 +1090,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);
+ return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, token);
}
+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**
@@ -1088,14 +1104,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf
* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
- return kmalloc_noprof(bytes, flags);
+ return _kmalloc_noprof(bytes, flags, token);
}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**
@@ -1138,9 +1155,9 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi
#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,
- unsigned long caller) __alloc_size(1);
+ unsigned long caller, kmalloc_token_t token) __alloc_size(1);
#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \
- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller)
+ __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller, __kmalloc_token(size))
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1157,17 +1174,18 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag
#define kmalloc_track_caller_noprof(...) \
kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_)

-static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
+ int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
if (__builtin_constant_p(n) && __builtin_constant_p(size))
- return kmalloc_node_noprof(bytes, flags, node);
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node);
+ return _kmalloc_node_noprof(bytes, flags, node, token);
+ return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node, token);
}
+#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

#define kcalloc_node(_n, _size, _flags, _node) \
@@ -1183,39 +1201,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_
* @size: how many bytes of memory are required.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags)
+static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return kmalloc_noprof(size, flags | __GFP_ZERO);
+ return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}
+#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node) __alloc_size(1);
+ gfp_t flags, int node, kmalloc_token_t token) __alloc_size(1);
#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \
- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node)
+ __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node, __kmalloc_token(_size))
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)
+#define kvmalloc_node_noprof(size, flags, node) \
+ kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)
+#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE)
#define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO)

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \
- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE))
+ alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE, __kmalloc_token(_size)))

static inline __alloc_size(1, 2) void *
-kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
+_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kvmalloc_node_align_noprof(bytes, 1, flags, node);
+ return __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), 1, flags, node, token);
}
-
+#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)
#define kvcalloc_node_noprof(_n,_s,_f,_node) kvmalloc_array_node_noprof(_n,_s,(_f)|__GFP_ZERO,_node)
#define kvcalloc_noprof(...) kvcalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE)
diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config
index 7c3924614e01..2963b6bd890f 100644
--- a/kernel/configs/hardening.config
+++ b/kernel/configs/hardening.config
@@ -22,7 +22,7 @@ CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
CONFIG_SLAB_BUCKETS=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-CONFIG_RANDOM_KMALLOC_CACHES=y
+CONFIG_PARTITION_KMALLOC_CACHES=y

# Sanity check userspace page table mappings.
CONFIG_PAGE_TABLE_CHECK=y
diff --git a/mm/Kconfig b/mm/Kconfig
index ebd8ea353687..fa4ffc1fcb80 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -247,22 +247,47 @@ config SLUB_STATS
out which slabs are relevant to a particular load.
Try running: slabinfo -DA

-config RANDOM_KMALLOC_CACHES
- default n
+config PARTITION_KMALLOC_CACHES
depends on !SLUB_TINY
- bool "Randomize slab caches for normal kmalloc"
+ bool "Partitioned slab caches for normal kmalloc"
help
- A hardening feature that creates multiple copies of slab caches for
- normal kmalloc allocation and makes kmalloc randomly pick one based
- on code address, which makes the attackers more difficult to spray
- vulnerable memory objects on the heap for the purpose of exploiting
- memory vulnerabilities.
+ A hardening feature that creates multiple isolated copies of slab
+ caches for normal kmalloc allocations. This makes it more difficult
+ to exploit memory-safety vulnerabilities by attacking vulnerable
+ co-located memory objects. Several modes are provided.

Currently the number of copies is set to 16, a reasonably large value
that effectively diverges the memory objects allocated for different
subsystems or modules into different caches, at the expense of a
- limited degree of memory and CPU overhead that relates to hardware and
- system workload.
+ limited degree of memory and CPU overhead that relates to hardware
+ and system workload.
+
+choice
+ prompt "Partitioned slab cache mode"
+ depends on PARTITION_KMALLOC_CACHES
+ default RANDOM_KMALLOC_CACHES
+ help
+ Selects the slab cache partitioning mode.
+
+config RANDOM_KMALLOC_CACHES
+ bool "Randomize slab caches for normal kmalloc"
+ help
+ Randomly pick a slab cache based on code address.
+
+config TYPED_KMALLOC_CACHES
+ bool "Type based slab cache selection for normal kmalloc"
+ depends on $(cc-option,-falloc-token-max=123)
+ help
+ Rely on Clang's allocation tokens to choose a slab cache, where token
+ IDs are derived from the allocated type.
+
+ The current effectiveness of Clang's type inference can be judged by
+ -Rpass=alloc-token, which provides diagnostics where (after dead-code
+ elimination) type inference failed.
+
+ Requires Clang 22 or later.
+
+endchoice

endmenu # Slab allocator options

diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
index 5725a367246d..8807ea8ed0d3 100644
--- a/mm/kfence/kfence_test.c
+++ b/mm/kfence/kfence_test.c
@@ -214,7 +214,7 @@ static void test_cache_destroy(void)
static inline size_t kmalloc_cache_alignment(size_t size)
{
/* just to get ->align so no need to pass in the real caller */
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, 0);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(0));
return kmalloc_caches[type][__kmalloc_index(size, false)]->align;
}

@@ -285,7 +285,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat

if (is_kfence_address(alloc)) {
struct slab *slab = virt_to_slab(alloc);
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, _RET_IP_);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(size));
struct kmem_cache *s = test_cache ?:
kmalloc_caches[type][__kmalloc_index(size, false)];

diff --git a/mm/slab.h b/mm/slab.h
index e9ab292acd22..dd49d37e253d 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -361,12 +361,12 @@ static inline unsigned int size_index_elem(unsigned int bytes)
* KMALLOC_MAX_CACHE_SIZE and the caller must check that.
*/
static inline struct kmem_cache *
-kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller)
+kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, kmalloc_token_t token)
{
unsigned int index;

if (!b)
- b = &kmalloc_caches[kmalloc_type(flags, caller)];
+ b = &kmalloc_caches[kmalloc_type(flags, token)];
if (size <= 192)
index = kmalloc_size_index[size_index_elem(size)];
else
diff --git a/mm/slab_common.c b/mm/slab_common.c
index d5a70a831a2a..21ab7dd79b5e 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -787,7 +787,7 @@ size_t kmalloc_size_roundup(size_t size)
* The flags don't matter since size_index is common to all.
* Neither does the caller for just getting ->object_size.
*/
- return kmalloc_slab(size, NULL, GFP_KERNEL, 0)->object_size;
+ return kmalloc_slab(size, NULL, GFP_KERNEL, __kmalloc_token(0))->object_size;
}

/* Above the smaller buckets, size is a multiple of page size. */
@@ -821,26 +821,26 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
#define KMALLOC_RCL_NAME(sz)
#endif

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define __KMALLOC_RANDOM_CONCAT(a, b) a ## b
-#define KMALLOC_RANDOM_NAME(N, sz) __KMALLOC_RANDOM_CONCAT(KMA_RAND_, N)(sz)
-#define KMA_RAND_1(sz) .name[KMALLOC_RANDOM_START + 1] = "kmalloc-rnd-01-" #sz,
-#define KMA_RAND_2(sz) KMA_RAND_1(sz) .name[KMALLOC_RANDOM_START + 2] = "kmalloc-rnd-02-" #sz,
-#define KMA_RAND_3(sz) KMA_RAND_2(sz) .name[KMALLOC_RANDOM_START + 3] = "kmalloc-rnd-03-" #sz,
-#define KMA_RAND_4(sz) KMA_RAND_3(sz) .name[KMALLOC_RANDOM_START + 4] = "kmalloc-rnd-04-" #sz,
-#define KMA_RAND_5(sz) KMA_RAND_4(sz) .name[KMALLOC_RANDOM_START + 5] = "kmalloc-rnd-05-" #sz,
-#define KMA_RAND_6(sz) KMA_RAND_5(sz) .name[KMALLOC_RANDOM_START + 6] = "kmalloc-rnd-06-" #sz,
-#define KMA_RAND_7(sz) KMA_RAND_6(sz) .name[KMALLOC_RANDOM_START + 7] = "kmalloc-rnd-07-" #sz,
-#define KMA_RAND_8(sz) KMA_RAND_7(sz) .name[KMALLOC_RANDOM_START + 8] = "kmalloc-rnd-08-" #sz,
-#define KMA_RAND_9(sz) KMA_RAND_8(sz) .name[KMALLOC_RANDOM_START + 9] = "kmalloc-rnd-09-" #sz,
-#define KMA_RAND_10(sz) KMA_RAND_9(sz) .name[KMALLOC_RANDOM_START + 10] = "kmalloc-rnd-10-" #sz,
-#define KMA_RAND_11(sz) KMA_RAND_10(sz) .name[KMALLOC_RANDOM_START + 11] = "kmalloc-rnd-11-" #sz,
-#define KMA_RAND_12(sz) KMA_RAND_11(sz) .name[KMALLOC_RANDOM_START + 12] = "kmalloc-rnd-12-" #sz,
-#define KMA_RAND_13(sz) KMA_RAND_12(sz) .name[KMALLOC_RANDOM_START + 13] = "kmalloc-rnd-13-" #sz,
-#define KMA_RAND_14(sz) KMA_RAND_13(sz) .name[KMALLOC_RANDOM_START + 14] = "kmalloc-rnd-14-" #sz,
-#define KMA_RAND_15(sz) KMA_RAND_14(sz) .name[KMALLOC_RANDOM_START + 15] = "kmalloc-rnd-15-" #sz,
-#else // CONFIG_RANDOM_KMALLOC_CACHES
-#define KMALLOC_RANDOM_NAME(N, sz)
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define __KMALLOC_PARTITION_CONCAT(a, b) a ## b
+#define KMALLOC_PARTITION_NAME(N, sz) __KMALLOC_PARTITION_CONCAT(KMA_PART_, N)(sz)
+#define KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 1] = "kmalloc-part-01-" #sz,
+#define KMA_PART_2(sz) KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 2] = "kmalloc-part-02-" #sz,
+#define KMA_PART_3(sz) KMA_PART_2(sz) .name[KMALLOC_PARTITION_START + 3] = "kmalloc-part-03-" #sz,
+#define KMA_PART_4(sz) KMA_PART_3(sz) .name[KMALLOC_PARTITION_START + 4] = "kmalloc-part-04-" #sz,
+#define KMA_PART_5(sz) KMA_PART_4(sz) .name[KMALLOC_PARTITION_START + 5] = "kmalloc-part-05-" #sz,
+#define KMA_PART_6(sz) KMA_PART_5(sz) .name[KMALLOC_PARTITION_START + 6] = "kmalloc-part-06-" #sz,
+#define KMA_PART_7(sz) KMA_PART_6(sz) .name[KMALLOC_PARTITION_START + 7] = "kmalloc-part-07-" #sz,
+#define KMA_PART_8(sz) KMA_PART_7(sz) .name[KMALLOC_PARTITION_START + 8] = "kmalloc-part-08-" #sz,
+#define KMA_PART_9(sz) KMA_PART_8(sz) .name[KMALLOC_PARTITION_START + 9] = "kmalloc-part-09-" #sz,
+#define KMA_PART_10(sz) KMA_PART_9(sz) .name[KMALLOC_PARTITION_START + 10] = "kmalloc-part-10-" #sz,
+#define KMA_PART_11(sz) KMA_PART_10(sz) .name[KMALLOC_PARTITION_START + 11] = "kmalloc-part-11-" #sz,
+#define KMA_PART_12(sz) KMA_PART_11(sz) .name[KMALLOC_PARTITION_START + 12] = "kmalloc-part-12-" #sz,
+#define KMA_PART_13(sz) KMA_PART_12(sz) .name[KMALLOC_PARTITION_START + 13] = "kmalloc-part-13-" #sz,
+#define KMA_PART_14(sz) KMA_PART_13(sz) .name[KMALLOC_PARTITION_START + 14] = "kmalloc-part-14-" #sz,
+#define KMA_PART_15(sz) KMA_PART_14(sz) .name[KMALLOC_PARTITION_START + 15] = "kmalloc-part-15-" #sz,
+#else // CONFIG_PARTITION_KMALLOC_CACHES
+#define KMALLOC_PARTITION_NAME(N, sz)
#endif

#define INIT_KMALLOC_INFO(__size, __short_size) \
@@ -849,7 +849,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
KMALLOC_RCL_NAME(__short_size) \
KMALLOC_CGROUP_NAME(__short_size) \
KMALLOC_DMA_NAME(__short_size) \
- KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size) \
+ KMALLOC_PARTITION_NAME(PARTITION_KMALLOC_CACHES_NR, __short_size) \
.size = __size, \
}

@@ -961,8 +961,8 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
flags |= SLAB_CACHE_DMA;
}

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- if (type >= KMALLOC_RANDOM_START && type <= KMALLOC_RANDOM_END)
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+ if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
flags |= SLAB_NO_MERGE;
#endif

diff --git a/mm/slub.c b/mm/slub.c
index 2b2d33cc735c..b13d1117c87d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2125,7 +2125,7 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,
if (!is_kmalloc_normal(s))
return sz;

- obj_exts_cache = kmalloc_slab(sz, NULL, gfp, 0);
+ obj_exts_cache = kmalloc_slab(sz, NULL, gfp, __kmalloc_token(0));
/*
* We can't simply compare s with obj_exts_cache, because random kmalloc
* caches have multiple caches per size, selected by caller address.
@@ -5239,7 +5239,7 @@ EXPORT_SYMBOL(__kmalloc_large_node_noprof);

static __always_inline
void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
- unsigned long caller)
+ unsigned long caller, kmalloc_token_t token)
{
struct kmem_cache *s;
void *ret;
@@ -5254,22 +5254,22 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
if (unlikely(!size))
return ZERO_SIZE_PTR;

- s = kmalloc_slab(size, b, flags, caller);
+ s = kmalloc_slab(size, b, flags, token);

ret = slab_alloc_node(s, NULL, flags, node, caller, size);
ret = kasan_kmalloc(s, ret, size, flags);
trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}
-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_, token);
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_);
+ return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_, token);
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5284,7 +5284,7 @@ EXPORT_SYMBOL(__kmalloc_noprof);
* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
* There is no reason to call it again and expect !NULL.
*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_token_t token)
{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
struct kmem_cache *s;
@@ -5307,7 +5307,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);
+ s = kmalloc_slab(size, NULL, alloc_gfp, token);

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*
@@ -5360,12 +5360,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
- int node, unsigned long caller)
+ int node, unsigned long caller, kmalloc_token_t token)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller, token);

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
@@ -6726,6 +6726,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
* @align: desired alignment.
* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
* @node: numa node to allocate from
+ * @token: allocation token.
*
* Only alignments up to those guaranteed by kmalloc() will be honored. Please see
* Documentation/core-api/memory-allocation.rst for more details.
@@ -6740,7 +6741,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
* Return: pointer to the allocated memory of %NULL in case of failure
*/
void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node)
+ gfp_t flags, int node, kmalloc_token_t token)
{
bool allow_block;
void *ret;
@@ -6751,7 +6752,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),
- node, _RET_IP_);
+ node, _RET_IP_, token);
if (ret || size <= PAGE_SIZE)
return ret;

@@ -8351,7 +8352,7 @@ static void __init bootstrap_kmalloc_sheaves(void)
{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--
2.53.0.1018.g2bb0e51243-goog

[Dan Carpenter]

Hi Marco,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url: https://github.com/intel-lab-lkp/linux/commits/Marco-Elver/slab-support-for-compiler-assisted-type-based-slab-cache-partitioning/20260401-035608
base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link: https://lore.kernel.org/r/20260331111240.153913-1-elver%40google.com
patch subject: [PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning
config: um-randconfig-r072-20260401
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
smatch: v0.5.0-9004-gb810ac53

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <l...@intel.com>
| Reported-by: Dan Carpenter <err...@gmail.com>
| Closes: https://lore.kernel.org/r/202604020400...@intel.com/

New smatch warnings:
drivers/misc/lkdtm/heap.c:118 lkdtm_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
drivers/misc/lkdtm/heap.c:169 lkdtm_KFENCE_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)

vim +118 drivers/misc/lkdtm/heap.c

73f62e60d80c2d drivers/misc/lkdtm/heap.c Kees Cook 2022-03-03 92 static void lkdtm_READ_AFTER_FREE(void)
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 93 {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 94 int *base, *val, saw;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 95 size_t len = 1024;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 96 /*
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 97 * The slub allocator will use the either the first word or
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 98 * the middle of the allocation to store the free pointer,
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 99 * depending on configurations. Store in the second word to
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 100 * avoid running into the freelist.
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 101 */
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 102 size_t offset = sizeof(*base);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 103
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 104 base = kmalloc(len, GFP_KERNEL);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 105 if (!base) {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 106 pr_info("Unable to allocate base memory.\n");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 107 return;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 108 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 109
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 110 val = kmalloc(len, GFP_KERNEL);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 111 if (!val) {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 112 pr_info("Unable to allocate val memory.\n");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 113 kfree(base);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 114 return;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 115 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 116
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 117 *val = 0x12345678;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 @118 base[offset] = *val;

This doesn't really matter, but the comment says we are writing to the
second word. The base[] array holds 256 integers, so:

base[sizeof(int)] = *val;

would be the third word, right? A word is unsigned long, right? All
of a sudden I am unsure. :P

ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 119 pr_info("Value in memory before free: %x\n", base[offset]);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 120
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 121 kfree(base);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 122
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 123 pr_info("Attempting bad read from freed memory\n");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 124 saw = base[offset];
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 125 if (saw != *val) {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 126 /* Good! Poisoning happened, so declare a win. */
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 127 pr_info("Memory correctly poisoned (%x)\n", saw);
5b777131bd8005 drivers/misc/lkdtm/heap.c Kees Cook 2021-06-23 128 } else {
5b777131bd8005 drivers/misc/lkdtm/heap.c Kees Cook 2021-06-23 129 pr_err("FAIL: Memory was not poisoned!\n");
5b777131bd8005 drivers/misc/lkdtm/heap.c Kees Cook 2021-06-23 130 pr_expected_config_param(CONFIG_INIT_ON_FREE_DEFAULT_ON, "init_on_free");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 131 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 132
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 133 kfree(val);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 134 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 135
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 136 static void lkdtm_KFENCE_READ_AFTER_FREE(void)
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 137 {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 138 int *base, val, saw;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 139 unsigned long timeout, resched_after;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 140 size_t len = 1024;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 141 /*
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 142 * The slub allocator will use the either the first word or
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 143 * the middle of the allocation to store the free pointer,
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 144 * depending on configurations. Store in the second word to
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 145 * avoid running into the freelist.
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 146 */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 147 size_t offset = sizeof(*base);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 148
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 149 /*
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 150 * 100x the sample interval should be more than enough to ensure we get
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 151 * a KFENCE allocation eventually.
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 152 */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 153 timeout = jiffies + msecs_to_jiffies(100 * kfence_sample_interval);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 154 /*
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 155 * Especially for non-preemption kernels, ensure the allocation-gate
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 156 * timer can catch up: after @resched_after, every failed allocation
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 157 * attempt yields, to ensure the allocation-gate timer is scheduled.
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 158 */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 159 resched_after = jiffies + msecs_to_jiffies(kfence_sample_interval);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 160 do {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 161 base = kmalloc(len, GFP_KERNEL);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 162 if (!base) {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 163 pr_err("FAIL: Unable to allocate kfence memory!\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 164 return;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 165 }
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 166
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 167 if (is_kfence_address(base)) {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 168 val = 0x12345678;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 @169 base[offset] = val;
^^^^^^^^^^^^^^^^^^
Same here.

aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 170 pr_info("Value in memory before free: %x\n", base[offset]);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 171
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 172 kfree(base);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 173
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 174 pr_info("Attempting bad read from freed memory\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 175 saw = base[offset];
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 176 if (saw != val) {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 177 /* Good! Poisoning happened, so declare a win. */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 178 pr_info("Memory correctly poisoned (%x)\n", saw);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 179 } else {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 180 pr_err("FAIL: Memory was not poisoned!\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 181 pr_expected_config_param(CONFIG_INIT_ON_FREE_DEFAULT_ON, "init_on_free");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 182 }
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 183 return;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 184 }
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 185
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 186 kfree(base);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 187 if (time_after(jiffies, resched_after))
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 188 cond_resched();
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 189 } while (time_before(jiffies, timeout));
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 190
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 191 pr_err("FAIL: kfence memory never allocated!\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 192 }

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[Marco Elver]

On Thu, 2 Apr 2026 at 15:33, Dan Carpenter <err...@gmail.com> wrote:
>
> Hi Marco,
>
> kernel test robot noticed the following build warnings:
>
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url: https://github.com/intel-lab-lkp/linux/commits/Marco-Elver/slab-support-for-compiler-assisted-type-based-slab-cache-partitioning/20260401-035608
> base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
> patch link: https://lore.kernel.org/r/20260331111240.153913-1-elver%40google.com
> patch subject: [PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning
> config: um-randconfig-r072-20260401
> compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
> smatch: v0.5.0-9004-gb810ac53
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <l...@intel.com>
> | Reported-by: Dan Carpenter <err...@gmail.com>
> | Closes: https://lore.kernel.org/r/202604020400...@intel.com/
>
> New smatch warnings:
> drivers/misc/lkdtm/heap.c:118 lkdtm_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
> drivers/misc/lkdtm/heap.c:169 lkdtm_KFENCE_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)

How is this related to the patch I sent? Did the <linux/slab.h> change
force rechecking of all these files and it found latent issues?

???

So this is clearly a minor defect in this existing code (comment wrong
or code might want to match what the comment said), but "slab: support
for compiler-assisted type-based slab cache partitioning" didn't touch
that.

Unrelated to "slab: support for compiler-assisted type-based slab
cache partitioning".

[Dan Carpenter]

On Thu, Apr 02, 2026 at 03:48:20PM +0200, Marco Elver wrote:
> On Thu, 2 Apr 2026 at 15:33, Dan Carpenter <err...@gmail.com> wrote:
> >
> > Hi Marco,
> >
> > kernel test robot noticed the following build warnings:
> >
> > https://git-scm.com/docs/git-format-patch#_base_tree_information]
> >
> > url: https://github.com/intel-lab-lkp/linux/commits/Marco-Elver/slab-support-for-compiler-assisted-type-based-slab-cache-partitioning/20260401-035608
> > base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
> > patch link: https://lore.kernel.org/r/20260331111240.153913-1-elver%40google.com
> > patch subject: [PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning
> > config: um-randconfig-r072-20260401
> > compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
> > smatch: v0.5.0-9004-gb810ac53
> >
> > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > the same patch/commit), kindly add following tags
> > | Reported-by: kernel test robot <l...@intel.com>
> > | Reported-by: Dan Carpenter <err...@gmail.com>
> > | Closes: https://lore.kernel.org/r/202604020400...@intel.com/
> >
> > New smatch warnings:
> > drivers/misc/lkdtm/heap.c:118 lkdtm_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
> > drivers/misc/lkdtm/heap.c:169 lkdtm_KFENCE_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
>
> How is this related to the patch I sent? Did the <linux/slab.h> change
> force rechecking of all these files and it found latent issues?
>

Oh, crud. It turns out that for this check Smatch allows
integer_array[sizeof()]so long as we know that the index is within
bounds. What happened is that your patch renamed the kmalloc()
function so Smatch stopped knowing the size of the buffer.

For these zero day bot warnings, the emails are automatically generated
so I don't have any context outside what's in the email. I saw that
Kees wrote the code, but I figured maybe you forwarded it or something.
Sorry about that.

regards,
dan carpenter

[Marco Elver]

Oh, I see. Smatch doesn't respect the __alloc_size attribute then?

> For these zero day bot warnings, the emails are automatically generated
> so I don't have any context outside what's in the email. I saw that
> Kees wrote the code, but I figured maybe you forwarded it or something.
> Sorry about that.

No worries and thanks!

[Harry Yoo (Oracle)]

Presumably because only the latest Clang supports it?

> RFC: https://lore.kernel.org/all/20250825154505....@google.com/
> ---
> Makefile | 5 ++
> include/linux/percpu.h | 2 +-
> include/linux/slab.h | 94 ++++++++++++++++++++-------------
> kernel/configs/hardening.config | 2 +-
> mm/Kconfig | 45 ++++++++++++----
> mm/kfence/kfence_test.c | 4 +-
> mm/slab.h | 4 +-
> mm/slab_common.c | 48 ++++++++---------
> mm/slub.c | 31 +++++------
> 9 files changed, 144 insertions(+), 91 deletions(-)
>

> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 15a60b501b95..c0bf00ee6025 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h

> @@ -864,10 +877,10 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
> * with the exception of kunit tests
> */
>
> -void *__kmalloc_noprof(size_t size, gfp_t flags)
> +void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
> __assume_kmalloc_alignment __alloc_size(1);
>
> -void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
> +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
> __assume_kmalloc_alignment __alloc_size(1);

So the @token parameter is unused when CONFIG_PARTITION_KMALLOC_CACHES is
disabled but still increases the kernel size by a few kilobytes...
but yeah I'm not sure if we can get avoid it without hurting readability.

Just saying. (does anybody care?)

> void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

Assuming not all people building the kernel are security experts...
(including myself) could you please add some insights/guidance on how to
decide between RANDOM_KMALLOC_CACHES and TYPED_KMALLOC_CACHES?

Something like what Florent wrote [1]:
| One more perspective on this: in a data center environment, attackers
| typically get a first foothold by compromising a userspace network
| service. If they can do that once, they can do that a bunch of times,
| and gain code execution on different machines every time.
|
| Before trying to exploit a kernel memory corruption to elevate
| privileges on a machine, they can test the SLAB properties of the
| running kernel to make sure it's as they wish (eg: with timing side
| channels like in the SLUBStick paper). So with RANDOM_KMALLOC_CACHES,
| attackers can just keep retrying their attacks until they land on a
| machine where the types T and S are collocated and only then proceed
| with their exploit.
|
| With TYPED_KMALLOC_CACHES (and with SLAB_VIRTUAL hopefully someday),
| they are simply never able to cross the "objects without pointers" to
| "objects with pointers" boundary which really gets in the way of many
| exploitation techniques and feels at least to me like a much stronger
| security boundary.
|
| This limit of RANDOM_KMALLOC_CACHES may not be as relevant in other
| deployments (eg: on a smartphone) but it makes me strongly prefer
| TYPED_KMALLOC_CACHES for server use cases at least.

[1] https://lore.kernel.org/all/CALGbS4U6fox7SwmdHfDuawmO...@mail.gmail.com

Otherwise the patch is really straightforward and looks good to me.

Thanks!

--
Cheers,
Harry / Hyeonggon

[Harry Yoo (Oracle)]

Now somewhat out-of-scope (or at least pre-existing) review comments
from Sashiko that I think are still worth mentioning...

> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h

> @@ -662,9 +662,20 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];

> -static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
> +static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
> {
> /*
> * The most common case is KMALLOC_NORMAL, so test for it
> @@ -672,9 +683,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
> */
> if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))

Sashiko pointed out KMALLOC_CGROUP caches are not partitioned [1]:
| Do allocations with the __GFP_ACCOUNT flag completely bypass typed
| and random partitioning? KMALLOC_NOT_NORMAL_BITS includes __GFP_ACCOUNT.

Right.

| If this bit is set, the code bypasses the partitioning logic and routes
| the allocation to the KMALLOC_CGROUP cache.

Right.

| Since user-controllable objects
| like msg_msg, file descriptors, and pipes are allocated with __GFP_ACCOUNT,

Right.

| they will all be clustered in the exact same unpartitioned cache.

Right.

From security perspective do you think it'd be worthwhile to partition
KMALLOC_CGROUP caches? (I see at least few hundreds of users, unlike
KMALLOC_RECLAIM where there are only few users).

Another valid concern from Sashiko [1]:
| Does this leave reallocation functions like krealloc() and kvrealloc()
| without allocation token propagation?
|
| When an object is reallocated and requires memory expansion, the underlying
| generic SLUB code allocates a new buffer. Because the token macro is not
| applied to these realloc paths, __builtin_infer_alloc_token() evaluates
| locally on a generic size_t variable rather than the original type.

I think this is a valid point and worth addressing.

| This causes it to return the fallback token (0), which silently migrates the
| object from its isolated typed cache to the shared fallback cache
| (kmalloc-part-00) when resized.

[1] https://sashiko.dev/#/patchset/20260331111240.153913-1-elver%40google.com

[Vlastimil Babka (SUSE)]

On 4/3/26 08:27, Harry Yoo (Oracle) wrote:
>> diff --git a/include/linux/slab.h b/include/linux/slab.h
>> index 15a60b501b95..c0bf00ee6025 100644
>> --- a/include/linux/slab.h
>> +++ b/include/linux/slab.h
>> @@ -864,10 +877,10 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
>> * with the exception of kunit tests
>> */
>>
>> -void *__kmalloc_noprof(size_t size, gfp_t flags)
>> +void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
>> __assume_kmalloc_alignment __alloc_size(1);
>>
>> -void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
>> +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
>> __assume_kmalloc_alignment __alloc_size(1);
>
> So the @token parameter is unused when CONFIG_PARTITION_KMALLOC_CACHES is
> disabled but still increases the kernel size by a few kilobytes...
> but yeah I'm not sure if we can get avoid it without hurting readability.
>
> Just saying. (does anybody care?)

Well we did care enough with CONFIG_SLAB_BUCKETS to hide the unused param
using DECL_BUCKET_PARAMS(), so maybe extend that idea?
I think it's not just kernel size, but increased register pressure etc.

[Harry Yoo (Oracle)]

Hmm yeah.

I wasn't sure if we could do this without hurting readability,
but perhaps we could...

> so maybe extend that idea?
> I think it's not just kernel size, but increased register pressure etc.

Something like this should work? (diff on top of this patch)

--
Cheers,
Harry / Hyeonggon

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c0bf00ee6025..0496d2e63f5e 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -871,16 +871,32 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
#define PASS_BUCKET_PARAM(_b) NULL
#endif

+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
+#define _PASS_TOKEN_PARAM(_token) , (_token)
+#define PASS_TOKEN_PARAM(_token) (_token)
+#else
+#define DECL_TOKEN_PARAM(_token)
+#define _PASS_TOKEN_PARAM(_token)
+#define PASS_TOKEN_PARAM(_token) ((kmalloc_token_t){})
+#endif
+
+#define DECL_KMALLOC_PARAMS(_size, _b, _token) DECL_BUCKET_PARAMS(_size, _b) \
+ DECL_TOKEN_PARAM(_token)
+
+#define PASS_KMALLOC_PARAMS(_size, _b, _token) PASS_BUCKET_PARAMS(_size, _b) \
+ _PASS_TOKEN_PARAM(_token)
+
/*
* The following functions are not to be used directly and are intended only
* for internal use from kmalloc() and kmalloc_node()

* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

@@ -964,7 +980,7 @@ static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t

kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}

- return __kmalloc_noprof(size, flags, token);
+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);

}
#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

@@ -1075,10 +1091,10 @@ void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_tok

__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \

- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, __kmalloc_token(_size)))
+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_, __kmalloc_token(_size)))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
@@ -1093,7 +1109,7 @@ static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, g

kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}

- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, token);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);

}
#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

@@ -1154,10 +1170,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi
*/

#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,
- unsigned long caller, kmalloc_token_t token) __alloc_size(1);
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node,
+ unsigned long caller) __alloc_size(1);

#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \

- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller, __kmalloc_token(size))
+ __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(size, NULL, __kmalloc_token(size)), flags, node, caller)
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1183,7 +1199,7 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size

return NULL;
if (__builtin_constant_p(n) && __builtin_constant_p(size))

return _kmalloc_node_noprof(bytes, flags, node, token);

- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node, token);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), flags, node);

}
#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

@@ -1209,10 +1225,10 @@ static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, km

#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node, kmalloc_token_t token) __alloc_size(1);
+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
+ gfp_t flags, int node) __alloc_size(1);

#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \

- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node, __kmalloc_token(_size))
+ __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)

#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)

@@ -1225,7 +1241,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \

- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE, __kmalloc_token(_size)))
+ alloc_hooks(__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), 1, _flags, NUMA_NO_NODE))

static inline __alloc_size(1, 2) void *

_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
@@ -1235,7 +1251,7 @@ _kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmallo

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), 1, flags, node, token);
+ return __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), 1, flags, node);

}
#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/mm/slub.c b/mm/slub.c
index 5630dde94df0..84f129d79c99 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5293,15 +5293,17 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_, token);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,

+ _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
{
- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_, token);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,

+ NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5394,10 +5396,11 @@ void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_tok
}
EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
- int node, unsigned long caller, kmalloc_token_t token)
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,
+ int node, unsigned long caller)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller, token);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,

+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
@@ -6812,8 +6815,8 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
*

* Return: pointer to the allocated memory of %NULL in case of failure
*/

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node, kmalloc_token_t token)
+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token),
+ unsigned long align, gfp_t flags, int node)
{
bool allow_block;
void *ret;
@@ -6824,7 +6827,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),

- node, _RET_IP_, token);
+ node, _RET_IP_, PASS_TOKEN_PARAM(token));

if (ret || size <= PAGE_SIZE)
return ret;

--
2.43.0

[Marco Elver]

I'll take a closer look at generated code. In some cases the compiler
ought to omit zero-sized arguments, so I want to be sure we're not
prematurely optimizing and the size increase is not some other effect.

> Something like this should work? (diff on top of this patch)

Thanks, I'll consider it.

Re your other comments:

> Assuming not all people building the kernel are security experts...
> (including myself) could you please add some insights/guidance on how to
> decide between RANDOM_KMALLOC_CACHES and TYPED_KMALLOC_CACHES?

You can find different arguments for either, and in the original RFC
that was part of the discussion. However, my biased view is that
type-based partitioning in general is the stronger security boundary.
Because it creates a deterministic separation; specifically isolating
pointer-containing objects from pointerless ones: it effectively kills
certain classes of exploit techniques that probabilistic defenses
(like randomization) only delay, especially in environments where
attackers can retry or use side-channels.

The current pointer/non-pointer scheme is relatively intuitive with
well-understood properties, and a good start. However, an open
research question is if better alloc-token ID schemes exist: one that
is tailored to kernel data structures that would meaningfully increase
exploitation difficulty further without increasing the number of
partitions. Since an improved scheme could simply be activated with a
compiler flag, having the baseline infrastructure available and
maintained is the first step.

> Now somewhat out-of-scope (or at least pre-existing) review comments
> from Sashiko that I think are still worth mentioning...

Indeed, these are pre-existing issues with RANDOM_KMALLOC_CACHES.
Worth follow-up patches, but this patch here wants to just get the
TYPED_KMALLOC_CACHES infrastructure in place so we can build on top of
it.

Thanks,
-- Marco

[Harry Yoo (Oracle)]

Oh, didn't know that was a thing.

Not sure if it's safe to do that for exported functions though (since
modules can be built w/ a different compiler).

> so I want to be sure we're not
> prematurely optimizing and the size increase is not some other effect.

Great, thanks for looking into it.

> > Something like this should work? (diff on top of this patch)
>
> Thanks, I'll consider it.

Thanks.

> Re your other comments:
>
> > Assuming not all people building the kernel are security experts...
> > (including myself) could you please add some insights/guidance on how to
> > decide between RANDOM_KMALLOC_CACHES and TYPED_KMALLOC_CACHES?
>
> You can find different arguments for either, and in the original RFC
> that was part of the discussion.

Yeah, I had fun time following the discussions :)

> However, my biased view is that type-based partitioning in general
> is the stronger security boundary.
> Because it creates a deterministic separation; specifically isolating
> pointer-containing objects from pointerless ones: it effectively kills
> certain classes of exploit techniques that probabilistic defenses
> (like randomization) only delay, especially in environments where
> attackers can retry or use side-channels.

That's a fair argument.

Could we possibly put some of those arguments (in a neutral/technical
sense) in help text to make people's lives easier?

Err, trying to provide unbiased (tm) help text...
Something like:

config RANDOM_KMALLOC_CACHES

bool "Randomize slab caches for normal kmalloc"

help
Randomly pick a slab cache based on code address and a per-boot
random seed.

This makes it harder for attackers to predict object co-location.
The placement is random: while attackers don't know which kmalloc
cache an object will be allocated from, they might circumvent
the randomization by retrying attacks across multiple machines until
the target objects are co-located.

config TYPED_KMALLOC_CACHES

bool "Type based slab cache selection for normal kmalloc"

depends on $(cc-option,-falloc-token-max=123)
help

Rely on Clang's allocation tokens to choose a slab cache, where token

IDs are derived from the allocated type.

Unlike RANDOM_KMALLOC_CACHES, the cache assignment is deterministic
and based on type, which guarantees that objects of certain types
are not placed in the same cache. This effectively mitigates
certain classes of exploits that probabilistic defenses like
RANDOM_KMALLOC_CACHES only make harder but not impossible.
However, this also means the cache assignment is predictable.

For example, a token ID scheme might separate pointer-containing
objects and pointerless objects to prevent buffer overflows on
primitive buffers from directly corrupting pointer-containing objects.

The current effectiveness of Clang's type inference can be judged by

-Rpass=alloc-token, which provides diagnostics where (after dead-code

elimination) type inference failed.

Requires Clang 22 or later.

endchoice

> The current pointer/non-pointer scheme is relatively intuitive with
> well-understood properties, and a good start.
>
> However, an open research question is if better alloc-token ID schemes exist:
> one that is tailored to kernel data structures that would meaningfully
> increase exploitation difficulty further without increasing the number of
> partitions.

Haha, that's too adventurous research question for me :P

> Since an improved scheme could simply be activated with a
> compiler flag, having the baseline infrastructure available and
> maintained is the first step.

Agreed.

> > Now somewhat out-of-scope (or at least pre-existing) review comments
> > from Sashiko that I think are still worth mentioning...
>
> Indeed, these are pre-existing issues with RANDOM_KMALLOC_CACHES.
> Worth follow-up patches, but this patch here wants to just get the
> TYPED_KMALLOC_CACHES infrastructure in place so we can build on top of
> it.

Yeah, that's totally fine!

[Marco Elver]

On Tue, 7 Apr 2026 at 14:54, 'Harry Yoo (Oracle)' via kasan-dev

So I checked with Clang and GCC. Looks like Clang does omit the
zero-sized struct argument, i.e. generated code is identical to
before. Whereas GCC wastes a few bytes of stack space at callsites.

Which is sad, because that means we need the macro workaround.

Do you want to be credited with Co-authored-by - in which case I need
your Signed-off-by.

> Not sure if it's safe to do that for exported functions though (since
> modules can be built w/ a different compiler).

Kernel modules built with a different config (implicit if different
compiler) are not supported, and never have been. If it works, it's
just luck (I know people do this, but it's just a disaster waiting to
happen).

Thanks,
-- Marco

[Harry Yoo (Oracle)]

Thanks for confirming that.

> Which is sad, because that means we need the macro workaround.
>
> Do you want to be credited with Co-authored-by

I'd appreciate that. (I guess you meant Co-developed-by)

> - in which case I need your Signed-off-by.

Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>

> > Not sure if it's safe to do that for exported functions though (since
> > modules can be built w/ a different compiler).
>
> Kernel modules built with a different config (implicit if different
> compiler) are not supported, and never have been. If it works, it's
> just luck (I know people do this, but it's just a disaster waiting to
> happen).

And if GCC folks somehow fix this at some point, even kernel modules
built with a different version of GCC might not be supported?

[Marco Elver]

On Fri, 10 Apr 2026 at 04:10, Harry Yoo (Oracle) <ha...@kernel.org> wrote:
[...]

> > Which is sad, because that means we need the macro workaround.
> >
> > Do you want to be credited with Co-authored-by
>
> I'd appreciate that. (I guess you meant Co-developed-by)

Thanks (and yes).

> > - in which case I need your Signed-off-by.
>
> Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>
>
> > > Not sure if it's safe to do that for exported functions though (since
> > > modules can be built w/ a different compiler).
> >
> > Kernel modules built with a different config (implicit if different
> > compiler) are not supported, and never have been. If it works, it's
> > just luck (I know people do this, but it's just a disaster waiting to
> > happen).
>
> And if GCC folks somehow fix this at some point, even kernel modules
> built with a different version of GCC might not be supported?

Yes, and different compiler version also implies different compiler
(so even GCC A.B != GCC X.Y), because newer compiler versions may
activate different codegen options (some of which we auto-enable if
available).

[GONG Ruiqi]

On 3/31/2026 7:12 PM, Marco Elver wrote:
> ...

> @@ -662,9 +662,20 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];
> (IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
> (IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))
>
> +#ifdef CONFIG_RANDOM_KMALLOC_CACHES
> extern unsigned long random_kmalloc_seed;
> +typedef struct { unsigned long ip; } kmalloc_token_t;
> +#define __kmalloc_token(...) ((kmalloc_token_t) { .ip = _RET_IP_ })
> +#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
> +typedef struct { unsigned long v; } kmalloc_token_t;
> +#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

One tiny suggestion: we could use the same name for kmalloc_token_t's
member in both cases, which would make the code a bit more concise.

Acked-by: GONG Ruiqi <gongr...@huawei.com>

> +#else
> +/* no-op */
> +typedef struct {} kmalloc_token_t;
> +#define __kmalloc_token(...) ((kmalloc_token_t){})
> +#endif
>
> -static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
> +static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
> {
> /*
> * The most common case is KMALLOC_NORMAL, so test for it
> @@ -672,9 +683,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
> */
> if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
> #ifdef CONFIG_RANDOM_KMALLOC_CACHES
> - /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
> - return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
> - ilog2(RANDOM_KMALLOC_CACHES_NR + 1));
> + /* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
> + return KMALLOC_PARTITION_START + hash_64(token.ip ^ random_kmalloc_seed,
> + ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
> +#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
> + return KMALLOC_PARTITION_START + token.v;
> #else
> return KMALLOC_NORMAL;
> #endif

> ...

[Marco Elver]

also does not prevent cross-cache attacks: while waiting for future
features like SLAB_VIRTUAL [3] to provide physical page isolation, this
feature should be deployed alongside SHUFFLE_PAGE_ALLOCATOR and
init_on_free=1 to mitigate cross-cache attacks and page-reuse attacks as
much as possible today.

With all that, my kernel (x86 defconfig) shows me a histogram of slab
cache object distribution per /proc/slabinfo (after boot):

<slab cache> <objs> <hist>

kmalloc-part-15 1465 ++++++++++++++
kmalloc-part-14 2988 +++++++++++++++++++++++++++++
kmalloc-part-13 1656 ++++++++++++++++
kmalloc-part-12 1045 ++++++++++
kmalloc-part-11 1697 ++++++++++++++++
kmalloc-part-10 1489 ++++++++++++++
kmalloc-part-09 965 +++++++++

kmalloc-part-08 710 +++++++
kmalloc-part-07 100 +
kmalloc-part-06 217 ++
kmalloc-part-05 105 +
kmalloc-part-04 4047 ++++++++++++++++++++++++++++++++++++++++

kmalloc-part-03 183 +

kmalloc-part-02 283 ++
kmalloc-part-01 316 +++

kmalloc 1422 ++++++++++++++

The above /proc/slabinfo snapshot shows me there are 6673 allocated

objects (slabs 00 - 07) that the compiler claims contain no pointers or

it was unable to infer the type of, and 12015 objects that contain

pointers (slabs 08 - 15). On a whole, this looks relatively sane.

Additionally, when I compile my kernel with -Rpass=alloc-token, which

provides diagnostics where (after dead-code elimination) type inference

failed, I see 186 allocation sites where the compiler failed to identify

a type (down from 966 when I sent the RFC [4]). Some initial review
confirms these are mostly variable sized buffers, but also include
structs with trailing flexible length arrays.

Link: https://clang.llvm.org/docs/AllocToken.html [1]
Link: https://blog.dfsec.com/ios/2025/05/30/blasting-past-ios-18/ [2]
Link: https://lwn.net/Articles/944647/ [3]
Link: https://lore.kernel.org/all/20250825154505....@google.com/ [4]
Link: https://discourse.llvm.org/t/rfc-a-framework-for-allocator-partitioning-hints/87434

Acked-by: GONG Ruiqi <gongr...@huawei.com>
Co-developed-by: Harry Yoo (Oracle) <ha...@kernel.org>

Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>

Signed-off-by: Marco Elver <el...@google.com>
---

v2:
* Avoid empty function argument if !PARTITION_KMALLOC_CACHES
(co-developed-by Harry). While Clang does optimize out the empty
struct argument (and generated code is identical to before if
PARTITION_KMALLOC_CACHES is disabled), GCC doesn't do so. So we need
to fully remove the argument if not actually required.
* Cover krealloc() which was missed before, resulting in ~100 additional
objects in the pointer-containing caches in above histogram.
* Unify kmalloc_token_t definition.
* Expand Kconfig help text.

v1: https://lore.kernel.org/all/20260331111240...@google.com/

* Rebase and switch to builtin name that was released in Clang 22.
* Keep RANDOM_KMALLOC_CACHES the default.

RFC: https://lore.kernel.org/all/20250825154505....@google.com/
---
Makefile | 5 ++
include/linux/percpu.h | 2 +-

include/linux/slab.h | 127 ++++++++++++++++++++------------
kernel/configs/hardening.config | 2 +-
mm/Kconfig | 65 +++++++++++++---

mm/kfence/kfence_test.c | 4 +-
mm/slab.h | 4 +-

mm/slab_common.c | 48 ++++++------
mm/slub.c | 54 +++++++-------
9 files changed, 200 insertions(+), 111 deletions(-)

diff --git a/Makefile b/Makefile
index 54e1ae602000..f70170ed1522 100644
--- a/Makefile
+++ b/Makefile
@@ -989,6 +989,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)

endif
endif

+ifdef CONFIG_TYPED_KMALLOC_CACHES
+# PARTITION_KMALLOC_CACHES_NR + 1
+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 85bf8dd9f087..271b41be314d 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..3bd0db0ec95f 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -499,14 +499,33 @@ int kmem_cache_shrink(struct kmem_cache *s);
.usersize = sizeof_field(struct __struct, __field), \
}, (__flags))

+#ifdef CONFIG_PARTITION_KMALLOC_CACHES

+typedef struct { unsigned long v; } kmalloc_token_t;

+#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+extern unsigned long random_kmalloc_seed;
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)

+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

+#endif

+#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
+#define _PASS_TOKEN_PARAM(_token) , (_token)
+#define PASS_TOKEN_PARAM(_token) (_token)

+#else /* !CONFIG_PARTITION_KMALLOC_CACHES */
+typedef struct {} kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){}) /* no-op */

+#define DECL_TOKEN_PARAM(_token)
+#define _PASS_TOKEN_PARAM(_token)
+#define PASS_TOKEN_PARAM(_token) ((kmalloc_token_t){})

+#endif /* CONFIG_PARTITION_KMALLOC_CACHES */
+
/*
* Common kmalloc functions provided by all allocators
*/
void * __must_check krealloc_node_align_noprof(const void *objp, size_t new_size,
unsigned long align,
- gfp_t flags, int nid) __realloc_size(2);
-#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE)
-#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__))
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token)) __realloc_size(2);
+#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE _PASS_TOKEN_PARAM(__kmalloc_token(_o, _s, _f)))
+#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__ _PASS_TOKEN_PARAM(__kmalloc_token(__VA_ARGS__))))
#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -612,10 +631,10 @@ static inline unsigned int arch_slab_minalign(void)

#define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \
(KMALLOC_MIN_SIZE) : 16)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define PARTITION_KMALLOC_CACHES_NR 15 // # of cache copies
#else
-#define RANDOM_KMALLOC_CACHES_NR 0
+#define PARTITION_KMALLOC_CACHES_NR 0
#endif

/*

@@ -634,8 +653,8 @@ enum kmalloc_cache_type {

#ifndef CONFIG_MEMCG
KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
- KMALLOC_RANDOM_START = KMALLOC_NORMAL,
- KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
+ KMALLOC_PARTITION_START = KMALLOC_NORMAL,
+ KMALLOC_PARTITION_END = KMALLOC_PARTITION_START + PARTITION_KMALLOC_CACHES_NR,
#ifdef CONFIG_SLUB_TINY
KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else

@@ -662,9 +681,7 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];

(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))

-extern unsigned long random_kmalloc_seed;
-

-static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
+static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
{
/*
* The most common case is KMALLOC_NORMAL, so test for it

@@ -672,9 +689,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne

*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
- return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
- ilog2(RANDOM_KMALLOC_CACHES_NR + 1));
+ /* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */

+ return KMALLOC_PARTITION_START + hash_64(token.v ^ random_kmalloc_seed,

+ ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+ return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;
#endif

@@ -858,16 +877,22 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
#define PASS_BUCKET_PARAM(_b) NULL
#endif

+#define DECL_KMALLOC_PARAMS(_size, _b, _token) DECL_BUCKET_PARAMS(_size, _b) \
+ DECL_TOKEN_PARAM(_token)
+
+#define PASS_KMALLOC_PARAMS(_size, _b, _token) PASS_BUCKET_PARAMS(_size, _b) \
+ _PASS_TOKEN_PARAM(_token)
+
/*
* The following functions are not to be used directly and are intended only
* for internal use from kmalloc() and kmalloc_node()

* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags)

+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)

__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)

+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

@@ -938,7 +963,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -948,14 +973,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],

+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}
- return __kmalloc_noprof(size, flags);

+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
}

+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node DECL_TOKEN_PARAM(token));
+#define kmalloc_nolock_noprof(...) _kmalloc_nolock_noprof(__VA_ARGS__ _PASS_TOKEN_PARAM(__kmalloc_token(__VA_ARGS__)))
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1087,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \

- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))

+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \

- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))

+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -1075,11 +1102,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],

+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);

+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);
}

+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**
@@ -1088,14 +1116,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)

+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kmalloc_noprof(bytes, flags);
+ return _kmalloc_noprof(bytes, flags, token);
}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**

@@ -1115,18 +1144,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz
* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/
-static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p,
+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
size_t new_n,
size_t new_size,
- gfp_t flags)
+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;

- return krealloc_noprof(p, bytes, flags);
+ return krealloc_node_align_noprof(p, bytes, 1, flags, NUMA_NO_NODE _PASS_TOKEN_PARAM(token));
}
+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))

/**
@@ -1137,10 +1167,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi

*/
#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,

+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node,

unsigned long caller) __alloc_size(1);
#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \

- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller)

+ __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(size, NULL, __kmalloc_token(size)), flags, node, caller)
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1157,17 +1187,18 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag

#define kmalloc_track_caller_noprof(...) \
kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_)

-static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
+ int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

if (__builtin_constant_p(n) && __builtin_constant_p(size))
- return kmalloc_node_noprof(bytes, flags, node);

- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node);
+ return _kmalloc_node_noprof(bytes, flags, node, token);

+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), flags, node);
}

+#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

#define kcalloc_node(_n, _size, _flags, _node) \

@@ -1183,39 +1214,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_

* @size: how many bytes of memory are required.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags)
+static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return kmalloc_noprof(size, flags | __GFP_ZERO);
+ return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}
+#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

gfp_t flags, int node) __alloc_size(1);
#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \

- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node)

+ __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)

+#define kvmalloc_node_noprof(size, flags, node) \
+ kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)
+#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE)
#define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO)

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \

- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE))

+ alloc_hooks(__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), 1, _flags, NUMA_NO_NODE))

static inline __alloc_size(1, 2) void *

-kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)

+_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kvmalloc_node_align_noprof(bytes, 1, flags, node);

+ return __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), 1, flags, node);
}

-
+#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)

#define kvcalloc_node_noprof(_n,_s,_f,_node) kvmalloc_array_node_noprof(_n,_s,(_f)|__GFP_ZERO,_node)
#define kvcalloc_noprof(...) kvcalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE)

@@ -1225,9 +1260,9 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
#define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__))

void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
- gfp_t flags, int nid) __realloc_size(2);
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token)) __realloc_size(2);
#define kvrealloc_node_align(...) \
- alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__))
+ alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__ _PASS_TOKEN_PARAM(__kmalloc_token(__VA_ARGS__))))
#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config
index 7c3924614e01..2963b6bd890f 100644
--- a/kernel/configs/hardening.config
+++ b/kernel/configs/hardening.config
@@ -22,7 +22,7 @@ CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
CONFIG_SLAB_BUCKETS=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-CONFIG_RANDOM_KMALLOC_CACHES=y
+CONFIG_PARTITION_KMALLOC_CACHES=y

# Sanity check userspace page table mappings.
CONFIG_PAGE_TABLE_CHECK=y

diff --git a/mm/Kconfig b/mm/Kconfig
index ebd8ea353687..47f323816de7 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -247,22 +247,67 @@ config SLUB_STATS

out which slabs are relevant to a particular load.
Try running: slabinfo -DA

-config RANDOM_KMALLOC_CACHES
- default n
+config PARTITION_KMALLOC_CACHES
depends on !SLUB_TINY

- bool "Randomize slab caches for normal kmalloc"
+ bool "Partitioned slab caches for normal kmalloc"
help

+ bool "Randomize slab caches for normal kmalloc"
+ help
+ Randomly pick a slab cache based on code address and a per-boot
+ random seed.
+
+ This makes it harder for attackers to predict object co-location.
+ The placement is random: while attackers don't know which kmalloc
+ cache an object will be allocated from, they might circumvent
+ the randomization by retrying attacks across multiple machines until
+ the target objects are co-located.

+
+config TYPED_KMALLOC_CACHES
+ bool "Type based slab cache selection for normal kmalloc"
+ depends on $(cc-option,-falloc-token-max=123)
+ help

+ Rely on Clang's allocation tokens to choose a slab cache, where token
+ IDs are derived from the allocated type.
+
+ Unlike RANDOM_KMALLOC_CACHES, cache assignment is deterministic based
+ on type, which guarantees that objects of certain types are not
+ placed in the same cache. This effectively mitigates certain classes
+ of exploits that probabilistic defenses like RANDOM_KMALLOC_CACHES
+ only make harder but not impossible. However, this also means the
+ cache assignment is predictable.
+
+ Clang's default token ID calculation returns a bounded hash with
+ disjoint ranges for pointer-containing and pointerless objects: when
+ used as the slab cache index, this prevents buffer overflows on
+ primitive buffers from directly corrupting pointer-containing
+ objects.

+
+ The current effectiveness of Clang's type inference can be judged by
+ -Rpass=alloc-token, which provides diagnostics where (after dead-code
+ elimination) type inference failed.
+

diff --git a/mm/slub.c b/mm/slub.c
index 2b2d33cc735c..f033aae90bdc 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2125,11 +2125,11 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,

if (!is_kmalloc_normal(s))
return sz;

- obj_exts_cache = kmalloc_slab(sz, NULL, gfp, 0);
+ obj_exts_cache = kmalloc_slab(sz, NULL, gfp, __kmalloc_token(0));
/*

- * We can't simply compare s with obj_exts_cache, because random kmalloc
- * caches have multiple caches per size, selected by caller address.
- * Since caller address may differ between kmalloc_slab() and actual
+ * We can't simply compare s with obj_exts_cache, because partitioned kmalloc
+ * caches have multiple caches per size, selected by caller address or type.
+ * Since caller address or type may differ between kmalloc_slab() and actual
* allocation, bump size when sizes are equal.
*/
if (s->object_size == obj_exts_cache->object_size)

@@ -5239,7 +5239,7 @@ EXPORT_SYMBOL(__kmalloc_large_node_noprof);

static __always_inline

void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

- unsigned long caller)
+ unsigned long caller, kmalloc_token_t token)
{
struct kmem_cache *s;
void *ret;

@@ -5254,22 +5254,24 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

if (unlikely(!size))
return ZERO_SIZE_PTR;

- s = kmalloc_slab(size, b, flags, caller);
+ s = kmalloc_slab(size, b, flags, token);

ret = slab_alloc_node(s, NULL, flags, node, caller, size);
ret = kasan_kmalloc(s, ret, size, flags);

trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)

+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
{

- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags)

+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
{

- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,
+ NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5284,7 +5286,7 @@ EXPORT_SYMBOL(__kmalloc_noprof);

* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
* There is no reason to call it again and expect !NULL.
*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node DECL_TOKEN_PARAM(token))

{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
struct kmem_cache *s;

@@ -5307,7 +5309,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);

+ s = kmalloc_slab(size, NULL, alloc_gfp, PASS_TOKEN_PARAM(token));

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*

@@ -5360,12 +5362,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,

+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,

int node, unsigned long caller)
{

- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);

@@ -6555,7 +6558,7 @@ void kfree_nolock(const void *object)
EXPORT_SYMBOL_GPL(kfree_nolock);

static __always_inline __realloc_size(2) void *
-__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid)
+__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid, kmalloc_token_t token)
{
void *ret;
size_t ks = 0;
@@ -6627,7 +6630,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
return (void *)p;

alloc_new:
- ret = kmalloc_node_track_caller_noprof(new_size, flags, nid, _RET_IP_);
+ ret = __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(new_size, NULL, token), flags, nid, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();
@@ -6677,7 +6680,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
* Return: pointer to the allocated memory or %NULL in case of error
*/
void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long align,
- gfp_t flags, int nid)
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token))
{
void *ret;

@@ -6686,7 +6689,7 @@ void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long a
return ZERO_SIZE_PTR;
}

- ret = __do_krealloc(p, new_size, align, flags, nid);
+ ret = __do_krealloc(p, new_size, align, flags, nid, PASS_TOKEN_PARAM(token));
if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret))
kfree(p);

@@ -6723,6 +6726,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
* failure, fall back to non-contiguous (vmalloc) allocation.
* @size: size of the request.
* @b: which set of kmalloc buckets to allocate from.

+ * @token: allocation token.
* @align: desired alignment.
* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
* @node: numa node to allocate from

@@ -6739,7 +6743,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

*
* Return: pointer to the allocated memory of %NULL in case of failure
*/
-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

gfp_t flags, int node)
{
bool allow_block;

@@ -6751,7 +6755,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),

- node, _RET_IP_);

+ node, _RET_IP_, PASS_TOKEN_PARAM(token));
if (ret || size <= PAGE_SIZE)
return ret;

@@ -6848,17 +6852,17 @@ EXPORT_SYMBOL(kvfree_sensitive);
* Return: pointer to the allocated memory or %NULL in case of error
*/
void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
- gfp_t flags, int nid)
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token))
{
void *n;

if (is_vmalloc_addr(p))
return vrealloc_node_align_noprof(p, size, align, flags, nid);

- n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid);
+ n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid _PASS_TOKEN_PARAM(token));
if (!n) {
/* We failed to krealloc(), fall back to kvmalloc(). */
- n = kvmalloc_node_align_noprof(size, align, flags, nid);
+ n = __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), align, flags, nid);
if (!n)
return NULL;

@@ -8351,7 +8355,7 @@ static void __init bootstrap_kmalloc_sheaves(void)

{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--

2.54.0.rc1.513.gad8abe7a5a-goog

[Marco Elver]

Sashiko found 2 latent issues in the diff context :
https://sashiko.dev/#/patchset/20260415143735.2974230-1-elver%40google.com
The fix is here:
https://lore.kernel.org/all/20260416132837....@google.com/

The irony is that TYPED_KMALLOC_CACHES would have helped mitigate this
kind of overflow bug.

[Harry Yoo (Oracle)]

[CC'ing RUST ALLOC folks for rust bindings]

Thanks. I find the help text much more useful.

>
> v1: https://lore.kernel.org/all/20260331111240...@google.com/
> * Rebase and switch to builtin name that was released in Clang 22.
> * Keep RANDOM_KMALLOC_CACHES the default.
>
> RFC: https://lore.kernel.org/all/20250825154505....@google.com/
> ---

A few comments on V2:

# comment 1

I'm not a big fan of how k[v]realloc_node_align()
and kmalloc_nolock() define and pass the token parameter.

IMHO it'll be fine to use {DECL,PASS}_KMALLOC_PARAMS() in those
functions, since SLAB_BUCKETS users already passes NULL bucket
to most of __kmalloc*() calls anyway.

# comment 2

This breaks Documentation/.

Problems:

- The document generator doesn't handle DECL_KMALLOC_PARAMS() well.

- The signature of the function that users call (krealloc_node_align())
and the function that has kerneldoc (krealloc_node_align_noprof())
don't match.

- Even worse, moving kerneldoc to the macro doesn't work because
it uses variable arguments (...)

# comment 3

Looking at how rust generates helper functions,
in rust/helpers/slab.c:
| // SPDX-License-Identifier: GPL-2.0
|
| #include <linux/slab.h>
|
| __rust_helper void *__must_check __realloc_size(2)
| rust_helper_krealloc_node_align(const void *objp, size_t new_size, unsigned long align,
| gfp_t flags, int node)
| {
| return krealloc_node_align(objp, new_size, align, flags, node);
| }
|
| __rust_helper void *__must_check __realloc_size(2)
| rust_helper_kvrealloc_node_align(const void *p, size_t size, unsigned long align,
| gfp_t flags, int node)
| {
| return kvrealloc_node_align(p, size, align, flags, node);
| }

Rust code probably won't pass any meaningful token?
(something you may want to address in the future)

--
Cheers,
Harry / Hyeonggon

[Not trimming the rest of the patch for those added to Cc]

[Marco Elver]

I'm not sure I agree. 2 reasons:

1. Even though it's "just" k[v]realloc_node_align() and
kmalloc_nolock() - despite their relatively less frequent use - just
put one of them in a hot path and you're sacrificing performance even
further. There are only so many arguments that can be passed in
registers (depending on arch), and may cause more stack spilling.

2. We'd misleadingly declare that these functions do something with
the bucket arg. This is wrong.

Both feels wrong, and would only make this change if you confirm both
are trade-offs that you strongly prefer.

> # comment 2
>
> This breaks Documentation/.
>
> Problems:
>
> - The document generator doesn't handle DECL_KMALLOC_PARAMS() well.
>
> - The signature of the function that users call (krealloc_node_align())
> and the function that has kerneldoc (krealloc_node_align_noprof())
> don't match.
>
> - Even worse, moving kerneldoc to the macro doesn't work because
> it uses variable arguments (...)

Well, some were broken before, now it's just broken more. :-)

We could move the documentation to macros and switch to explicit args
instead of (...).

Otherwise, I don't see any way to fix this. Preferences?

> # comment 3
>
> Looking at how rust generates helper functions,
> in rust/helpers/slab.c:
> | // SPDX-License-Identifier: GPL-2.0
> |
> | #include <linux/slab.h>
> |
> | __rust_helper void *__must_check __realloc_size(2)
> | rust_helper_krealloc_node_align(const void *objp, size_t new_size, unsigned long align,
> | gfp_t flags, int node)
> | {
> | return krealloc_node_align(objp, new_size, align, flags, node);
> | }
> |
> | __rust_helper void *__must_check __realloc_size(2)
> | rust_helper_kvrealloc_node_align(const void *p, size_t size, unsigned long align,
> | gfp_t flags, int node)
> | {
> | return kvrealloc_node_align(p, size, align, flags, node);
> | }
>
> Rust code probably won't pass any meaningful token?
> (something you may want to address in the future)

Yes, it'll just pass '0' by default. We could force Rust's allocation
to be in the pointer-containing range - if we assume Rust code is less
prone to contain bugs, but the assumption is that such allocations
both originate and are confined to the Rust side. One easy way to do
this is to write:

return kvrealloc_node_align(p, size + 0 * sizeof(long*), align,
flags, node);

But I'd defer that for now, until we're sure the above assumption
holds (Rust originated + confined).

[Harry Yoo (Oracle)]

On Mon, Apr 20, 2026 at 11:30:23AM +0200, Marco Elver wrote:
> On Mon, 20 Apr 2026 at 09:25, Harry Yoo (Oracle) <ha...@kernel.org> wrote:
> > [CC'ing RUST ALLOC folks for rust bindings]
> > On Wed, Apr 15, 2026 at 04:37:05PM +0200, Marco Elver wrote:
> > A few comments on V2:
> >
> > # comment 1
> >
> > I'm not a big fan of how k[v]realloc_node_align()
> > and kmalloc_nolock() define and pass the token parameter.
> >
> > IMHO it'll be fine to use {DECL,PASS}_KMALLOC_PARAMS() in those
> > functions, since SLAB_BUCKETS users already passes NULL bucket
> > to most of __kmalloc*() calls anyway.
>
> I'm not sure I agree. 2 reasons:
>
> 1. Even though it's "just" k[v]realloc_node_align() and
> kmalloc_nolock() - despite their relatively less frequent use - just
> put one of them in a hot path and you're sacrificing performance even
> further. There are only so many arguments that can be passed in
> registers (depending on arch), and may cause more stack spilling.
>
> 2. We'd misleadingly declare that these functions do something with
> the bucket arg. This is wrong.

Both are valid points. But it still feels wrong to have:

void *krealloc_node_align_noprof(const void *objp, size_t new_size,
unsigned long align,
gfp_t flags, int nid DECL_TOKEN_PARAM(token));

n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid _PASS_TOKEN_PARAM(token));

Actually the problem here is that some of parameters in
DECL_KMALLOC_PARAMS() are not necessary in some functions.

Perhaps we could have

DECL_KMALLOC_PARAMS(size, b, token) # declare size, bucket, token

DECL_BUCKET_PARAMS(size, token) # declare size, bucket;
# but, actually, we don't need this!

DECL_TOKEN_PARAMS(size, b) # declare size, token only;
# for kmalloc_nolock and k[v]realloc_node_align()

and use DECL_TOKEN_PARAMS(), PASS_TOKEN_PARAMS() for those functions?
(just like how DECL_BUCKET_PARAMS() worked before)

What do you think?

> Both feels wrong, and would only make this change if you confirm both
> are trade-offs that you strongly prefer.
>
> > # comment 2
> >
> > This breaks Documentation/.
> >
> > Problems:
> >
> > - The document generator doesn't handle DECL_KMALLOC_PARAMS() well.
> >
> > - The signature of the function that users call (krealloc_node_align())
> > and the function that has kerneldoc (krealloc_node_align_noprof())
> > don't match.
> >
> > - Even worse, moving kerneldoc to the macro doesn't work because
> > it uses variable arguments (...)
>
> Well, some were broken before, now it's just broken more. :-)

Ouch... ;-)

> We could move the documentation to macros and switch to explicit args
> instead of (...).

That works for me!

Ack.

By the way, since Allocator trait uses realloc() to allocate new memory,
IIUC all k[v]malloc, k[v]realloc usage from Rust will be affected.

[Kees Cook]

On Wed, Apr 15, 2026 at 04:37:05PM +0200, Marco Elver wrote:

> The builtin __builtin_infer_alloc_token(<malloc-args>, ...) instructs
> the compiler to infer an allocation type from arguments commonly passed
> to memory-allocating functions and returns a type-derived token ID. The
> implementation passes kmalloc-args to the builtin: the compiler performs
> best-effort type inference, and then recognizes common patterns such as
> `kmalloc(sizeof(T), ...)`, `kmalloc(sizeof(T) * n, ...)`, but also
> `(T *)kmalloc(...)`. Where the compiler fails to infer a type the
> fallback token (default: 0) is chosen.
>
> Note: kmalloc_obj(..) APIs fix the pattern how size and result type are
> expressed, and therefore ensures there's not much drift in which
> patterns the compiler needs to recognize. Specifically, kmalloc_obj()
> and friends expand to `(TYPE *)KMALLOC(__obj_size, GFP)`, which the
> compiler recognizes via the cast to TYPE*.

Great! I'm glad this gets deterministically handled for the kmalloc_obj*
cases.

> Additionally, when I compile my kernel with -Rpass=alloc-token, which
> provides diagnostics where (after dead-code elimination) type inference
> failed, I see 186 allocation sites where the compiler failed to identify
> a type (down from 966 when I sent the RFC [4]). Some initial review
> confirms these are mostly variable sized buffers, but also include
> structs with trailing flexible length arrays.

For the call-site-partitioning series[1] I sent before, I had
per-site caches for fixed-sized allocations and size bucket caches for
variably-sized allocations. I'd like to see something similar for this
series. Specifically, I replaced "kmalloc_slab" with "choose_slab" that
did O(1) to find the dedicated cache/bucket for the allocation[2].

In this case, we now have a build-time-constant value that it should be
possible to use to look up a _single_ dedicated cache/bucket for the
given unique type: there is no need to do hashing.

> [...]

> -config RANDOM_KMALLOC_CACHES
> - default n
> +config PARTITION_KMALLOC_CACHES
> depends on !SLUB_TINY
> - bool "Randomize slab caches for normal kmalloc"
> + bool "Partitioned slab caches for normal kmalloc"
> help
> - A hardening feature that creates multiple copies of slab caches for
> - normal kmalloc allocation and makes kmalloc randomly pick one based
> - on code address, which makes the attackers more difficult to spray
> - vulnerable memory objects on the heap for the purpose of exploiting
> - memory vulnerabilities.
> + A hardening feature that creates multiple isolated copies of slab
> + caches for normal kmalloc allocations. This makes it more difficult
> + to exploit memory-safety vulnerabilities by attacking vulnerable
> + co-located memory objects. Several modes are provided.
>
> Currently the number of copies is set to 16, a reasonably large value

The "16" buckets seems to hold for TYPED_KMALLOC_CACHES too? My goal
with the earlier type-partitioning was to get _total_ isolation, not
simply bucketed: 1 cache (or sizes-bucket) for each type. The "16"
limitation from RANDOM_KMALLOC_CACHES was kind of arbitrary due to the
hashing.

> that effectively diverges the memory objects allocated for different
> subsystems or modules into different caches, at the expense of a
> - limited degree of memory and CPU overhead that relates to hardware and
> - system workload.
> + limited degree of memory and CPU overhead that relates to hardware
> + and system workload.
> +
> +choice
> + prompt "Partitioned slab cache mode"
> + depends on PARTITION_KMALLOC_CACHES
> + default RANDOM_KMALLOC_CACHES

I think this should be adjusted a bit:

config CC_HAS_ALLOC_TOKEN
def_bool $(cc-option,-falloc-token-max=123)

...
choice

prompt "Partitioned slab cache mode"

depends on PARTITION_KMALLOC_CACHES
default TYPED_KMALLOC_CACHES if CC_HAS_ALLOC_TOKEN
default RANDOM_KMALLOC_CACHES

And actually, perhaps a global rename of the options so the selection
naming is at the end of the CONFIG phrase, and bundle the on/off into
the choice:


choice

prompt "Partitioned slab cache mode"

depends on PARTITION_KMALLOC_CACHES
default KMALLOC_PARTITION_TYPED if !SLUB_TINY && CC_HAS_ALLOC_TOKEN
default KMALLOC_PARTITION_RANDOM if !SLUB_TINY
default KMALLOC_PARTITION_NONE

config KMALLOC_PARTITION_NONE
...
config KMALLOC_PARTITION_RANDOM
depends on !SLUB_TINY
...
config KMALLOC_PARTITION_TYPED
depends on !SLUB_TINY && CC_HAS_ALLOC_TOKEN

-Kees

[1] https://lore.kernel.org/lkml/20240809072532...@kernel.org/
[2] https://lore.kernel.org/lkml/20240809073309...@kernel.org/

--
Kees Cook

[Marco Elver]

That should be a separate series; I know what you're getting at, but
it's a significant rework and a different design with different
properties. This simpler patch is likely ready for the next merge
window (once I send v3), and in light of recent developments, I'd like
this to land sooner than later.

> > [...]
> > -config RANDOM_KMALLOC_CACHES
> > - default n
> > +config PARTITION_KMALLOC_CACHES
> > depends on !SLUB_TINY
> > - bool "Randomize slab caches for normal kmalloc"
> > + bool "Partitioned slab caches for normal kmalloc"
> > help
> > - A hardening feature that creates multiple copies of slab caches for
> > - normal kmalloc allocation and makes kmalloc randomly pick one based
> > - on code address, which makes the attackers more difficult to spray
> > - vulnerable memory objects on the heap for the purpose of exploiting
> > - memory vulnerabilities.
> > + A hardening feature that creates multiple isolated copies of slab
> > + caches for normal kmalloc allocations. This makes it more difficult
> > + to exploit memory-safety vulnerabilities by attacking vulnerable
> > + co-located memory objects. Several modes are provided.
> >
> > Currently the number of copies is set to 16, a reasonably large value
>
> The "16" buckets seems to hold for TYPED_KMALLOC_CACHES too? My goal
> with the earlier type-partitioning was to get _total_ isolation, not
> simply bucketed: 1 cache (or sizes-bucket) for each type. The "16"
> limitation from RANDOM_KMALLOC_CACHES was kind of arbitrary due to the
> hashing.

The token ID is also a hash, although it can be configured to be
unbounded to effectively give unique hash per type. As-is, limiting to
16 keeps it comparable to the RANDOM mode, albeit IMHO with better
isolation properties with the same overheads. As-is, performance
properties of RANDOM and TYPED are comparable, and the friction to
switch between them is minimal.

Unlike a completely new design, which will have comletely different
performance and memory usage properties - and wouldn't be comparable.

> > that effectively diverges the memory objects allocated for different
> > subsystems or modules into different caches, at the expense of a
> > - limited degree of memory and CPU overhead that relates to hardware and
> > - system workload.
> > + limited degree of memory and CPU overhead that relates to hardware
> > + and system workload.
> > +
> > +choice
> > + prompt "Partitioned slab cache mode"
> > + depends on PARTITION_KMALLOC_CACHES
> > + default RANDOM_KMALLOC_CACHES
>
> I think this should be adjusted a bit:
>
> config CC_HAS_ALLOC_TOKEN
> def_bool $(cc-option,-falloc-token-max=123)
>
> ...
> choice
> prompt "Partitioned slab cache mode"
> depends on PARTITION_KMALLOC_CACHES
> default TYPED_KMALLOC_CACHES if CC_HAS_ALLOC_TOKEN
> default RANDOM_KMALLOC_CACHES

Sure.

> And actually, perhaps a global rename of the options so the selection
> naming is at the end of the CONFIG phrase, and bundle the on/off into
> the choice:
>
>
> choice
> prompt "Partitioned slab cache mode"
> depends on PARTITION_KMALLOC_CACHES
> default KMALLOC_PARTITION_TYPED if !SLUB_TINY && CC_HAS_ALLOC_TOKEN
> default KMALLOC_PARTITION_RANDOM if !SLUB_TINY
> default KMALLOC_PARTITION_NONE
>
> config KMALLOC_PARTITION_NONE
> ...
> config KMALLOC_PARTITION_RANDOM
> depends on !SLUB_TINY
> ...
> config KMALLOC_PARTITION_TYPED
> depends on !SLUB_TINY && CC_HAS_ALLOC_TOKEN

There was a comment somewhere else that even introducing
PARTITION_KMALLOC_CACHES might confuse users of RANDOM_KMALLOC_CACHES.
I think completely getting rid of and renaming RANDOM_KMALLOC_CACHES
has marginal benefit, and will cause friction for existing users (even
moreso than already). I see little benefit here, and would prefer not
to break user configs more than needed: configs that already set
RANDOM_KMALLOC_CACHES, upon rebuild will be prompted to enable
PARTITION_KMALLOC_CACHES; if user says Y, then their previous
selection (RANDOM) would already be picked and they don't have to
rediscover that it exists under a new name.

I can make this change, but only if you're sure the benefit outweighs
the downsides here.

Thanks,
-- Marco

[Marco Elver]

On Tue, Apr 21, 2026 at 09:13PM +0200, Marco Elver wrote:
[...]

Upon further reflection, since the transition isn't smooth anyway, I'm
probably going to rename, but have them all use the PARTITION_KMALLOC_*
prefix so it's easy to just search for "CONFIG_PARTITION_KMALLOC_". I
don't see the need for a "NONE" variant - I've seen this pattern
elsewhere, and then you end up with users reading the .config and
concluding "CONFIG_PARTITION_KMALLOC_CACHES is enabled ... but oh never
mind actually it isn't" which I find confusing. This could be useful if
we had a dynamic on/off toggle and the default is NONE, but that's not
the case here.

diff --git a/Makefile b/Makefile
index f70170ed1522..d1f63ffb85f0 100644
--- a/Makefile
+++ b/Makefile
@@ -989,7 +989,7 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)
endif
endif

-ifdef CONFIG_TYPED_KMALLOC_CACHES
+ifdef CONFIG_PARTITION_KMALLOC_TYPED
# PARTITION_KMALLOC_CACHES_NR + 1
KBUILD_CFLAGS += -falloc-token-max=16
endif
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 0537c3596163..b46300037ca5 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -501,10 +501,10 @@ int kmem_cache_shrink(struct kmem_cache *s);

#ifdef CONFIG_PARTITION_KMALLOC_CACHES

typedef struct { unsigned long v; } kmalloc_token_t;

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM
extern unsigned long random_kmalloc_seed;

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })

-#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+#elif defined(CONFIG_PARTITION_KMALLOC_TYPED)

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

#endif
#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
@@ -732,11 +732,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc
* with a single branch for all the relevant flags.

*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM

/* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */

return KMALLOC_PARTITION_START + hash_64(token.v ^ random_kmalloc_seed,

ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
-#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+#elif defined(CONFIG_PARTITION_KMALLOC_TYPED)

return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;

diff --git a/mm/Kconfig b/mm/Kconfig
index 6d44bd2633bb..d8510913fbeb 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -265,12 +265,12 @@ config PARTITION_KMALLOC_CACHES

choice
prompt "Partitioned slab cache mode"
depends on PARTITION_KMALLOC_CACHES

- default TYPED_KMALLOC_CACHES if CC_HAS_ALLOC_TOKEN
- default RANDOM_KMALLOC_CACHES
+ default PARTITION_KMALLOC_TYPED if CC_HAS_ALLOC_TOKEN
+ default PARTITION_KMALLOC_RANDOM
help

Selects the slab cache partitioning mode.

-config RANDOM_KMALLOC_CACHES
+config PARTITION_KMALLOC_RANDOM

bool "Randomize slab caches for normal kmalloc"

help

Randomly pick a slab cache based on code address and a per-boot

@@ -282,17 +282,17 @@ config RANDOM_KMALLOC_CACHES

the randomization by retrying attacks across multiple machines until

the target objects are co-located.

-config TYPED_KMALLOC_CACHES
+config PARTITION_KMALLOC_TYPED

bool "Type based slab cache selection for normal kmalloc"

depends on CC_HAS_ALLOC_TOKEN
help

Rely on Clang's allocation tokens to choose a slab cache, where token

IDs are derived from the allocated type.

- Unlike RANDOM_KMALLOC_CACHES, cache assignment is deterministic based
+ Unlike PARTITION_KMALLOC_RANDOM, cache assignment is deterministic based

on type, which guarantees that objects of certain types are not

placed in the same cache. This effectively mitigates certain classes

- of exploits that probabilistic defenses like RANDOM_KMALLOC_CACHES
+ of exploits that probabilistic defenses like PARTITION_KMALLOC_RANDOM

only make harder but not impossible. However, this also means the

cache assignment is predictable.

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 21ab7dd79b5e..ca5e2a6d4e46 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -742,7 +742,7 @@ kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init =
{ /* initialization for https://llvm.org/pr42570 */ };
EXPORT_SYMBOL(kmalloc_caches);

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM
unsigned long random_kmalloc_seed __ro_after_init;
EXPORT_SYMBOL(random_kmalloc_seed);
#endif
@@ -1010,7 +1010,7 @@ void __init create_kmalloc_caches(void)
for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
new_kmalloc_cache(i, type);
}
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM
random_kmalloc_seed = get_random_u64();
#endif

[Kees Cook]

There is now the "transitional" Kconfig keyword too, which should make it easy to move to the new name.

For the naming, I tend to prefer the noun-verb (e.g. timer_start()) and noun-feature-option ordering (e.g. CONFIG_SLAB_FREELIST_RANDOM). This feature isn't about partition tables, so I don't think the first word should be "partition". :)

As far as doing the full isolation later, yeah, that's fine, and I think with this Kconfig change it's easy to add a new mode.

-Kees

--
Kees Cook

[Marco Elver]

Rework the general infrastructure around RANDOM_KMALLOC_CACHES into more

flexible KMALLOC_PARTITION_CACHES, with the former being a partitioning
mode of the latter.

Introduce a new mode, KMALLOC_PARTITION_TYPED, which leverages a feature

available in Clang 22 and later, called "allocation tokens" via

__builtin_infer_alloc_token() [1]. Unlike KMALLOC_PARTITION_RANDOM
(formerly RANDOM_KMALLOC_CACHES), this mode deterministically assigns a

slab cache to an allocation of type T, regardless of allocation site.

The builtin __builtin_infer_alloc_token(<malloc-args>, ...) instructs
the compiler to infer an allocation type from arguments commonly passed
to memory-allocating functions and returns a type-derived token ID. The
implementation passes kmalloc-args to the builtin: the compiler performs
best-effort type inference, and then recognizes common patterns such as
`kmalloc(sizeof(T), ...)`, `kmalloc(sizeof(T) * n, ...)`, but also
`(T *)kmalloc(...)`. Where the compiler fails to infer a type the
fallback token (default: 0) is chosen.

Note: kmalloc_obj(..) APIs fix the pattern how size and result type are
expressed, and therefore ensures there's not much drift in which
patterns the compiler needs to recognize. Specifically, kmalloc_obj()
and friends expand to `(TYPE *)KMALLOC(__obj_size, GFP)`, which the
compiler recognizes via the cast to TYPE*.

Additionally, when I compile my kernel with -Rpass=alloc-token, which
provides diagnostics where (after dead-code elimination) type inference
failed, I see 186 allocation sites where the compiler failed to identify
a type (down from 966 when I sent the RFC [4]). Some initial review
confirms these are mostly variable sized buffers, but also include
structs with trailing flexible length arrays.

Link: https://clang.llvm.org/docs/AllocToken.html [1]
Link: https://blog.dfsec.com/ios/2025/05/30/blasting-past-ios-18/ [2]
Link: https://lwn.net/Articles/944647/ [3]
Link: https://lore.kernel.org/all/20250825154505....@google.com/ [4]
Link: https://discourse.llvm.org/t/rfc-a-framework-for-allocator-partitioning-hints/87434
Acked-by: GONG Ruiqi <gongr...@huawei.com>
Co-developed-by: Harry Yoo (Oracle) <ha...@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>
Signed-off-by: Marco Elver <el...@google.com>
---

v3:
* Introduce DECL_TOKEN_PARAMS / PASS_TOKEN_PARAMS (suggested-by Harry).
* Rename {RANDOM,TYPED}_KMALLOC_CACHES to KMALLOC_PARTITION_{RANDOM,TYPED},
so it's more easily searchable and consistent with the top-level name
with the variant at the end of the CONFIG phrase (reported-by Kees).
* Make KMALLOC_PARTITION_TYPED default if available (suggested-by Kees).
* Add RANDOM_KMALLOC_CACHES transitional option (suggested-by Kees).
* Add CC_HAS_ALLOC_TOKEN and use it (reported-by Kees).

v2: https://lore.kernel.org/all/20260415143735....@google.com/
* Avoid empty function argument if !KMALLOC_PARTITION_CACHES

(co-developed-by Harry). While Clang does optimize out the empty
struct argument (and generated code is identical to before if

KMALLOC_PARTITION_CACHES is disabled), GCC doesn't do so. So we need

to fully remove the argument if not actually required.
* Cover krealloc() which was missed before, resulting in ~100 additional
objects in the pointer-containing caches in above histogram.
* Unify kmalloc_token_t definition.
* Expand Kconfig help text.

v1: https://lore.kernel.org/all/20260331111240...@google.com/
* Rebase and switch to builtin name that was released in Clang 22.
* Keep RANDOM_KMALLOC_CACHES the default.

RFC: https://lore.kernel.org/all/20250825154505....@google.com/
---

Makefile | 5 ++
include/linux/percpu.h | 2 +-

include/linux/slab.h | 136 +++++++++++++++++++++-----------
init/Kconfig | 3 +
kernel/configs/hardening.config | 2 +-
mm/Kconfig | 73 ++++++++++++++---

mm/kfence/kfence_test.c | 4 +-
mm/slab.h | 4 +-

mm/slab_common.c | 52 ++++++------
mm/slub.c | 54 +++++++------
10 files changed, 220 insertions(+), 115 deletions(-)

diff --git a/Makefile b/Makefile
index 54e1ae602000..afe663bdb9c0 100644
--- a/Makefile
+++ b/Makefile
@@ -989,6 +989,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)
endif
endif

+ifdef CONFIG_KMALLOC_PARTITION_TYPED
+# KMALLOC_PARTITION_CACHES_NR + 1

+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h

index 85bf8dd9f087..bdb721dac0e3 100644

--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES

# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..c232f8a10af6 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -499,14 +499,38 @@ int kmem_cache_shrink(struct kmem_cache *s);

.usersize = sizeof_field(struct __struct, __field), \
}, (__flags))

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES
+typedef struct { unsigned long v; } kmalloc_token_t;
+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
+extern unsigned long random_kmalloc_seed;
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
+#elif defined(CONFIG_KMALLOC_PARTITION_TYPED)

+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })
+#endif
+#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
+#define _PASS_TOKEN_PARAM(_token) , (_token)
+#define PASS_TOKEN_PARAM(_token) (_token)

+#define DECL_TOKEN_PARAMS(_size, _token) size_t (_size), kmalloc_token_t (_token)
+#define PASS_TOKEN_PARAMS(_size, _token) (_size), (_token)
+#else /* !CONFIG_KMALLOC_PARTITION_CACHES */

+typedef struct {} kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){}) /* no-op */
+#define DECL_TOKEN_PARAM(_token)
+#define _PASS_TOKEN_PARAM(_token)
+#define PASS_TOKEN_PARAM(_token) ((kmalloc_token_t){})

+#define DECL_TOKEN_PARAMS(_size, _token) size_t (_size)
+#define PASS_TOKEN_PARAMS(_size, _token) (_size)
+#endif /* CONFIG_KMALLOC_PARTITION_CACHES */

+
/*
* Common kmalloc functions provided by all allocators
*/

-void * __must_check krealloc_node_align_noprof(const void *objp, size_t new_size,
+void * __must_check krealloc_node_align_noprof(const void *objp,
+ DECL_TOKEN_PARAMS(new_size, token),
unsigned long align,

gfp_t flags, int nid) __realloc_size(2);
-#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE)
-#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__))

+#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)
+#define krealloc_node_align(_o, _s, _a, _f, _n) alloc_hooks(krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -612,10 +636,10 @@ static inline unsigned int arch_slab_minalign(void)

#define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \
(KMALLOC_MIN_SIZE) : 16)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES
+#define KMALLOC_PARTITION_CACHES_NR 15 // # of cache copies
#else
-#define RANDOM_KMALLOC_CACHES_NR 0
+#define KMALLOC_PARTITION_CACHES_NR 0
#endif

/*
@@ -634,8 +658,8 @@ enum kmalloc_cache_type {

#ifndef CONFIG_MEMCG
KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
- KMALLOC_RANDOM_START = KMALLOC_NORMAL,
- KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
+ KMALLOC_PARTITION_START = KMALLOC_NORMAL,

+ KMALLOC_PARTITION_END = KMALLOC_PARTITION_START + KMALLOC_PARTITION_CACHES_NR,

#ifdef CONFIG_SLUB_TINY
KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else

@@ -662,19 +686,19 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];

(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))

-extern unsigned long random_kmalloc_seed;
-
-static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
+static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
{
/*
* The most common case is KMALLOC_NORMAL, so test for it

* with a single branch for all the relevant flags.
*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

- /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
- return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
- ilog2(RANDOM_KMALLOC_CACHES_NR + 1));

+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
+ /* KMALLOC_PARTITION_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
+ return KMALLOC_PARTITION_START + hash_64(token.v ^ random_kmalloc_seed,
+ ilog2(KMALLOC_PARTITION_CACHES_NR + 1));
+#elif defined(CONFIG_KMALLOC_PARTITION_TYPED)
+ return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;
#endif
@@ -858,16 +882,22 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);

#define PASS_BUCKET_PARAM(_b) NULL
#endif

+#define DECL_KMALLOC_PARAMS(_size, _b, _token) DECL_BUCKET_PARAMS(_size, _b) \
+ DECL_TOKEN_PARAM(_token)
+
+#define PASS_KMALLOC_PARAMS(_size, _b, _token) PASS_BUCKET_PARAMS(_size, _b) \
+ _PASS_TOKEN_PARAM(_token)
+
/*
* The following functions are not to be used directly and are intended only
* for internal use from kmalloc() and kmalloc_node()
* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

@@ -938,7 +968,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -948,14 +978,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}
- return __kmalloc_noprof(size, flags);
+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
}
+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

+void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);
+#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1092,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))
+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -1075,11 +1107,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);
}
+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**

@@ -1088,14 +1121,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
- return kmalloc_noprof(bytes, flags);
+ return _kmalloc_noprof(bytes, flags, token);
}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**

@@ -1115,18 +1149,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/
-static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p,
+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
size_t new_n,
size_t new_size,
- gfp_t flags)
+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;

- return krealloc_noprof(p, bytes, flags);

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);

}
+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))

/**

@@ -1137,10 +1172,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi

*/
#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node,
unsigned long caller) __alloc_size(1);
#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \
- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller)
+ __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(size, NULL, __kmalloc_token(size)), flags, node, caller)
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1157,17 +1192,18 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag

#define kmalloc_track_caller_noprof(...) \
kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_)

-static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
+ int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
if (__builtin_constant_p(n) && __builtin_constant_p(size))
- return kmalloc_node_noprof(bytes, flags, node);
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node);
+ return _kmalloc_node_noprof(bytes, flags, node, token);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), flags, node);
}
+#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

#define kcalloc_node(_n, _size, _flags, _node) \

@@ -1183,39 +1219,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_

@@ -1224,10 +1264,10 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
#define kvcalloc_node(...) alloc_hooks(kvcalloc_node_noprof(__VA_ARGS__))
#define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__))

-void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
+void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid) __realloc_size(2);
-#define kvrealloc_node_align(...) \
- alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__))
+#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
+ alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/init/Kconfig b/init/Kconfig
index 2937c4d308ae..dba97bc8e2ce 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -170,6 +170,9 @@ config CC_HAS_BROKEN_COUNTED_BY_REF
# https://github.com/llvm/llvm-project/issues/182575
default y if CC_IS_CLANG && CLANG_VERSION < 220100

+config CC_HAS_ALLOC_TOKEN
+ def_bool $(cc-option,-falloc-token-max=123)
+
config CC_HAS_MULTIDIMENSIONAL_NONSTRING
def_bool $(success,echo 'char tag[][4] __attribute__((__nonstring__)) = { };' | $(CC) $(CLANG_FLAGS) -x c - -c -o /dev/null -Werror)

diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config
index 7c3924614e01..26831a2a5739 100644

--- a/kernel/configs/hardening.config
+++ b/kernel/configs/hardening.config
@@ -22,7 +22,7 @@ CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
CONFIG_SLAB_BUCKETS=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-CONFIG_RANDOM_KMALLOC_CACHES=y

+CONFIG_KMALLOC_PARTITION_CACHES=y

# Sanity check userspace page table mappings.
CONFIG_PAGE_TABLE_CHECK=y

diff --git a/mm/Kconfig b/mm/Kconfig
index e8bf1e9e6ad9..4f187b07eb48 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -248,22 +248,75 @@ config SLUB_STATS

out which slabs are relevant to a particular load.
Try running: slabinfo -DA

-config RANDOM_KMALLOC_CACHES
- default n

+config KMALLOC_PARTITION_CACHES
depends on !SLUB_TINY
- bool "Randomize slab caches for normal kmalloc"
+ bool "Partitioned slab caches for normal kmalloc"
+ default RANDOM_KMALLOC_CACHES

help
- A hardening feature that creates multiple copies of slab caches for
- normal kmalloc allocation and makes kmalloc randomly pick one based
- on code address, which makes the attackers more difficult to spray
- vulnerable memory objects on the heap for the purpose of exploiting
- memory vulnerabilities.
+ A hardening feature that creates multiple isolated copies of slab
+ caches for normal kmalloc allocations. This makes it more difficult
+ to exploit memory-safety vulnerabilities by attacking vulnerable
+ co-located memory objects. Several modes are provided.

Currently the number of copies is set to 16, a reasonably large value

that effectively diverges the memory objects allocated for different
subsystems or modules into different caches, at the expense of a
- limited degree of memory and CPU overhead that relates to hardware and
- system workload.
+ limited degree of memory and CPU overhead that relates to hardware
+ and system workload.
+
+choice

+ prompt "Partitioned slab cache mode"
+ depends on KMALLOC_PARTITION_CACHES
+ default KMALLOC_PARTITION_TYPED if CC_HAS_ALLOC_TOKEN
+ default KMALLOC_PARTITION_RANDOM
+ help
+ Selects the slab cache partitioning mode.
+
+config KMALLOC_PARTITION_RANDOM

+ bool "Randomize slab caches for normal kmalloc"
+ help
+ Randomly pick a slab cache based on code address and a per-boot
+ random seed.
+
+ This makes it harder for attackers to predict object co-location.
+ The placement is random: while attackers don't know which kmalloc
+ cache an object will be allocated from, they might circumvent

+ the randomization by retrying attacks across multiple machines until
+ the target objects are co-located.
+
+config KMALLOC_PARTITION_TYPED
+ bool "Type based slab cache selection for normal kmalloc"
+ depends on CC_HAS_ALLOC_TOKEN
+ help
+ Rely on Clang's allocation tokens to choose a slab cache, where token
+ IDs are derived from the allocated type.
+
+ Unlike KMALLOC_PARTITION_RANDOM, cache assignment is deterministic based
+ on type, which guarantees that objects of certain types are not
+ placed in the same cache. This effectively mitigates certain classes
+ of exploits that probabilistic defenses like KMALLOC_PARTITION_RANDOM

+ only make harder but not impossible. However, this also means the
+ cache assignment is predictable.
+
+ Clang's default token ID calculation returns a bounded hash with
+ disjoint ranges for pointer-containing and pointerless objects: when
+ used as the slab cache index, this prevents buffer overflows on
+ primitive buffers from directly corrupting pointer-containing
+ objects.
+
+ The current effectiveness of Clang's type inference can be judged by
+ -Rpass=alloc-token, which provides diagnostics where (after dead-code
+ elimination) type inference failed.
+
+ Requires Clang 22 or later.
+
+endchoice

+
+config RANDOM_KMALLOC_CACHES
+ bool

+ transitional
+ help
+ Transitional config for migration to KMALLOC_PARTITION_CACHES.

endmenu # Slab allocator options

diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
index 5725a367246d..8807ea8ed0d3 100644
--- a/mm/kfence/kfence_test.c
+++ b/mm/kfence/kfence_test.c
@@ -214,7 +214,7 @@ static void test_cache_destroy(void)
static inline size_t kmalloc_cache_alignment(size_t size)
{
/* just to get ->align so no need to pass in the real caller */
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, 0);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(0));
return kmalloc_caches[type][__kmalloc_index(size, false)]->align;
}

@@ -285,7 +285,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat

if (is_kfence_address(alloc)) {
struct slab *slab = virt_to_slab(alloc);
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, _RET_IP_);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(size));
struct kmem_cache *s = test_cache ?:
kmalloc_caches[type][__kmalloc_index(size, false)];

diff --git a/mm/slab.h b/mm/slab.h

index bf2f87acf5e3..1bf9c3021ae3 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -362,12 +362,12 @@ static inline unsigned int size_index_elem(unsigned int bytes)

* KMALLOC_MAX_CACHE_SIZE and the caller must check that.
*/
static inline struct kmem_cache *
-kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller)
+kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, kmalloc_token_t token)
{
unsigned int index;

if (!b)
- b = &kmalloc_caches[kmalloc_type(flags, caller)];
+ b = &kmalloc_caches[kmalloc_type(flags, token)];
if (size <= 192)
index = kmalloc_size_index[size_index_elem(size)];
else

diff --git a/mm/slab_common.c b/mm/slab_common.c
index d5a70a831a2a..388eb5980859 100644

--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -742,7 +742,7 @@ kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init =
{ /* initialization for https://llvm.org/pr42570 */ };
EXPORT_SYMBOL(kmalloc_caches);

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM

unsigned long random_kmalloc_seed __ro_after_init;
EXPORT_SYMBOL(random_kmalloc_seed);
#endif

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES

+#define __KMALLOC_PARTITION_CONCAT(a, b) a ## b
+#define KMALLOC_PARTITION_NAME(N, sz) __KMALLOC_PARTITION_CONCAT(KMA_PART_, N)(sz)
+#define KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 1] = "kmalloc-part-01-" #sz,
+#define KMA_PART_2(sz) KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 2] = "kmalloc-part-02-" #sz,
+#define KMA_PART_3(sz) KMA_PART_2(sz) .name[KMALLOC_PARTITION_START + 3] = "kmalloc-part-03-" #sz,
+#define KMA_PART_4(sz) KMA_PART_3(sz) .name[KMALLOC_PARTITION_START + 4] = "kmalloc-part-04-" #sz,
+#define KMA_PART_5(sz) KMA_PART_4(sz) .name[KMALLOC_PARTITION_START + 5] = "kmalloc-part-05-" #sz,
+#define KMA_PART_6(sz) KMA_PART_5(sz) .name[KMALLOC_PARTITION_START + 6] = "kmalloc-part-06-" #sz,
+#define KMA_PART_7(sz) KMA_PART_6(sz) .name[KMALLOC_PARTITION_START + 7] = "kmalloc-part-07-" #sz,
+#define KMA_PART_8(sz) KMA_PART_7(sz) .name[KMALLOC_PARTITION_START + 8] = "kmalloc-part-08-" #sz,
+#define KMA_PART_9(sz) KMA_PART_8(sz) .name[KMALLOC_PARTITION_START + 9] = "kmalloc-part-09-" #sz,
+#define KMA_PART_10(sz) KMA_PART_9(sz) .name[KMALLOC_PARTITION_START + 10] = "kmalloc-part-10-" #sz,
+#define KMA_PART_11(sz) KMA_PART_10(sz) .name[KMALLOC_PARTITION_START + 11] = "kmalloc-part-11-" #sz,
+#define KMA_PART_12(sz) KMA_PART_11(sz) .name[KMALLOC_PARTITION_START + 12] = "kmalloc-part-12-" #sz,
+#define KMA_PART_13(sz) KMA_PART_12(sz) .name[KMALLOC_PARTITION_START + 13] = "kmalloc-part-13-" #sz,
+#define KMA_PART_14(sz) KMA_PART_13(sz) .name[KMALLOC_PARTITION_START + 14] = "kmalloc-part-14-" #sz,
+#define KMA_PART_15(sz) KMA_PART_14(sz) .name[KMALLOC_PARTITION_START + 15] = "kmalloc-part-15-" #sz,

+#else // CONFIG_KMALLOC_PARTITION_CACHES

+#define KMALLOC_PARTITION_NAME(N, sz)
#endif

#define INIT_KMALLOC_INFO(__size, __short_size) \
@@ -849,7 +849,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
KMALLOC_RCL_NAME(__short_size) \
KMALLOC_CGROUP_NAME(__short_size) \
KMALLOC_DMA_NAME(__short_size) \
- KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size) \

+ KMALLOC_PARTITION_NAME(KMALLOC_PARTITION_CACHES_NR, __short_size) \

.size = __size, \
}

@@ -961,8 +961,8 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
flags |= SLAB_CACHE_DMA;
}

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- if (type >= KMALLOC_RANDOM_START && type <= KMALLOC_RANDOM_END)

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES

+ if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
flags |= SLAB_NO_MERGE;

#endif

@@ -1010,7 +1010,7 @@ void __init create_kmalloc_caches(void)
for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
new_kmalloc_cache(i, type);
}
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
random_kmalloc_seed = get_random_u64();

#endif

diff --git a/mm/slub.c b/mm/slub.c

index 161079ac5ba1..ccb208cfbecd 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2129,11 +2129,11 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,

if (!is_kmalloc_normal(s))
return sz;

- obj_exts_cache = kmalloc_slab(sz, NULL, gfp, 0);
+ obj_exts_cache = kmalloc_slab(sz, NULL, gfp, __kmalloc_token(0));
/*
- * We can't simply compare s with obj_exts_cache, because random kmalloc
- * caches have multiple caches per size, selected by caller address.
- * Since caller address may differ between kmalloc_slab() and actual
+ * We can't simply compare s with obj_exts_cache, because partitioned kmalloc
+ * caches have multiple caches per size, selected by caller address or type.
+ * Since caller address or type may differ between kmalloc_slab() and actual
* allocation, bump size when sizes are equal.
*/
if (s->object_size == obj_exts_cache->object_size)

@@ -5274,7 +5274,7 @@ EXPORT_SYMBOL(__kmalloc_large_node_noprof);

static __always_inline
void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
- unsigned long caller)
+ unsigned long caller, kmalloc_token_t token)
{
struct kmem_cache *s;
void *ret;

@@ -5289,22 +5289,24 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

if (unlikely(!size))
return ZERO_SIZE_PTR;

- s = kmalloc_slab(size, b, flags, caller);
+ s = kmalloc_slab(size, b, flags, token);

ret = slab_alloc_node(s, NULL, flags, node, caller, size);
ret = kasan_kmalloc(s, ret, size, flags);
trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}
-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
{
- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,
+ NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5319,7 +5321,7 @@ EXPORT_SYMBOL(__kmalloc_noprof);

* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
* There is no reason to call it again and expect !NULL.
*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

+void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)

{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
struct kmem_cache *s;

@@ -5342,7 +5344,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);
+ s = kmalloc_slab(size, NULL, alloc_gfp, PASS_TOKEN_PARAM(token));

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*

@@ -5395,12 +5397,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,
int node, unsigned long caller)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);

@@ -6631,7 +6634,7 @@ void kfree_nolock(const void *object)

EXPORT_SYMBOL_GPL(kfree_nolock);

static __always_inline __realloc_size(2) void *
-__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid)
+__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid, kmalloc_token_t token)
{
void *ret;
size_t ks = 0;

@@ -6703,7 +6706,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,

return (void *)p;

alloc_new:
- ret = kmalloc_node_track_caller_noprof(new_size, flags, nid, _RET_IP_);
+ ret = __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(new_size, NULL, token), flags, nid, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();

@@ -6752,7 +6755,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
*

* Return: pointer to the allocated memory or %NULL in case of error
*/

-void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long align,
+void *krealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(new_size, token), unsigned long align,
gfp_t flags, int nid)
{
void *ret;
@@ -6762,7 +6765,7 @@ void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long a

return ZERO_SIZE_PTR;
}

- ret = __do_krealloc(p, new_size, align, flags, nid);
+ ret = __do_krealloc(p, new_size, align, flags, nid, PASS_TOKEN_PARAM(token));
if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret))
kfree(p);

@@ -6799,6 +6802,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

* failure, fall back to non-contiguous (vmalloc) allocation.
* @size: size of the request.
* @b: which set of kmalloc buckets to allocate from.
+ * @token: allocation token.
* @align: desired alignment.
* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
* @node: numa node to allocate from

@@ -6815,7 +6819,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

*
* Return: pointer to the allocated memory of %NULL in case of failure
*/
-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node)
{
bool allow_block;
@@ -6827,7 +6831,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),
- node, _RET_IP_);
+ node, _RET_IP_, PASS_TOKEN_PARAM(token));
if (ret || size <= PAGE_SIZE)
return ret;

@@ -6923,7 +6927,7 @@ EXPORT_SYMBOL(kvfree_sensitive);
*

* Return: pointer to the allocated memory or %NULL in case of error
*/

-void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
+void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid)
{
void *n;
@@ -6931,10 +6935,10 @@ void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long alig

if (is_vmalloc_addr(p))
return vrealloc_node_align_noprof(p, size, align, flags, nid);

- n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid);

+ n = krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(size, token), align, kmalloc_gfp_adjust(flags, size), nid);

if (!n) {
/* We failed to krealloc(), fall back to kvmalloc(). */
- n = kvmalloc_node_align_noprof(size, align, flags, nid);
+ n = __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), align, flags, nid);
if (!n)
return NULL;

@@ -8449,7 +8453,7 @@ static void __init bootstrap_kmalloc_sheaves(void)

{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--

2.54.0.545.g6539524ca2-goog

[Marco Elver]

The mm-api kernel-doc comments have been broken for a while, as many
documented symbols shifted from being direct function definitions to
macros wrapping _noprof implementations during the introduction of
allocation tagging (starting with commit 7bd230a26648 "mm/slab: enable
slab allocation tagging for kmalloc and friends").

When the kernel-doc block remains above the internal implementation
function but uses the public API name, the documentation generator fails
to associate the documented symbol and generates warnings and fails to
emit the documentation.

Fix this by:

1. Moving the kernel-doc comment blocks from slub.c to slab.h, placing
them directly above the user-facing macros.

2. Converting the variadic macros for the documented APIs to use
explicit arguments.

No functional change intended.

Signed-off-by: Marco Elver <el...@google.com>
---

include/linux/slab.h | 192 ++++++++++++++++++++++++++++++++-----------
mm/slub.c | 98 ----------------------
2 files changed, 144 insertions(+), 146 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c232f8a10af6..dcff0da1c01c 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -530,7 +530,46 @@ void * __must_check krealloc_node_align_noprof(const void *objp,

unsigned long align,
gfp_t flags, int nid) __realloc_size(2);

#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)

-#define krealloc_node_align(_o, _s, _a, _f, _n) alloc_hooks(krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))
+/**
+ * krealloc_node_align - reallocate memory. The contents will remain unchanged.
+ * @p: object to reallocate memory for.
+ * @new_size: how many bytes of memory are required.
+ * @align: desired alignment.
+ * @flags: the type of memory to allocate.
+ * @nid: NUMA node or NUMA_NO_NODE
+ *
+ * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size
+ * is 0 and @p is not a %NULL pointer, the object pointed to is freed.
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
+ * initial memory allocation, every subsequent call to this API for the same
+ * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
+ * __GFP_ZERO is not fully honored by this API.
+ *
+ * When slub_debug_orig_size() is off, krealloc() only knows about the bucket
+ * size of an allocation (but not the exact size it was allocated with) and
+ * hence implements the following semantics for shrinking and growing buffers
+ * with __GFP_ZERO::
+ *
+ * new bucket
+ * 0 size size
+ * |--------|----------------|
+ * | keep | zero |
+ *
+ * Otherwise, the original allocation size 'orig_size' could be used to
+ * precisely clear the requested size, and the new size will also be stored
+ * as the new 'orig_size'.
+ *
+ * In any case, the contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes.
+ *
+ * Return: pointer to the allocated memory or %NULL in case of error
+ */
+#define krealloc_node_align(p, new_size, align, flags, nid) \
+ alloc_hooks(krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(new_size, __kmalloc_token(new_size)), align, flags, nid))

#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -913,6 +952,22 @@ void *__kmalloc_large_noprof(size_t size, gfp_t flags)

void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

__assume_page_alignment __alloc_size(1);

+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)

+{
+ if (__builtin_constant_p(size) && size) {
+ unsigned int index;
+
+ if (size > KMALLOC_MAX_CACHE_SIZE)
+ return __kmalloc_large_noprof(size, flags);
+
+ index = kmalloc_index(size);
+ return __kmalloc_cache_noprof(

+ kmalloc_caches[kmalloc_type(flags, token)][index],

+ flags, size);
+ }

+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
+}

+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
/**
* kmalloc - allocate kernel memory

* @size: how many bytes of memory are required.

@@ -968,27 +1023,22 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/

-static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
-{
- if (__builtin_constant_p(size) && size) {
- unsigned int index;
-
- if (size > KMALLOC_MAX_CACHE_SIZE)
- return __kmalloc_large_noprof(size, flags);
-
- index = kmalloc_index(size);
- return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, token)][index],
- flags, size);
- }
- return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
-}
-#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))
+#define kmalloc(size, flags) alloc_hooks(kmalloc_noprof(size, flags))

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);

#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)

-#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))
+/**
+ * kmalloc_nolock - Allocate an object of given size from any context.
+ * @size: size to allocate
+ * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
+ * allowed.
+ * @node: node number of the target node.
+ *
+ * Return: pointer to the new object or NULL in case of error.
+ * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
+ * There is no reason to call it again and expect !NULL.
+ */
+#define kmalloc_nolock(size, gfp_flags, node) alloc_hooks(kmalloc_nolock_noprof(size, gfp_flags, node))

/**
* __alloc_objs - Allocate objects of a given type using
@@ -1115,23 +1165,36 @@ static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, g

#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)

+{
+ size_t bytes;
+
+ if (unlikely(check_mul_overflow(n, size, &bytes)))
+ return NULL;

+ return _kmalloc_noprof(bytes, flags, token);
+}

+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
/**
* kmalloc_array - allocate memory for an array.
* @n: number of elements.

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/

-static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
+#define kmalloc_array(n, size, flags) alloc_hooks(kmalloc_array_noprof(n, size, flags))
+

+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,

+ size_t new_n,
+ size_t new_size,

+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

- if (unlikely(check_mul_overflow(n, size, &bytes)))
+ if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;
- return _kmalloc_noprof(bytes, flags, token);
-}
-#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
+}

+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
/**
* krealloc_array - reallocate memory for an array.
* @p: pointer to the memory chunk to reallocate
@@ -1149,20 +1212,7 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t si

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/

-static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
- size_t new_n,
- size_t new_size,
- gfp_t flags, kmalloc_token_t token)
-{
- size_t bytes;
-
- if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
- return NULL;
-
- return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
-}
-#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))
+#define krealloc_array(p, new_n, new_size, flags) alloc_hooks(krealloc_array_noprof(p, new_n, new_size, flags))

/**
* kcalloc - allocate memory for an array. The memory is set to zero.
@@ -1214,17 +1264,17 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size
*/
#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO)

-/**
- * kzalloc - allocate memory. The memory is set to zero.
- * @size: how many bytes of memory are required.
- * @flags: the type of memory to allocate (see kmalloc).
- */

static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{

return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}

#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
+/**
+ * kzalloc - allocate memory. The memory is set to zero.
+ * @size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate (see kmalloc).
+ */
+#define kzalloc(size, flags) alloc_hooks(kzalloc_noprof(size, flags))

#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

@@ -1233,7 +1283,26 @@ void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long

__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))

-#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)
+/**
+ * kvmalloc_node - attempt to allocate physically contiguous memory, but upon
+ * failure, fall back to non-contiguous (vmalloc) allocation.
+ * @size: size of the request.
+ * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
+ * @node: numa node to allocate from
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * Uses kmalloc to get the memory but if the allocation fails then falls back
+ * to the vmalloc allocator. Use kvfree for freeing the memory.
+ *
+ * GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not.
+ * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is
+ * preferable to the vmalloc fallback, due to visible performance drawbacks.
+ *
+ * Return: pointer to the allocated memory of %NULL in case of failure
+ */
+#define kvmalloc_node(size, flags, node) kvmalloc_node_align(size, 1, flags, node)
#define kvmalloc_node_noprof(size, flags, node) \

kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -1266,8 +1335,35 @@ _kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmallo

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid) __realloc_size(2);

-#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
- alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))
+/**
+ * kvrealloc_node_align - reallocate memory; contents remain unchanged
+ * @p: object to reallocate memory for
+ * @size: the size to reallocate
+ * @align: desired alignment
+ * @flags: the flags for the page level allocator
+ * @nid: NUMA node id
+ *
+ * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0
+ * and @p is not a %NULL pointer, the object pointed to is freed.
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
+ * initial memory allocation, every subsequent call to this API for the same
+ * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
+ * __GFP_ZERO is not fully honored by this API.
+ *
+ * In any case, the contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes.
+ *
+ * This function must not be called concurrently with itself or kvfree() for the
+ * same memory allocation.
+ *
+ * Return: pointer to the allocated memory or %NULL in case of error
+ */
+#define kvrealloc_node_align(p, size, align, flags, nid) \
+ alloc_hooks(kvrealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(size, __kmalloc_token(size)), align, flags, nid))

#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/mm/slub.c b/mm/slub.c
index ccb208cfbecd..dbc3c947e5be 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5310,17 +5310,6 @@ void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
}
EXPORT_SYMBOL(__kmalloc_noprof);

-/**
- * kmalloc_nolock - Allocate an object of given size from any context.
- * @size: size to allocate
- * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
- * allowed.
- * @node: node number of the target node.
- *
- * Return: pointer to the new object or NULL in case of error.
- * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
- * There is no reason to call it again and expect !NULL.
- */

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)
{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;

@@ -6717,44 +6706,6 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
return ret;
}

-/**
- * krealloc_node_align - reallocate memory. The contents will remain unchanged.
- * @p: object to reallocate memory for.
- * @new_size: how many bytes of memory are required.
- * @align: desired alignment.
- * @flags: the type of memory to allocate.
- * @nid: NUMA node or NUMA_NO_NODE
- *
- * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size
- * is 0 and @p is not a %NULL pointer, the object pointed to is freed.
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
- * initial memory allocation, every subsequent call to this API for the same
- * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
- * __GFP_ZERO is not fully honored by this API.
- *
- * When slub_debug_orig_size() is off, krealloc() only knows about the bucket
- * size of an allocation (but not the exact size it was allocated with) and
- * hence implements the following semantics for shrinking and growing buffers
- * with __GFP_ZERO::
- *
- * new bucket
- * 0 size size
- * |--------|----------------|
- * | keep | zero |
- *
- * Otherwise, the original allocation size 'orig_size' could be used to
- * precisely clear the requested size, and the new size will also be stored
- * as the new 'orig_size'.
- *
- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */

void *krealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(new_size, token), unsigned long align,
gfp_t flags, int nid)
{

@@ -6797,28 +6748,6 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
return flags;
}

-/**
- * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon
- * failure, fall back to non-contiguous (vmalloc) allocation.
- * @size: size of the request.
- * @b: which set of kmalloc buckets to allocate from.
- * @token: allocation token.
- * @align: desired alignment.
- * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
- * @node: numa node to allocate from
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * Uses kmalloc to get the memory but if the allocation fails then falls back
- * to the vmalloc allocator. Use kvfree for freeing the memory.
- *
- * GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not.
- * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is
- * preferable to the vmalloc fallback, due to visible performance drawbacks.
- *
- * Return: pointer to the allocated memory of %NULL in case of failure
- */

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node)

{
@@ -6900,33 +6829,6 @@ void kvfree_sensitive(const void *addr, size_t len)
}
EXPORT_SYMBOL(kvfree_sensitive);

-/**
- * kvrealloc_node_align - reallocate memory; contents remain unchanged
- * @p: object to reallocate memory for
- * @size: the size to reallocate
- * @align: desired alignment
- * @flags: the flags for the page level allocator
- * @nid: NUMA node id
- *
- * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0
- * and @p is not a %NULL pointer, the object pointed to is freed.
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
- * initial memory allocation, every subsequent call to this API for the same
- * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
- * __GFP_ZERO is not fully honored by this API.
- *
- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * This function must not be called concurrently with itself or kvfree() for the
- * same memory allocation.
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid)

{
--
2.54.0.545.g6539524ca2-goog

[Vlastimil Babka (SUSE)]

On 4/24/26 15:24, Marco Elver wrote:

> @@ -938,7 +968,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
> * Try really hard to succeed the allocation but fail
> * eventually.
> */
> -static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
> +static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
> {
> if (__builtin_constant_p(size) && size) {
> unsigned int index;
> @@ -948,14 +978,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f
>
> index = kmalloc_index(size);
> return __kmalloc_cache_noprof(
> - kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
> + kmalloc_caches[kmalloc_type(flags, token)][index],

While reviewing this, it occured to me we might have been using _RET_IP_
here in a suboptimal way ever since this was introduced. Since this is all
inlined, shouldn't have we been using _THIS_IP_ to really randomize using
the kmalloc() callsite, and not its parent?

And after this patch, we get the token passed to _kmalloc_noprof()...

> flags, size);
> }
> - return __kmalloc_noprof(size, flags);
> + return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);

... and used also here for the non-constant-size, where previously
__kmalloc_noprof() (not inline function) would correctly use _RET_IP_ on its
own ...

> }
> +#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

... and the token comes from here. With random partitioning that's

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })

so that AFAIK makes the situation worse as now the cases without constant
size also start randomizing by the parent callsite and not the kmalloc callsite.

But there are many users of __kmalloc_token() and maybe some are corrent in
using _RET_IP_, I haven't checked, maybe we'll need two variants, or further
change things around.

> #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))
>
> -void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
> +void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);
> +#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)
> #define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))
>
> /**

<snip>

We're renaming the caches visible in /proc/slabinfo. Maybe it's fine as
people who care will just adapt their tools without complaining. Not the
first time we've changes something here.

[Vlastimil Babka (SUSE)]

On 4/24/26 15:24, Marco Elver wrote:

> The mm-api kernel-doc comments have been broken for a while, as many
> documented symbols shifted from being direct function definitions to
> macros wrapping _noprof implementations during the introduction of
> allocation tagging (starting with commit 7bd230a26648 "mm/slab: enable
> slab allocation tagging for kmalloc and friends").
>
> When the kernel-doc block remains above the internal implementation
> function but uses the public API name, the documentation generator fails
> to associate the documented symbol and generates warnings and fails to
> emit the documentation.
>
> Fix this by:
>
> 1. Moving the kernel-doc comment blocks from slub.c to slab.h, placing
> them directly above the user-facing macros.
>
> 2. Converting the variadic macros for the documented APIs to use
> explicit arguments.
>
> No functional change intended.
>
> Signed-off-by: Marco Elver <el...@google.com>

+Cc Jon

I thought it was supposed to work because the kernel-doc scripts were at the
time taught by commit 51a7bf0238c2 ("scripts/kernel-doc: drop "_noprof" on
function prototypes") to handle _noprof. In the current form git grep finds:

tools/lib/python/kdoc/kdoc_parser.py: suffixes = [ '_noprof' ]
tools/lib/python/kdoc/xforms_lists.py: (KernRe("_noprof"), ""),

Doesn't it work for you then?

[Marco Elver]

Ah, I see. So it doesn't work anymore because we add the '_' prefix, too.

I guess the question is if we want to proliferate more kdoc parser
special cases, or just move the docs to the macros. The downside of
macros is that they lose the types in the displayed function
signature.

Preferences?

[Marco Elver]

How about the below, i.e. adding type decls that only the kernel-doc
parser sees? One complication is also DECL_KMALLOC_PARAMS, and adding
kernel-doc parser hacks for that looks pretty awful, so this is a lot
cleaner.

------ >8 ------

From: Marco Elver <el...@google.com>
Date: Tue, 21 Apr 2026 13:48:21 +0200
Subject: [PATCH] slab: fix kernel-docs for mm-api

The mm-api kernel-docs have been disconnected from their symbols. While
the scripts were previously taught to handle the _noprof suffix added by
allocation tagging (in 51a7bf0238c2 "scripts/kernel-doc: drop "_noprof"
on function prototypes"), this does not handle cases where the internal
implementation function has an additional leading underscore. The added
optional parameters (via DECL_KMALLOC_PARAMS) further complicate parsing
the internal signatures.

When the kernel-doc block remains above the internal implementation
function but uses the public API name, the documentation generator fails

to associate the documented symbol.

Simply moving the docs to the macros in slab.h fixes the association but
causes loss of types in the generated documentation (rendering as e.g.
untyped 'kmalloc(size, flags)' macro).

Fix this by:

1. Moving the kernel-doc comment blocks from slub.c to slab.h, placing
them directly above the user-facing macros.

2. Providing explicit, typed C prototypes for the documented APIs inside
'#if 0 /* kernel-doc */' blocks.

3. Converting the variadic macros for the documented APIs to use
explicit arguments to match the documentation.

No functional change intended.

Signed-off-by: Marco Elver <el...@google.com>

---
v4:
* Provide typed C prototypes in '#if 0' blocks to properly render API.
---
include/linux/slab.h | 216 +++++++++++++++++++++++++++++++++----------
mm/slub.c | 98 --------------------
2 files changed, 168 insertions(+), 146 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c232f8a10af6..5e1249e36b0d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -530,7 +530,49 @@ void * __must_check krealloc_node_align_noprof(const void *objp,

unsigned long align,
gfp_t flags, int nid) __realloc_size(2);
#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)
-#define krealloc_node_align(_o, _s, _a, _f, _n) alloc_hooks(krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

+#if 0 /* kernel-doc */

+void *krealloc_node_align(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid);
+#endif

+#define krealloc_node_align(p, new_size, align, flags, nid) \
+ alloc_hooks(krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(new_size, __kmalloc_token(new_size)), align, flags, nid))
#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -913,6 +955,23 @@ void *__kmalloc_large_noprof(size_t size, gfp_t flags)

void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
__assume_page_alignment __alloc_size(1);

+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
+{
+ if (__builtin_constant_p(size) && size) {
+ unsigned int index;
+
+ if (size > KMALLOC_MAX_CACHE_SIZE)
+ return __kmalloc_large_noprof(size, flags);
+
+ index = kmalloc_index(size);
+ return __kmalloc_cache_noprof(
+ kmalloc_caches[kmalloc_type(flags, token)][index],
+ flags, size);
+ }
+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
+}
+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

+#if 0 /* kernel-doc */

/**
* kmalloc - allocate kernel memory
* @size: how many bytes of memory are required.

@@ -968,27 +1027,27 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
-{
- if (__builtin_constant_p(size) && size) {
- unsigned int index;
-
- if (size > KMALLOC_MAX_CACHE_SIZE)
- return __kmalloc_large_noprof(size, flags);
-
- index = kmalloc_index(size);
- return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, token)][index],
- flags, size);
- }
- return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
-}
-#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

+void *kmalloc(size_t size, gfp_t flags);
+#endif

+#define kmalloc(size, flags) alloc_hooks(kmalloc_noprof(size, flags))

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);
#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)
-#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

+#if 0 /* kernel-doc */

+/**
+ * kmalloc_nolock - Allocate an object of given size from any context.
+ * @size: size to allocate
+ * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
+ * allowed.
+ * @node: node number of the target node.
+ *
+ * Return: pointer to the new object or NULL in case of error.
+ * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
+ * There is no reason to call it again and expect !NULL.
+ */

+void *kmalloc_nolock(size_t size, gfp_t gfp_flags, int node);
+#endif

+#define kmalloc_nolock(size, gfp_flags, node) alloc_hooks(kmalloc_nolock_noprof(size, gfp_flags, node))

/**
* __alloc_objs - Allocate objects of a given type using

@@ -1115,23 +1174,40 @@ static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, g

#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
+{
+ size_t bytes;
+
+ if (unlikely(check_mul_overflow(n, size, &bytes)))
+ return NULL;
+ return _kmalloc_noprof(bytes, flags, token);
+}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

+#if 0 /* kernel-doc */

/**
* kmalloc_array - allocate memory for an array.
* @n: number of elements.
* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)

+void *kmalloc_array(size_t n, size_t size, gfp_t flags);
+#endif

+#define kmalloc_array(n, size, flags) alloc_hooks(kmalloc_array_noprof(n, size, flags))
+
+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
+ size_t new_n,
+ size_t new_size,
+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

- if (unlikely(check_mul_overflow(n, size, &bytes)))
+ if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;
- return _kmalloc_noprof(bytes, flags, token);
-}
-#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
+}
+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

+#if 0 /* kernel-doc */

/**
* krealloc_array - reallocate memory for an array.
* @p: pointer to the memory chunk to reallocate

@@ -1149,20 +1225,9 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t si

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/
-static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
- size_t new_n,
- size_t new_size,
- gfp_t flags, kmalloc_token_t token)
-{
- size_t bytes;
-
- if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
- return NULL;
-
- return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
-}
-#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))

+void *krealloc_array(void *p, size_t new_n, size_t new_size, gfp_t flags);
+#endif

+#define krealloc_array(p, new_n, new_size, flags) alloc_hooks(krealloc_array_noprof(p, new_n, new_size, flags))

/**
* kcalloc - allocate memory for an array. The memory is set to zero.

@@ -1214,17 +1279,20 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size

*/
#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO)

-/**
- * kzalloc - allocate memory. The memory is set to zero.
- * @size: how many bytes of memory are required.
- * @flags: the type of memory to allocate (see kmalloc).
- */
static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}
#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))

+#if 0 /* kernel-doc */

+/**
+ * kzalloc - allocate memory. The memory is set to zero.
+ * @size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate (see kmalloc).
+ */

+void *kzalloc(size_t size, gfp_t flags);
+#endif

+#define kzalloc(size, flags) alloc_hooks(kzalloc_noprof(size, flags))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

@@ -1233,7 +1301,29 @@ void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long

__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
-#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)

+#if 0 /* kernel-doc */

+/**
+ * kvmalloc_node - attempt to allocate physically contiguous memory, but upon
+ * failure, fall back to non-contiguous (vmalloc) allocation.
+ * @size: size of the request.
+ * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
+ * @node: numa node to allocate from
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * Uses kmalloc to get the memory but if the allocation fails then falls back
+ * to the vmalloc allocator. Use kvfree for freeing the memory.
+ *
+ * GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not.
+ * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is
+ * preferable to the vmalloc fallback, due to visible performance drawbacks.
+ *
+ * Return: pointer to the allocated memory of %NULL in case of failure
+ */

+void *kvmalloc_node(size_t size, gfp_t flags, int node);
+#endif

+#define kvmalloc_node(size, flags, node) kvmalloc_node_align(size, 1, flags, node)
#define kvmalloc_node_noprof(size, flags, node) \
kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -1266,8 +1356,38 @@ _kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmallo

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid) __realloc_size(2);
-#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
- alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

+#if 0 /* kernel-doc */

+void *kvrealloc_node_align(const void *p, size_t size, unsigned long align, gfp_t flags, int nid);
+#endif

[Marco Elver]

Good catch. I don't think we need multiple variants (otherwise the TYPED
variant would be broken) - we're moving token generation to the callers
(not even inlined anymore) with all this macro magic.

I think this is all we need:

--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -503,7 +503,7 @@ int kmem_cache_shrink(struct kmem_cache *s);

typedef struct { unsigned long v; } kmalloc_token_t;

#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
extern unsigned long random_kmalloc_seed;
-#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _THIS_IP_ })
#elif defined(CONFIG_KMALLOC_PARTITION_TYPED)
#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })
#endif

Plus a paragraph in the commit message. Let me add that.

[Marco Elver]

Bah, this is why it doesn't work:

>> drivers/gpu/drm/msm/msm_gpu.c:272:4: error: cannot jump from this indirect goto statement to one of its possible targets
272 | drm_exec_retry_on_contention(&exec);
| ^
include/drm/drm_exec.h:123:4: note: expanded from macro
'drm_exec_retry_on_contention'
123 | goto *__drm_exec_retry_ptr; \
| ^
drivers/gpu/drm/msm/msm_gpu.c:304:16: note: possible target of
indirect goto statement
304 | state->bos = kcalloc(submit->nr_bos,
| ^
include/linux/slab.h:1173:34: note: expanded from macro 'kcalloc'
1173 | #define kcalloc(n, size, flags) kmalloc_array(n,
size, (flags) | __GFP_ZERO)
| ^
include/linux/slab.h:1133:42: note: expanded from macro 'kmalloc_array'
1133 | #define kmalloc_array(...)
alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))
| ^
include/linux/slab.h:1132:71: note: expanded from macro
'kmalloc_array_noprof'
1132 | #define kmalloc_array_noprof(...)
_kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
|
^
include/linux/slab.h:506:55: note: expanded from macro '__kmalloc_token'
506 | #define __kmalloc_token(...) ((kmalloc_token_t){ .v = _THIS_IP_ })
| ^
include/linux/instruction_pointer.h:10:41: note: expanded from
macro '_THIS_IP_'
10 | #define _THIS_IP_ ({ __label__ __here; __here: (unsigned
long)&&__here; })
| ^
drivers/gpu/drm/msm/msm_gpu.c:304:16: note: jump enters a statement
expression


Apparently using _THIS_IP_ creates a possible indirect jump target,
but because it's in a statement expression, it's invalid, so the
compiler complains. This is obviously nonsense, because the actual
indirect jump in this gpu driver code would never jump to the
_THIS_IP_ __here label, but that's what it is.

Given this pre-existing issue, we probably need to continue using
_RET_IP_, as before. I tried to fix _THIS_IP_, but it's incredibly
brittle (e.g. __always_inline function returning address of label
doesn't work on Clang, but would on GCC).

[Nathan Chancellor]

For what it's worth, both LLVM and GCC consider the generic version of
_THIS_IP_ to be broken (even if it works currently), as the address of a
label is only expected to be used with a computed goto (hence the clang
error above, it just looks for possible computed goto targets):

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=44298
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071
https://github.com/llvm/llvm-project/issues/138272

--
Cheers,
Nathan

[Harry Yoo (Oracle)]

Err, I was like "yes, this is the way to go!"

and then...

Didn't even realize people use indirect gotos, heh :)

> but because it's in a statement expression, it's invalid, so the
> compiler complains. This is obviously nonsense, because the actual
> indirect jump in this gpu driver code would never jump to the
> _THIS_IP_ __here label, but that's what it is.

Yeah, I guess it's quite tricky to handle when you don't know where
it'd jump to as it's an indirect one, and there's an invalid jump
label...

> Given this pre-existing issue, we probably need to continue using
> _RET_IP_, as before.

Agreed!

[Marco Elver]

I think I have a solution for this mess, see below.

I would not send it as 1 series, but only include the slab changes (+
instruction_pointer.h change to introduce _CODE_LOCATION_) as one
series, to go through the slab tree. The rest of the patches would go to
respective arch maintainers.

------ >8 ------

diff --git a/arch/alpha/include/asm/linkage.h b/arch/alpha/include/asm/linkage.h
index aa8661fa60dc..88617cfaa0f7 100644
--- a/arch/alpha/include/asm/linkage.h
+++ b/arch/alpha/include/asm/linkage.h
@@ -6,4 +6,6 @@
#define SYSCALL_ALIAS(alias, name) \
asm ( #alias " = " #name "\n\t.globl " #alias)

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("br %0, 1f\n1:" : "=r" (__ip)); __ip; })
+
#endif
diff --git a/arch/arc/include/asm/linkage.h b/arch/arc/include/asm/linkage.h
index ba3cb65b5eaa..3fb91d1672ba 100644
--- a/arch/arc/include/asm/linkage.h
+++ b/arch/arc/include/asm/linkage.h
@@ -75,6 +75,8 @@
#define __arcfp_data __section(".data")
#endif

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("mov %0, pcl" : "=r" (__ip)); __ip; })
+
#endif /* __ASSEMBLER__ */

#endif
diff --git a/arch/arm/include/asm/linkage.h b/arch/arm/include/asm/linkage.h
index c4670694ada7..416e6a242dc4 100644
--- a/arch/arm/include/asm/linkage.h
+++ b/arch/arm/include/asm/linkage.h
@@ -9,4 +9,6 @@
.type name, %function; \
END(name)

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("1: adr %0, 1b" : "=r" (__ip)); __ip; })
+
#endif
diff --git a/arch/arm64/include/asm/linkage.h b/arch/arm64/include/asm/linkage.h
index 40bd17add539..73eabc82a6bb 100644
--- a/arch/arm64/include/asm/linkage.h
+++ b/arch/arm64/include/asm/linkage.h
@@ -43,4 +43,6 @@
SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \
bti c ;

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("adr %0, ." : "=r" (__ip)); __ip; })
+
#endif
diff --git a/arch/csky/include/asm/linkage.h b/arch/csky/include/asm/linkage.h
new file mode 100644
index 000000000000..04afd3583e25
--- /dev/null
+++ b/arch/csky/include/asm/linkage.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_CSKY_LINKAGE_H
+#define __ASM_CSKY_LINKAGE_H
+
+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("grs %0, ." : "=r" (__ip)); __ip; })
+
+#endif /* __ASM_CSKY_LINKAGE_H */
diff --git a/arch/hexagon/include/asm/linkage.h b/arch/hexagon/include/asm/linkage.h
index ebdb581939e8..b3808f093e62 100644
--- a/arch/hexagon/include/asm/linkage.h
+++ b/arch/hexagon/include/asm/linkage.h
@@ -9,4 +9,6 @@
#define __ALIGN .align 4
#define __ALIGN_STR ".align 4"

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("call 1f\n1: %0 = r31" : "=r" (__ip) : : "r31"); __ip; })
+
#endif
diff --git a/arch/loongarch/include/asm/linkage.h b/arch/loongarch/include/asm/linkage.h
index a1bd6a3ee03a..f175b25068d7 100644
--- a/arch/loongarch/include/asm/linkage.h
+++ b/arch/loongarch/include/asm/linkage.h
@@ -77,4 +77,6 @@

#define SYM_SIGFUNC_END(name) SYM_FUNC_END(name)

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("pcaddi %0, 0" : "=r" (__ip)); __ip; })
+
#endif
diff --git a/arch/m68k/include/asm/linkage.h b/arch/m68k/include/asm/linkage.h
index c8b84282764c..9ed2f36830d0 100644
--- a/arch/m68k/include/asm/linkage.h
+++ b/arch/m68k/include/asm/linkage.h
@@ -35,4 +35,6 @@
__asmlinkage_protect_n(ret, "m" (arg1), "m" (arg2), "m" (arg3), \
"m" (arg4), "m" (arg5), "m" (arg6))

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("lea %%pc@(.), %0" : "=a" (__ip)); __ip; })
+
#endif
diff --git a/arch/microblaze/include/asm/linkage.h b/arch/microblaze/include/asm/linkage.h
new file mode 100644
index 000000000000..fc3873e0e9b6
--- /dev/null
+++ b/arch/microblaze/include/asm/linkage.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_MICROBLAZE_LINKAGE_H
+#define _ASM_MICROBLAZE_LINKAGE_H
+
+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("mfs %0, rpc" : "=r" (__ip)); __ip; })
+
+#endif /* _ASM_MICROBLAZE_LINKAGE_H */
diff --git a/arch/mips/include/asm/linkage.h b/arch/mips/include/asm/linkage.h
index fd44ba754f1a..0579eac57def 100644
--- a/arch/mips/include/asm/linkage.h
+++ b/arch/mips/include/asm/linkage.h
@@ -10,4 +10,14 @@
#define SYSCALL_ALIAS(alias, name) \
asm ( #alias " = " #name "\n\t.globl " #alias)

+#define _THIS_IP_ ({ \
+ unsigned long __ip; \
+ asm volatile("bal 1f\n\t" \
+ " nop\n\t" \
+ "1: move %0, $ra" \
+ : "=r" (__ip) : : "$ra" \
+ ); \
+ __ip; \
+})
+
#endif
diff --git a/arch/nios2/include/asm/linkage.h b/arch/nios2/include/asm/linkage.h
index 211302301a8a..c4073235852b 100644
--- a/arch/nios2/include/asm/linkage.h
+++ b/arch/nios2/include/asm/linkage.h
@@ -12,4 +12,6 @@
#define __ALIGN .align 4
#define __ALIGN_STR ".align 4"

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("nextpc %0" : "=r" (__ip)); __ip; })
+
#endif
diff --git a/arch/openrisc/include/asm/linkage.h b/arch/openrisc/include/asm/linkage.h
index 25aa449ac30e..a96e808b5d1a 100644
--- a/arch/openrisc/include/asm/linkage.h
+++ b/arch/openrisc/include/asm/linkage.h
@@ -18,4 +18,14 @@
#define __ALIGN .align 0
#define __ALIGN_STR ".align 0"

+#define _THIS_IP_ ({ \
+ unsigned long __ip; \
+ asm volatile("l.jal 1f\n\t" \
+ " l.nop\n\t" \
+ "1: l.ori %0, r9, 0" \
+ : "=r" (__ip) : : "r9" \
+ ); \
+ __ip; \
+})
+
#endif /* __ASM_OPENRISC_LINKAGE_H */
diff --git a/arch/parisc/include/asm/linkage.h b/arch/parisc/include/asm/linkage.h
index d4cad492b971..d4d8ff7735c7 100644
--- a/arch/parisc/include/asm/linkage.h
+++ b/arch/parisc/include/asm/linkage.h
@@ -37,4 +37,12 @@ name: ASM_NL\

#endif /* __ASSEMBLER__ */

+#define _THIS_IP_ ({ \
+ unsigned long __ip; \
+ asm volatile("b,l 1f, %0\n\t" \
+ " nop\n\t" \
+ "1:" : "=r" (__ip)); \
+ __ip; \
+})
+
#endif /* __ASM_PARISC_LINKAGE_H */
diff --git a/arch/powerpc/include/asm/linkage.h b/arch/powerpc/include/asm/linkage.h
index b71b9582e754..aa469e7bef0b 100644
--- a/arch/powerpc/include/asm/linkage.h
+++ b/arch/powerpc/include/asm/linkage.h
@@ -13,4 +13,13 @@
"\t.globl ." #alias "\n\t.set ." #alias ", ." #name)
#endif

+#define _THIS_IP_ ({ \
+ unsigned long __ip; \
+ asm volatile("bcl 20,31,1f\n\t" \
+ "1: mflr %0" \
+ : "=r" (__ip) : : "lr" \
+ ); \
+ __ip; \
+})
+
#endif /* _ASM_POWERPC_LINKAGE_H */
diff --git a/arch/riscv/include/asm/linkage.h b/arch/riscv/include/asm/linkage.h
index 9e88ba23cd2b..7e0210ef4eb4 100644
--- a/arch/riscv/include/asm/linkage.h
+++ b/arch/riscv/include/asm/linkage.h
@@ -9,4 +9,6 @@
#define __ALIGN .balign 4
#define __ALIGN_STR ".balign 4"

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("auipc %0, 0" : "=r" (__ip)); __ip; })
+
#endif /* _ASM_RISCV_LINKAGE_H */
diff --git a/arch/s390/include/asm/linkage.h b/arch/s390/include/asm/linkage.h
index df3fb7d8227b..1b3ac553a642 100644
--- a/arch/s390/include/asm/linkage.h
+++ b/arch/s390/include/asm/linkage.h
@@ -7,4 +7,6 @@
#define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT, 0x07
#define __ALIGN_STR __stringify(__ALIGN)

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("larl %0, ." : "=d" (__ip)); __ip; })
+
#endif
diff --git a/arch/sh/include/asm/linkage.h b/arch/sh/include/asm/linkage.h
index 7c2fa27a43f8..af56b38b6001 100644
--- a/arch/sh/include/asm/linkage.h
+++ b/arch/sh/include/asm/linkage.h
@@ -5,4 +5,6 @@
#define __ALIGN .balign 4
#define __ALIGN_STR ".balign 4"

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("mova 1f, %0\n1:" : "=z" (__ip)); __ip; })
+
#endif
diff --git a/arch/sparc/include/asm/linkage.h b/arch/sparc/include/asm/linkage.h
new file mode 100644
index 000000000000..3f24e2da88be
--- /dev/null
+++ b/arch/sparc/include/asm/linkage.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_SPARC_LINKAGE_H
+#define _ASM_SPARC_LINKAGE_H
+
+#ifdef CONFIG_SPARC64
+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("rd %%pc, %0" : "=r" (__ip)); __ip; })
+#else
+#define _THIS_IP_ ({ \
+ unsigned long __ip; \
+ asm volatile("call 1f\n\t" \
+ " nop\n\t" \
+ "1: mov %%o7, %0" \
+ : "=r" (__ip) : : "o7" \
+ ); \
+ __ip; \
+})
+#endif
+
+#endif /* _ASM_SPARC_LINKAGE_H */
diff --git a/arch/x86/include/asm/linkage.h b/arch/x86/include/asm/linkage.h
index a7294656ad90..bce3c6f4b94f 100644
--- a/arch/x86/include/asm/linkage.h
+++ b/arch/x86/include/asm/linkage.h
@@ -13,11 +13,12 @@
* The generic version tends to create spurious ENDBR instructions under
* certain conditions.
*/
-#define _THIS_IP_ ({ unsigned long __here; asm ("lea 0(%%rip), %0" : "=r" (__here)); __here; })
+#define _THIS_IP_ ({ unsigned long __here; asm volatile("lea 0(%%rip), %0" : "=r" (__here)); __here; })
#endif

#ifdef CONFIG_X86_32
#define asmlinkage CPP_ASMLINKAGE __attribute__((regparm(0)))
+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("call 1f\n1: pop %0" : "=r" (__ip)); __ip; })
#endif /* CONFIG_X86_32 */

#define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT, 0x90;
diff --git a/arch/xtensa/include/asm/linkage.h b/arch/xtensa/include/asm/linkage.h
index 0ba9973235d9..9e6f5cc81964 100644
--- a/arch/xtensa/include/asm/linkage.h
+++ b/arch/xtensa/include/asm/linkage.h
@@ -6,4 +6,6 @@
#define __ALIGN .align 4
#define __ALIGN_STR ".align 4"

+#define _THIS_IP_ ({ unsigned long __ip; asm volatile("call0 1f\n1: mov %0, a0" : "=r" (__ip) : : "a0"); __ip; })
+
#endif
diff --git a/include/linux/instruction_pointer.h b/include/linux/instruction_pointer.h
index aa0b3ffea935..dfe73aafddb8 100644
--- a/include/linux/instruction_pointer.h
+++ b/include/linux/instruction_pointer.h
@@ -8,6 +8,30 @@

#ifndef _THIS_IP_

#define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })

+/*
+ * The current generic definition of _THIS_IP_ is considered broken by GCC [1]
+ * and Clang [2]. In particular, the address of a label is only expected to be
+ * used with a computed goto.
+ *
+ * [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071
+ * [2] https://github.com/llvm/llvm-project/issues/138272
+ *
+ * Mark it as broken, so that appropriate fallback options can be implemented
+ * for architectures that do not define their won _THIS_IP_.
+ */
+#define HAS_BROKEN_THIS_IP
+#endif
+
+/*
+ * _CODE_LOCATION_ provides a unique identifier for the current code location.
+ * When _THIS_IP_ is broken (generic version), we fall back to a static marker
+ * which guarantees uniqueness and resolves to a constant address at link time,
+ * avoiding runtime overhead and compiler optimizations breaking it.
+ */
+#ifdef HAS_BROKEN_THIS_IP
+#define _CODE_LOCATION_ ({ static const char __here; (unsigned long)&__here; })
+#else
+#define _CODE_LOCATION_ _THIS_IP_
#endif

#endif /* _LINUX_INSTRUCTION_POINTER_H */
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 5e1249e36b0d..a4bf1585411f 100644

--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -503,7 +503,7 @@ int kmem_cache_shrink(struct kmem_cache *s);
typedef struct { unsigned long v; } kmalloc_token_t;
#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
extern unsigned long random_kmalloc_seed;
-#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })

+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _CODE_LOCATION_ })

[Harry Yoo (Oracle)]

On Fri, May 08, 2026 at 04:21:36PM +0200, Marco Elver wrote:
> I think I have a solution for this mess, see below.
>
> I would not send it as 1 series, but only include the slab changes (+
> instruction_pointer.h change to introduce _CODE_LOCATION_) as one
> series, to go through the slab tree. The rest of the patches would go to
> respective arch maintainers.

I'm assuming this will be a follow-up and reviewing patch 1
(and waiting for Jon's thuoghts on patch 2)

> diff --git a/include/linux/instruction_pointer.h b/include/linux/instruction_pointer.h
> index aa0b3ffea935..dfe73aafddb8 100644
> --- a/include/linux/instruction_pointer.h
> +++ b/include/linux/instruction_pointer.h
> @@ -8,6 +8,30 @@
>
> #ifndef _THIS_IP_
> #define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
> +/*
> + * The current generic definition of _THIS_IP_ is considered broken by GCC [1]
> + * and Clang [2]. In particular, the address of a label is only expected to be
> + * used with a computed goto.
> + *
> + * [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071
> + * [2] https://github.com/llvm/llvm-project/issues/138272
> + *
> + * Mark it as broken, so that appropriate fallback options can be implemented
> + * for architectures that do not define their won _THIS_IP_.
> + */
> +#define HAS_BROKEN_THIS_IP
> +#endif

As long as _THIS_IP_ is broken on some arches, it cannot be used anyway
when in a general API that can be used by arbitrary users?

Is it something that can be fixed in all arches over time?

> +/*
> + * _CODE_LOCATION_ provides a unique identifier for the current code location.
> + * When _THIS_IP_ is broken (generic version), we fall back to a static marker
> + * which guarantees uniqueness and resolves to a constant address at link time,
> + * avoiding runtime overhead and compiler optimizations breaking it.
> + */
> +#ifdef HAS_BROKEN_THIS_IP
> +#define _CODE_LOCATION_ ({ static const char __here; (unsigned long)&__here; })

Nice!

Yes, we don't really need the exact code location
for partitioning kmalloc caches.

IIRC lockdep does a similar thing to define lock classes (unique for
each lock init location)

> +#else
> +#define _CODE_LOCATION_ _THIS_IP_
> #endif

Probably we don't need this fallback?

> #endif /* _LINUX_INSTRUCTION_POINTER_H */
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 5e1249e36b0d..a4bf1585411f 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -503,7 +503,7 @@ int kmem_cache_shrink(struct kmem_cache *s);
> typedef struct { unsigned long v; } kmalloc_token_t;
> #ifdef CONFIG_KMALLOC_PARTITION_RANDOM
> extern unsigned long random_kmalloc_seed;
> -#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
> +#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _CODE_LOCATION_ })
> #elif defined(CONFIG_KMALLOC_PARTITION_TYPED)
> #define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })
> #endif

[Marco Elver]

On Mon, 11 May 2026 at 10:31, 'Harry Yoo (Oracle)' via kasan-dev
<kasa...@googlegroups.com> wrote:
>
> On Fri, May 08, 2026 at 04:21:36PM +0200, Marco Elver wrote:
> > I think I have a solution for this mess, see below.
> >
> > I would not send it as 1 series, but only include the slab changes (+
> > instruction_pointer.h change to introduce _CODE_LOCATION_) as one
> > series, to go through the slab tree. The rest of the patches would go to
> > respective arch maintainers.
>
> I'm assuming this will be a follow-up and reviewing patch 1
> (and waiting for Jon's thuoghts on patch 2)

I'll be sending v4 shortly.

> > diff --git a/include/linux/instruction_pointer.h b/include/linux/instruction_pointer.h
> > index aa0b3ffea935..dfe73aafddb8 100644
> > --- a/include/linux/instruction_pointer.h
> > +++ b/include/linux/instruction_pointer.h
> > @@ -8,6 +8,30 @@
> >
> > #ifndef _THIS_IP_
> > #define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
> > +/*
> > + * The current generic definition of _THIS_IP_ is considered broken by GCC [1]
> > + * and Clang [2]. In particular, the address of a label is only expected to be
> > + * used with a computed goto.
> > + *
> > + * [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071
> > + * [2] https://github.com/llvm/llvm-project/issues/138272
> > + *
> > + * Mark it as broken, so that appropriate fallback options can be implemented
> > + * for architectures that do not define their won _THIS_IP_.
> > + */
> > +#define HAS_BROKEN_THIS_IP
> > +#endif
>
> As long as _THIS_IP_ is broken on some arches, it cannot be used anyway
> when in a general API that can be used by arbitrary users?

It more or less works today, and for debugging or tracing it's "good
enough" in most cases.

The plan would be to phase out the generic _THIS_IP_ once all
architectures have a correct _THIS_IP_ implementation.

> Is it something that can be fixed in all arches over time?

Yes, I have patches for that.

> > +/*
> > + * _CODE_LOCATION_ provides a unique identifier for the current code location.
> > + * When _THIS_IP_ is broken (generic version), we fall back to a static marker
> > + * which guarantees uniqueness and resolves to a constant address at link time,
> > + * avoiding runtime overhead and compiler optimizations breaking it.
> > + */
> > +#ifdef HAS_BROKEN_THIS_IP
> > +#define _CODE_LOCATION_ ({ static const char __here; (unsigned long)&__here; })
>
> Nice!
>
> Yes, we don't really need the exact code location
> for partitioning kmalloc caches.
>
> IIRC lockdep does a similar thing to define lock classes (unique for
> each lock init location)
>
> > +#else
> > +#define _CODE_LOCATION_ _THIS_IP_
> > #endif
>
> Probably we don't need this fallback?

x86-64 is the only arch that has working fast _THIS_IP_, and adding
static __here markers to bss is rather wasteful.

More architectures will be supporting _THIS_IP_ properly once I get to
send the patches. The mainstream architectures all have a reasonable
and fast way to get the current IP, so we don't need to waste bss
space there.

[Vlastimil Babka (SUSE)]

Looks like a good workaround to me, unless something gets confused by seeing
both the declaration and the define.

[Jonathan Corbet]

Marco Elver <el...@google.com> writes:

> How about the below, i.e. adding type decls that only the kernel-doc
> parser sees? One complication is also DECL_KMALLOC_PARAMS, and adding
> kernel-doc parser hacks for that looks pretty awful, so this is a lot
> cleaner.

I'm going to be a while catching up with things, so this is just a first
take. I strongly suspect that the people who object so strongly to
documentation markup in general would be less than fully thrilled by the
addition of this kind of workaround. I'd like to ponder a bit and see
if I can some up with something better...but again, it won't happen
right away.

Thanks,

jon

[Marco Elver]

Thanks, Jon.

The technical debt (hacks) that would accumulate in the kernel-doc
parser just for accommodating slab.h might be quite high; slab.h is
rather special, so the "#if 0" solution might be justified. Then
again, if there's a reasonable kernel-doc parser solution, that also
helps in other places, I won't object.

I tested the "#if 0" version and it works as expected. So to move
forward, we could consider it in the short term, and in the longer
term, see what new powers the kernel-doc parser can provide.

[Kees Cook]

Teeny typo above: "won" -> "own".

Thanks for finding a solution for this!

--
Kees Cook

[Marco Elver]

Signed-off-by: Marco Elver <el...@google.com>
---

v3: https://lore.kernel.org/all/20260424132427....@google.com/

index 9f88dcaae382..13d804391949 100644
--- a/Makefile
+++ b/Makefile
@@ -991,6 +991,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)

endif
endif

+ifdef CONFIG_KMALLOC_PARTITION_TYPED
+# KMALLOC_PARTITION_CACHES_NR + 1
+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 85bf8dd9f087..bdb721dac0e3 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_KMALLOC_PARTITION_CACHES
# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..c232f8a10af6 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h

-void * __must_check krealloc_node_align_noprof(const void *objp, size_t new_size,
+void * __must_check krealloc_node_align_noprof(const void *objp,
+ DECL_TOKEN_PARAMS(new_size, token),
unsigned long align,

gfp_t flags, int nid) __realloc_size(2);
-#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE)
-#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__))

+#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)
+#define krealloc_node_align(_o, _s, _a, _f, _n) alloc_hooks(krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -938,7 +968,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -948,14 +978,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],

flags, size);
}
- return __kmalloc_noprof(size, flags);
+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
}

+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
+void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);
+#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1092,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))
+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)

{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -1075,11 +1107,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(

- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],

flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);
}

+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**
@@ -1088,14 +1121,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/

-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
- return kmalloc_noprof(bytes, flags);

+ return _kmalloc_noprof(bytes, flags, token);
}

+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**

@@ -1115,18 +1149,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/

-static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p,

+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,

size_t new_n,
size_t new_size,
- gfp_t flags)

+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;

- return krealloc_noprof(p, bytes, flags);

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
}

+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))

/**

@@ -1137,10 +1172,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi
*/

#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

@@ -1183,39 +1219,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_

* @size: how many bytes of memory are required.

* @flags: the type of memory to allocate (see kmalloc).
*/

-static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags)
+static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return kmalloc_noprof(size, flags | __GFP_ZERO);
+ return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}
+#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))

#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node) __alloc_size(1);
#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \
- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node)

+ __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))

#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)

+#define kvmalloc_node_noprof(size, flags, node) \
+ kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)

+#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE)
#define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO)

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \
- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE))
+ alloc_hooks(__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), 1, _flags, NUMA_NO_NODE))

static inline __alloc_size(1, 2) void *

-kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
+_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kvmalloc_node_align_noprof(bytes, 1, flags, node);
+ return __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), 1, flags, node);
}
-
+#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)
#define kvcalloc_node_noprof(_n,_s,_f,_node) kvmalloc_array_node_noprof(_n,_s,(_f)|__GFP_ZERO,_node)
#define kvcalloc_noprof(...) kvcalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE)
@@ -1224,10 +1264,10 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
#define kvcalloc_node(...) alloc_hooks(kvcalloc_node_noprof(__VA_ARGS__))
#define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__))

-void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
+void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,

gfp_t flags, int nid) __realloc_size(2);

-#define kvrealloc_node_align(...) \
- alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__))
+#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
+ alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

#define INIT_KMALLOC_INFO(__size, __short_size) \
@@ -849,7 +849,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
KMALLOC_RCL_NAME(__short_size) \
KMALLOC_CGROUP_NAME(__short_size) \
KMALLOC_DMA_NAME(__short_size) \
- KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size) \
+ KMALLOC_PARTITION_NAME(KMALLOC_PARTITION_CACHES_NR, __short_size) \
.size = __size, \
}

@@ -961,8 +961,8 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
flags |= SLAB_CACHE_DMA;
}

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- if (type >= KMALLOC_RANDOM_START && type <= KMALLOC_RANDOM_END)
+#ifdef CONFIG_KMALLOC_PARTITION_CACHES
+ if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
flags |= SLAB_NO_MERGE;
#endif

@@ -1010,7 +1010,7 @@ void __init create_kmalloc_caches(void)
for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
new_kmalloc_cache(i, type);
}
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
random_kmalloc_seed = get_random_u64();
#endif

diff --git a/mm/slub.c b/mm/slub.c
index 0baa906f39ab..a6e9015601d6 100644
--- a/mm/slub.c
+++ b/mm/slub.c

* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.

* There is no reason to call it again and expect !NULL.

*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
+void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)

{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;

struct kmem_cache *s;
@@ -5346,7 +5348,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);
+ s = kmalloc_slab(size, NULL, alloc_gfp, PASS_TOKEN_PARAM(token));

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*

@@ -5399,12 +5401,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,
int node, unsigned long caller)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);

@@ -6635,7 +6638,7 @@ void kfree_nolock(const void *object)

EXPORT_SYMBOL_GPL(kfree_nolock);

static __always_inline __realloc_size(2) void *
-__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid)
+__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid, kmalloc_token_t token)
{
void *ret;
size_t ks = 0;

@@ -6707,7 +6710,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,

return (void *)p;

alloc_new:
- ret = kmalloc_node_track_caller_noprof(new_size, flags, nid, _RET_IP_);
+ ret = __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(new_size, NULL, token), flags, nid, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();

@@ -6756,7 +6759,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
*

* Return: pointer to the allocated memory or %NULL in case of error

*/
-void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long align,
+void *krealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(new_size, token), unsigned long align,
gfp_t flags, int nid)
{
void *ret;
@@ -6766,7 +6769,7 @@ void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long a

return ZERO_SIZE_PTR;
}

- ret = __do_krealloc(p, new_size, align, flags, nid);
+ ret = __do_krealloc(p, new_size, align, flags, nid, PASS_TOKEN_PARAM(token));
if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret))
kfree(p);

@@ -6803,6 +6806,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

* failure, fall back to non-contiguous (vmalloc) allocation.

* @size: size of the request.

* @b: which set of kmalloc buckets to allocate from.

+ * @token: allocation token.
* @align: desired alignment.

* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.

* @node: numa node to allocate from

@@ -6819,7 +6823,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
*

* Return: pointer to the allocated memory of %NULL in case of failure

*/
-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node)
{
bool allow_block;
@@ -6831,7 +6835,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),
- node, _RET_IP_);
+ node, _RET_IP_, PASS_TOKEN_PARAM(token));
if (ret || size <= PAGE_SIZE)
return ret;

@@ -6927,7 +6931,7 @@ EXPORT_SYMBOL(kvfree_sensitive);
*

* Return: pointer to the allocated memory or %NULL in case of error

*/
-void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
+void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid)
{
void *n;
@@ -6935,10 +6939,10 @@ void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long alig

if (is_vmalloc_addr(p))
return vrealloc_node_align_noprof(p, size, align, flags, nid);

- n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid);
+ n = krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(size, token), align, kmalloc_gfp_adjust(flags, size), nid);
if (!n) {
/* We failed to krealloc(), fall back to kvmalloc(). */
- n = kvmalloc_node_align_noprof(size, align, flags, nid);
+ n = __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), align, flags, nid);
if (!n)
return NULL;

@@ -8453,7 +8457,7 @@ static void __init bootstrap_kmalloc_sheaves(void)

{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--

2.54.0.563.g4f69b47b94-goog

[Marco Elver]

When using CONFIG_KMALLOC_PARTITION_RANDOM, _RET_IP_ was previously used
to identify the allocation site. _RET_IP_, however, evaluates to the
caller's parent's instruction pointer rather than the actual allocation
site; this would lead to collisions where a function performs multiple
allocations.

With the generalization to kmalloc_token_t, we now generate the token at
the outermost macro, and using _THIS_IP_ would fix this for all cases.

Unfortunately, the generic implementation of _THIS_IP_ relies on taking
the address of a local label, which is considered broken by both GCC [1]
and Clang [2] because label addresses are only expected to be used with
computed gotos. While the generic version more or less works today, it
is known to be brittle. For example, Clang -O2 always returns 1 when
this function is inlined:

static inline unsigned long get_ip(void)
{ return ({ __label__ __here; __here: (unsigned long)&&__here; }); }

To provide a reliable unique identifier without breaking architectures
relying on the generic _THIS_IP_, introduce _CODE_LOCATION_: it resolves
to _THIS_IP_ where architectures provide a safe implementation, and
falls back to a zero-cost static marker where _THIS_IP_ is broken.

Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071 [1]
Link: https://github.com/llvm/llvm-project/issues/138272 [2]

Signed-off-by: Marco Elver <el...@google.com>
---

v4:
* New patch.
---
include/linux/instruction_pointer.h | 24 ++++++++++++++++++++++++
include/linux/slab.h | 2 +-
2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/include/linux/instruction_pointer.h b/include/linux/instruction_pointer.h
index aa0b3ffea935..ea5bc756bd99 100644

--- a/include/linux/instruction_pointer.h
+++ b/include/linux/instruction_pointer.h
@@ -8,6 +8,30 @@

#ifndef _THIS_IP_
#define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
+/*
+ * The current generic definition of _THIS_IP_ is considered broken by GCC [1]
+ * and Clang [2]. In particular, the address of a label is only expected to be
+ * used with a computed goto.
+ *
+ * [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071
+ * [2] https://github.com/llvm/llvm-project/issues/138272
+ *
+ * Mark it as broken, so that appropriate fallback options can be implemented

+ * for architectures that do not define their own _THIS_IP_.

+ */
+#define HAS_BROKEN_THIS_IP
+#endif

+

+/*
+ * _CODE_LOCATION_ provides a unique identifier for the current code location.
+ * When _THIS_IP_ is broken (generic version), we fall back to a static marker
+ * which guarantees uniqueness and resolves to a constant address at link time,
+ * avoiding runtime overhead and compiler optimizations breaking it.
+ */
+#ifdef HAS_BROKEN_THIS_IP
+#define _CODE_LOCATION_ ({ static const char __here; (unsigned long)&__here; })

+#else
+#define _CODE_LOCATION_ _THIS_IP_
#endif

#endif /* _LINUX_INSTRUCTION_POINTER_H */
diff --git a/include/linux/slab.h b/include/linux/slab.h
index c232f8a10af6..efab6b2ccf21 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -503,7 +503,7 @@ int kmem_cache_shrink(struct kmem_cache *s);

typedef struct { unsigned long v; } kmalloc_token_t;

#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
extern unsigned long random_kmalloc_seed;
-#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _CODE_LOCATION_ })
#elif defined(CONFIG_KMALLOC_PARTITION_TYPED)

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

#endif
--
2.54.0.563.g4f69b47b94-goog

[Marco Elver]

The mm-api kernel-docs have been disconnected from their symbols. While
the scripts were previously taught to handle the _noprof suffix added by
allocation tagging (in 51a7bf0238c2 "scripts/kernel-doc: drop "_noprof"
on function prototypes"), this does not handle cases where the internal
implementation function has an additional leading underscore. The added
optional parameters (via DECL_KMALLOC_PARAMS) further complicate parsing
the internal signatures.

When the kernel-doc block remains above the internal implementation
function but uses the public API name, the documentation generator fails

to associate the documented symbol.

Simply moving the docs to the macros in slab.h fixes the association but
causes loss of types in the generated documentation (rendering as e.g.
untyped 'kmalloc(size, flags)' macro).

Fix this by:

1. Moving the kernel-doc comment blocks from slub.c to slab.h, placing
them directly above the user-facing macros.

2. Providing explicit, typed C prototypes for the documented APIs inside
'#if 0 /* kernel-doc */' blocks.

3. Converting the variadic macros for the documented APIs to use
explicit arguments to match the documentation.

No functional change intended.

Signed-off-by: Marco Elver <el...@google.com>
---

v4:
* Provide typed C prototypes in '#if 0' blocks to properly render API.
---
include/linux/slab.h | 216 +++++++++++++++++++++++++++++++++----------
mm/slub.c | 98 --------------------
2 files changed, 168 insertions(+), 146 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index efab6b2ccf21..a4bf1585411f 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -530,7 +530,49 @@ void * __must_check krealloc_node_align_noprof(const void *objp,

unsigned long align,
gfp_t flags, int nid) __realloc_size(2);

#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)

#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -913,6 +955,23 @@ void *__kmalloc_large_noprof(size_t size, gfp_t flags)

void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

__assume_page_alignment __alloc_size(1);

+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)

+{
+ if (__builtin_constant_p(size) && size) {
+ unsigned int index;
+
+ if (size > KMALLOC_MAX_CACHE_SIZE)
+ return __kmalloc_large_noprof(size, flags);
+
+ index = kmalloc_index(size);
+ return __kmalloc_cache_noprof(

+ kmalloc_caches[kmalloc_type(flags, token)][index],

+ flags, size);
+ }

+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);

+}
+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
+#if 0 /* kernel-doc */
/**
* kmalloc - allocate kernel memory

* @size: how many bytes of memory are required.

@@ -968,27 +1027,27 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/

-static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
-{
- if (__builtin_constant_p(size) && size) {
- unsigned int index;
-
- if (size > KMALLOC_MAX_CACHE_SIZE)
- return __kmalloc_large_noprof(size, flags);
-
- index = kmalloc_index(size);
- return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, token)][index],
- flags, size);
- }
- return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
-}
-#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))
+void *kmalloc(size_t size, gfp_t flags);
+#endif
+#define kmalloc(size, flags) alloc_hooks(kmalloc_noprof(size, flags))

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);

#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)

-#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))
+#if 0 /* kernel-doc */
+/**
+ * kmalloc_nolock - Allocate an object of given size from any context.
+ * @size: size to allocate
+ * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
+ * allowed.
+ * @node: node number of the target node.
+ *
+ * Return: pointer to the new object or NULL in case of error.
+ * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
+ * There is no reason to call it again and expect !NULL.
+ */
+void *kmalloc_nolock(size_t size, gfp_t gfp_flags, int node);
+#endif
+#define kmalloc_nolock(size, gfp_flags, node) alloc_hooks(kmalloc_nolock_noprof(size, gfp_flags, node))

/**
* __alloc_objs - Allocate objects of a given type using

@@ -1115,23 +1174,40 @@ static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, g

#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)

+{
+ size_t bytes;
+
+ if (unlikely(check_mul_overflow(n, size, &bytes)))
+ return NULL;

+ return _kmalloc_noprof(bytes, flags, token);

+}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
+#if 0 /* kernel-doc */
/**
* kmalloc_array - allocate memory for an array.
* @n: number of elements.

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/

-static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
+void *kmalloc_array(size_t n, size_t size, gfp_t flags);
+#endif
+#define kmalloc_array(n, size, flags) alloc_hooks(kmalloc_array_noprof(n, size, flags))
+

+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,

+ size_t new_n,
+ size_t new_size,

+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

- if (unlikely(check_mul_overflow(n, size, &bytes)))

+ if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;

- return _kmalloc_noprof(bytes, flags, token);
-}
-#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);

+}
+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
+#if 0 /* kernel-doc */
/**
* krealloc_array - reallocate memory for an array.
* @p: pointer to the memory chunk to reallocate

@@ -1149,20 +1225,9 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t si

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/

-static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
- size_t new_n,
- size_t new_size,
- gfp_t flags, kmalloc_token_t token)
-{
- size_t bytes;
-
- if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
- return NULL;
-
- return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
-}
-#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))
+void *krealloc_array(void *p, size_t new_n, size_t new_size, gfp_t flags);
+#endif
+#define krealloc_array(p, new_n, new_size, flags) alloc_hooks(krealloc_array_noprof(p, new_n, new_size, flags))

/**
* kcalloc - allocate memory for an array. The memory is set to zero.
@@ -1214,17 +1279,20 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size
*/
#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO)

-/**
- * kzalloc - allocate memory. The memory is set to zero.

- * @size: how many bytes of memory are required.
- * @flags: the type of memory to allocate (see kmalloc).
- */

static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{

return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}

#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
+#if 0 /* kernel-doc */
+/**
+ * kzalloc - allocate memory. The memory is set to zero.
+ * @size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate (see kmalloc).
+ */
+void *kzalloc(size_t size, gfp_t flags);
+#endif
+#define kzalloc(size, flags) alloc_hooks(kzalloc_noprof(size, flags))

#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

@@ -1233,7 +1301,29 @@ void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long

__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))

kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -1266,8 +1356,38 @@ _kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmallo

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid) __realloc_size(2);

-#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
- alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))
+#if 0 /* kernel-doc */
+/**
+ * kvrealloc_node_align - reallocate memory; contents remain unchanged
+ * @p: object to reallocate memory for
+ * @size: the size to reallocate
+ * @align: desired alignment
+ * @flags: the flags for the page level allocator
+ * @nid: NUMA node id
+ *
+ * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0
+ * and @p is not a %NULL pointer, the object pointed to is freed.
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *

+ * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
+ * initial memory allocation, every subsequent call to this API for the same
+ * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
+ * __GFP_ZERO is not fully honored by this API.
+ *

+ * In any case, the contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes.
+ *
+ * This function must not be called concurrently with itself or kvfree() for the
+ * same memory allocation.
+ *
+ * Return: pointer to the allocated memory or %NULL in case of error
+ */
+void *kvrealloc_node_align(const void *p, size_t size, unsigned long align, gfp_t flags, int nid);
+#endif
+#define kvrealloc_node_align(p, size, align, flags, nid) \
+ alloc_hooks(kvrealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(size, __kmalloc_token(size)), align, flags, nid))

#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/mm/slub.c b/mm/slub.c
index a6e9015601d6..00cb88af79f8 100644
--- a/mm/slub.c
+++ b/mm/slub.c

@@ -5310,17 +5310,6 @@ void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
}
EXPORT_SYMBOL(__kmalloc_noprof);

-/**
- * kmalloc_nolock - Allocate an object of given size from any context.
- * @size: size to allocate
- * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
- * allowed.
- * @node: node number of the target node.
- *
- * Return: pointer to the new object or NULL in case of error.

- * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
- * There is no reason to call it again and expect !NULL.
- */

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)
{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;

@@ -6721,44 +6710,6 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,

- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */

void *krealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(new_size, token), unsigned long align,
gfp_t flags, int nid)
{

@@ -6801,28 +6752,6 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

return flags;
}

-/**
- * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon
- * failure, fall back to non-contiguous (vmalloc) allocation.
- * @size: size of the request.
- * @b: which set of kmalloc buckets to allocate from.
- * @token: allocation token.
- * @align: desired alignment.
- * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
- * @node: numa node to allocate from
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * Uses kmalloc to get the memory but if the allocation fails then falls back
- * to the vmalloc allocator. Use kvfree for freeing the memory.
- *
- * GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not.
- * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is
- * preferable to the vmalloc fallback, due to visible performance drawbacks.
- *

- * Return: pointer to the allocated memory of %NULL in case of failure
- */

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node)

{
@@ -6904,33 +6833,6 @@ void kvfree_sensitive(const void *addr, size_t len)

}
EXPORT_SYMBOL(kvfree_sensitive);

-/**
- * kvrealloc_node_align - reallocate memory; contents remain unchanged
- * @p: object to reallocate memory for
- * @size: the size to reallocate
- * @align: desired alignment
- * @flags: the flags for the page level allocator
- * @nid: NUMA node id
- *
- * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0
- * and @p is not a %NULL pointer, the object pointed to is freed.
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *

- * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
- * initial memory allocation, every subsequent call to this API for the same
- * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
- * __GFP_ZERO is not fully honored by this API.
- *

- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * This function must not be called concurrently with itself or kvfree() for the
- * same memory allocation.
- *

- * Return: pointer to the allocated memory or %NULL in case of error
- */

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid)

{
--
2.54.0.563.g4f69b47b94-goog

[Harry Yoo (Oracle)]

On Mon, May 11, 2026 at 11:34:53AM +0200, Marco Elver wrote:

> On Mon, 11 May 2026 at 10:31, 'Harry Yoo (Oracle)' via kasan-dev
> <kasa...@googlegroups.com> wrote:
> >
> > On Fri, May 08, 2026 at 04:21:36PM +0200, Marco Elver wrote:
> > > I think I have a solution for this mess, see below.
> > >
> > > I would not send it as 1 series, but only include the slab changes (+
> > > instruction_pointer.h change to introduce _CODE_LOCATION_) as one
> > > series, to go through the slab tree. The rest of the patches would go to
> > > respective arch maintainers.
> >
> > I'm assuming this will be a follow-up and reviewing patch 1
> > (and waiting for Jon's thuoghts on patch 2)
>
> I'll be sending v4 shortly.

Thanks!

Ack.

> > Is it something that can be fixed in all arches over time?
>
> Yes, I have patches for that.
>
> > > +/*
> > > + * _CODE_LOCATION_ provides a unique identifier for the current code location.
> > > + * When _THIS_IP_ is broken (generic version), we fall back to a static marker
> > > + * which guarantees uniqueness and resolves to a constant address at link time,
> > > + * avoiding runtime overhead and compiler optimizations breaking it.
> > > + */
> > > +#ifdef HAS_BROKEN_THIS_IP
> > > +#define _CODE_LOCATION_ ({ static const char __here; (unsigned long)&__here; })
> >
> > Nice!
> >
> > Yes, we don't really need the exact code location
> > for partitioning kmalloc caches.
> >
> > IIRC lockdep does a similar thing to define lock classes (unique for
> > each lock init location)
> >
> > > +#else
> > > +#define _CODE_LOCATION_ _THIS_IP_
> > > #endif
> >
> > Probably we don't need this fallback?
>
> x86-64 is the only arch that has working fast _THIS_IP_, and adding
> static __here markers to bss is rather wasteful.

Ouch, I see.

> More architectures will be supporting _THIS_IP_ properly once I get to
> send the patches. The mainstream architectures all have a reasonable
> and fast way to get the current IP, so we don't need to waste bss
> space there.

Ack.

[Harry Yoo (Oracle)]

Looks good to me,
Reviewed-by: Harry Yoo (Oracle) <ha...@kernel.org>

> # Sanity check userspace page table mappings.
> CONFIG_PAGE_TABLE_CHECK=y

> +config RANDOM_KMALLOC_CACHES
> + bool
> + transitional
> + help
> + Transitional config for migration to KMALLOC_PARTITION_CACHES.

Heh, didn't realize transitional config was a thing, nice!

[Harry Yoo (Oracle)]

On Mon, May 11, 2026 at 10:00:49PM +0200, Marco Elver wrote:
> When using CONFIG_KMALLOC_PARTITION_RANDOM, _RET_IP_ was previously used
> to identify the allocation site. _RET_IP_, however, evaluates to the
> caller's parent's instruction pointer rather than the actual allocation
> site; this would lead to collisions where a function performs multiple
> allocations.
>
> With the generalization to kmalloc_token_t, we now generate the token at
> the outermost macro, and using _THIS_IP_ would fix this for all cases.
>
> Unfortunately, the generic implementation of _THIS_IP_ relies on taking
> the address of a local label, which is considered broken by both GCC [1]
> and Clang [2] because label addresses are only expected to be used with
> computed gotos. While the generic version more or less works today, it
> is known to be brittle. For example, Clang -O2 always returns 1 when
> this function is inlined:
>
> static inline unsigned long get_ip(void)
> { return ({ __label__ __here; __here: (unsigned long)&&__here; }); }
>
> To provide a reliable unique identifier without breaking architectures
> relying on the generic _THIS_IP_, introduce _CODE_LOCATION_: it resolves
> to _THIS_IP_ where architectures provide a safe implementation, and
> falls back to a zero-cost static marker where _THIS_IP_ is broken.
>
> Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=120071 [1]
> Link: https://github.com/llvm/llvm-project/issues/138272 [2]
> Signed-off-by: Marco Elver <el...@google.com>
> ---

Looks good to me,
Reviewed-by: Harry Yoo (Oracle) <ha...@kernel.org>

with one suggestion below.

nit: perhaps it can be __initdata to free these after boot?
... if we want to save actual memory allocated rather than the
vmlinux size.

apparently ".init.bss" is a not thing :(

> +#else
> +#define _CODE_LOCATION_ _THIS_IP_
> #endif

[Marco Elver]

Thanks!

Not sure - it might cause CONFIG_DEBUG_SECTION_MISMATCH warnings.
Also, if this memory is reclaimed, it may be reused for kernel
modules, at which point there's a chance for collisions.

[Vlastimil Babka (SUSE)]

On 5/11/26 22:00, Marco Elver wrote:
> When using CONFIG_KMALLOC_PARTITION_RANDOM, _RET_IP_ was previously used
> to identify the allocation site. _RET_IP_, however, evaluates to the
> caller's parent's instruction pointer rather than the actual allocation
> site; this would lead to collisions where a function performs multiple
> allocations.
>
> With the generalization to kmalloc_token_t, we now generate the token at
> the outermost macro, and using _THIS_IP_ would fix this for all cases.

Hm but it means in patch 1 we make things even worse and then fix them
again, and also improve what was suboptimal prior to the series.
Would it be instead possible to reorder patches 1 and 2 so we improve the
current state first, and then introduce typed partitioning without any
changes to the randomized one? (aside from changing the previously correcly
used cases _RET_IP_ to _CODE_LOCATION_).

[Marco Elver]

On Tue, 12 May 2026 at 12:37, 'Vlastimil Babka (SUSE)' via kasan-dev
<kasa...@googlegroups.com> wrote:
> On 5/11/26 22:00, Marco Elver wrote:
> > When using CONFIG_KMALLOC_PARTITION_RANDOM, _RET_IP_ was previously used
> > to identify the allocation site. _RET_IP_, however, evaluates to the
> > caller's parent's instruction pointer rather than the actual allocation
> > site; this would lead to collisions where a function performs multiple
> > allocations.
> >
> > With the generalization to kmalloc_token_t, we now generate the token at
> > the outermost macro, and using _THIS_IP_ would fix this for all cases.
>
> Hm but it means in patch 1 we make things even worse and then fix them
> again, and also improve what was suboptimal prior to the series.
> Would it be instead possible to reorder patches 1 and 2 so we improve the
> current state first, and then introduce typed partitioning without any
> changes to the randomized one? (aside from changing the previously correcly
> used cases _RET_IP_ to _CODE_LOCATION_).

It won't work (it could be made to work if _THIS_IP_ wasn't broken).
The compiler is supposed to maintain semantics of a static variable in
a function, even inline functions, and refer to the same static
variable -- and because kmalloc_type is an inline function, if
_CODE_LOCATION_ is the non-_THIS_IP_ version, it'd break.

Even if _THIS_IP_ wasn't broken, the other complication is introducing
the slab.c vs. outside use of kmalloc_type differentiation.

Both these problems go away if we make this patch 2 (using
_CODE_LOCATION_ on the outer macro, not in an inline function).

While I understand that maybe we could have considered this as a
stable backport, I think it's borderline; the feature isn't broken
per-se, just slightly lower randomness than perhaps intended if size
is a constant expression. A minimal fix prior to the macro rework
currently eludes me.

[Vlastimil Babka (SUSE)]

On 5/12/26 14:54, Marco Elver wrote:
> On Tue, 12 May 2026 at 12:37, 'Vlastimil Babka (SUSE)' via kasan-dev
> <kasa...@googlegroups.com> wrote:
>> On 5/11/26 22:00, Marco Elver wrote:
>> > When using CONFIG_KMALLOC_PARTITION_RANDOM, _RET_IP_ was previously used
>> > to identify the allocation site. _RET_IP_, however, evaluates to the
>> > caller's parent's instruction pointer rather than the actual allocation
>> > site; this would lead to collisions where a function performs multiple
>> > allocations.
>> >
>> > With the generalization to kmalloc_token_t, we now generate the token at
>> > the outermost macro, and using _THIS_IP_ would fix this for all cases.
>>
>> Hm but it means in patch 1 we make things even worse and then fix them
>> again, and also improve what was suboptimal prior to the series.
>> Would it be instead possible to reorder patches 1 and 2 so we improve the
>> current state first, and then introduce typed partitioning without any
>> changes to the randomized one? (aside from changing the previously correcly
>> used cases _RET_IP_ to _CODE_LOCATION_).
>
> It won't work (it could be made to work if _THIS_IP_ wasn't broken).
> The compiler is supposed to maintain semantics of a static variable in
> a function, even inline functions, and refer to the same static
> variable -- and because kmalloc_type is an inline function, if
> _CODE_LOCATION_ is the non-_THIS_IP_ version, it'd break.

Oh, I see.

> Even if _THIS_IP_ wasn't broken, the other complication is introducing
> the slab.c vs. outside use of kmalloc_type differentiation.
>
> Both these problems go away if we make this patch 2 (using
> _CODE_LOCATION_ on the outer macro, not in an inline function).
>
> While I understand that maybe we could have considered this as a
> stable backport, I think it's borderline; the feature isn't broken
> per-se, just slightly lower randomness than perhaps intended if size
> is a constant expression. A minimal fix prior to the macro rework
> currently eludes me.

Fair enough. I wanted to avoid a bisection hazard, but since it's not
feasible and it shouldn't actually break anything (compilation, run), it's
acceptable. Thanks.

[David Laight]

On Mon, 11 May 2026 22:00:49 +0200
Marco Elver <el...@google.com> wrote:

> When using CONFIG_KMALLOC_PARTITION_RANDOM, _RET_IP_ was previously used
> to identify the allocation site. _RET_IP_, however, evaluates to the
> caller's parent's instruction pointer rather than the actual allocation
> site; this would lead to collisions where a function performs multiple
> allocations.
>
> With the generalization to kmalloc_token_t, we now generate the token at
> the outermost macro, and using _THIS_IP_ would fix this for all cases.
>
> Unfortunately, the generic implementation of _THIS_IP_ relies on taking
> the address of a local label, which is considered broken by both GCC [1]
> and Clang [2] because label addresses are only expected to be used with
> computed gotos. While the generic version more or less works today, it
> is known to be brittle. For example, Clang -O2 always returns 1 when
> this function is inlined:
>
> static inline unsigned long get_ip(void)
> { return ({ __label__ __here; __here: (unsigned long)&&__here; }); }
>
> To provide a reliable unique identifier without breaking architectures
> relying on the generic _THIS_IP_, introduce _CODE_LOCATION_: it resolves
> to _THIS_IP_ where architectures provide a safe implementation, and
> falls back to a zero-cost static marker where _THIS_IP_ is broken.

Doesn't that mean that all the other uses of _THIS_IP_ (which seem to mostly
be tracking lock requests) are basically broken on everything except x86-64.

Would it be better to actually fit that?
It isn't as though it is hard asm, you just need to look at how gcc generates
PIC references to static data.

-- David

[Vlastimil Babka (SUSE)]

On 5/11/26 22:00, Marco Elver wrote:

Applied [1] to slab/for-next, thanks. That means including the kernel-doc
workarounds in patch 3. I know Jon said someone might hate it, but maybe it
will motivate them for creating a proper fix :) It seems better than leaving
doc generation broken or not applying this series at all.

https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-7.2/alloc_token

I did the following fixup to remove passing an unnecessary NULL argument for
__kmalloc_nolock() with buckets enabled. Made bloat-o-meter happier a bit.

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c232f8a10af6..795455256329 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -894,7 +894,7 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);

* with the exception of kunit tests
*/

-void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
+void *__kmalloc_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)

@@ -981,7 +981,7 @@ static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t

kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}

- return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
+ return __kmalloc_noprof(PASS_TOKEN_PARAMS(size, token), flags);

}
#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

diff --git a/mm/slub.c b/mm/slub.c
index a6e9015601d6..74652bbdd591 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5303,10 +5303,10 @@ void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, in
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
+void *__kmalloc_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,
- NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
+ return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_,
+ PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

[Harry Yoo (Oracle)]

On Thu, May 14, 2026 at 11:01:26AM +0200, Vlastimil Babka (SUSE) wrote:
> Applied [1] to slab/for-next, thanks. That means including the kernel-doc
> workarounds in patch 3. I know Jon said someone might hate it, but maybe it
> will motivate them for creating a proper fix :) It seems better than leaving
> doc generation broken or not applying this series at all.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-7.2/alloc_token
>
> I did the following fixup to remove passing an unnecessary NULL argument for
> __kmalloc_nolock() with buckets enabled. Made bloat-o-meter happier a bit.

[...]

Looks reasonable to me, thanks!

[Marco Elver]

It seems to mostly work, but yeah, it's probably broken in some places.

> Would it be better to actually fit that?
> It isn't as though it is hard asm, you just need to look at how gcc generates
> PIC references to static data.

Yeah, I have patches waiting to be sent - I sent one for arm64:
https://lore.kernel.org/all/20260511201711....@google.com/

The rest of my patches are to-be-sent, when I get around to testing -
which I didn't yet for most of them since setting up test environments
for all of them is a PITA. Though I can send and let arch maintainers
review them - this would be the tentative list:
https://git.kernel.org/pub/scm/linux/kernel/git/melver/linux.git/log/?h=alloc-token

[Marco Elver]

Thanks!

> https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-7.2/alloc_token
>
> I did the following fixup to remove passing an unnecessary NULL argument for
> __kmalloc_nolock() with buckets enabled. Made bloat-o-meter happier a bit.

Good.

Reviewed-by: Marco Elver <el...@google.com>

Thanks!

[Pedro Falcato]

On Mon, May 11, 2026 at 10:00:48PM +0200, Marco Elver wrote:

Hi,

A couple of questions (I apologise if this was asked before, I wasn't involved
in this thread):

1) What's the object behind kmalloc-part-04? I imagine it's a single type
getting allocated a lot?

2) The bucketing looks quite skewed. Do you have plans to implement something
more similar to what's in the original Apple blog post (with the smaller
granularity and all)? I'm asking because most of our types have pointers in
some way.

3) Obligatory "how about GCC?" :) I quite like the idea behind this feature,
and it would be awesome if it could be more broadly deployed!


In any case, really cool work!

--
Pedro

[Marco Elver]

That's from __kmemdup_nul().

> 2) The bucketing looks quite skewed. Do you have plans to implement something
> more similar to what's in the original Apple blog post (with the smaller
> granularity and all)? I'm asking because most of our types have pointers in
> some way.

Having a scheme more tailored towards kernel data structures would be nice.

But we first need to build experience with this, and get more data. I
think I agree with you that a smaller granularty scheme that tries to
bucket similarly-shaped objects (e.g. with a pointer bitmap) will work
better for the kernel, but I have no evidence of that yet. We need an
analysis of "this scheme would have stopped X out of Y exploit
chains". There are plans to look into that. But if an improvement
comes out of it, it's just a compiler-flag flip away for the kernel,
once the compiler supports it.

There's also what Kees had proposed:
https://lore.kernel.org/lkml/20240809072532...@kernel.org/
.. but that trades memory and performance for stronger partitioning.
Probably should become another KMALLOC_PARTITION variant.

But if performance and memory usage are a concern (which is the case
for environments where I'd like to enable this), we need a smarter
token calculation scheme (if the current one is not good enough after
some analysis).

> 3) Obligatory "how about GCC?" :) I quite like the idea behind this feature,
> and it would be awesome if it could be more broadly deployed!

+cc linux-toolchains

We'd need __builtin_infer_alloc_token + -falloc-token-max= for kernel
support (-fsanitize=alloc-token is not needed if kernel support is all
you'd care about):
https://clang.llvm.org/docs/AllocToken.html#querying-token-ids-with-builtin-infer-alloc-token

Thanks,
-- Marco

[Harry Yoo]

__kmemdup_nul() is probably a good fit for SLAB_BUCKETS?