[PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning

[Marco Elver]

Rework the general infrastructure around RANDOM_KMALLOC_CACHES into more
flexible PARTITION_KMALLOC_CACHES, with the former being a partitioning
mode of the latter.

Introduce a new mode, TYPED_KMALLOC_CACHES, which leverages a feature
available in Clang 22 and later, called "allocation tokens" via
__builtin_infer_alloc_token [1]. Unlike RANDOM_KMALLOC_CACHES, this mode
deterministically assigns a slab cache to an allocation of type T,
regardless of allocation site.

The builtin __builtin_infer_alloc_token(<malloc-args>, ...) instructs
the compiler to infer an allocation type from arguments commonly passed
to memory-allocating functions and returns a type-derived token ID. The
implementation passes kmalloc-args to the builtin: the compiler performs
best-effort type inference, and then recognizes common patterns such as
`kmalloc(sizeof(T), ...)`, `kmalloc(sizeof(T) * n, ...)`, but also
`(T *)kmalloc(...)`. Where the compiler fails to infer a type the
fallback token (default: 0) is chosen.

Note: kmalloc_obj(..) APIs fix the pattern how size and result type are
expressed, and therefore ensures there's not much drift in which
patterns the compiler needs to recognize. Specifically, kmalloc_obj()
and friends expand to `(TYPE *)KMALLOC(__obj_size, GFP)`, which the
compiler recognizes via the cast to TYPE*.

Clang's default token ID calculation is described as [1]:

typehashpointersplit: This mode assigns a token ID based on the hash
of the allocated type's name, where the top half ID-space is reserved
for types that contain pointers and the bottom half for types that do
not contain pointers.

Separating pointer-containing objects from pointerless objects and data
allocations can help mitigate certain classes of memory corruption
exploits [2]: attackers who gains a buffer overflow on a primitive
buffer cannot use it to directly corrupt pointers or other critical
metadata in an object residing in a different, isolated heap region.

It is important to note that heap isolation strategies offer a
best-effort approach, and do not provide a 100% security guarantee,
albeit achievable at relatively low performance cost. Note that this
also does not prevent cross-cache attacks, and SLAB_VIRTUAL [3] should
be used as a complementary mitigation (once available).

With all that, my kernel (x86 defconfig) shows me a histogram of slab
cache object distribution per /proc/slabinfo (after boot):

<slab cache> <objs> <hist>
kmalloc-part-15 1537 +++++++++++++++
kmalloc-part-14 2996 +++++++++++++++++++++++++++++
kmalloc-part-13 1555 +++++++++++++++
kmalloc-part-12 1045 ++++++++++
kmalloc-part-11 1717 +++++++++++++++++
kmalloc-part-10 1489 ++++++++++++++
kmalloc-part-09 851 ++++++++
kmalloc-part-08 710 +++++++
kmalloc-part-07 100 +
kmalloc-part-06 217 ++
kmalloc-part-05 105 +
kmalloc-part-04 4047 ++++++++++++++++++++++++++++++++++++++++
kmalloc-part-03 276 ++
kmalloc-part-02 283 ++
kmalloc-part-01 316 +++
kmalloc 1599 +++++++++++++++

The above /proc/slabinfo snapshot shows me there are 6943 allocated
objects (slabs 00 - 07) that the compiler claims contain no pointers or
it was unable to infer the type of, and 11900 objects that contain
pointers (slabs 08 - 15). On a whole, this looks relatively sane.

Additionally, when I compile my kernel with -Rpass=alloc-token, which
provides diagnostics where (after dead-code elimination) type inference
failed, I see 179 allocation sites where the compiler failed to identify
a type (down from 966 when I sent the RFC [4]). Some initial review
confirms these are mostly variable sized buffers, but also include
structs with trailing flexible length arrays.

Link: https://clang.llvm.org/docs/AllocToken.html [1]
Link: https://blog.dfsec.com/ios/2025/05/30/blasting-past-ios-18/ [2]
Link: https://lwn.net/Articles/944647/ [3]
Link: https://lore.kernel.org/all/20250825154505....@google.com/ [4]
Link: https://discourse.llvm.org/t/rfc-a-framework-for-allocator-partitioning-hints/87434
Signed-off-by: Marco Elver <el...@google.com>
---
Changelog:
v1:
* Rebase and switch to builtin name that was released in Clang 22.
* Keep RANDOM_KMALLOC_CACHES the default.

RFC: https://lore.kernel.org/all/20250825154505....@google.com/
---
Makefile | 5 ++
include/linux/percpu.h | 2 +-
include/linux/slab.h | 94 ++++++++++++++++++++-------------
kernel/configs/hardening.config | 2 +-
mm/Kconfig | 45 ++++++++++++----
mm/kfence/kfence_test.c | 4 +-
mm/slab.h | 4 +-
mm/slab_common.c | 48 ++++++++---------
mm/slub.c | 31 +++++------
9 files changed, 144 insertions(+), 91 deletions(-)

diff --git a/Makefile b/Makefile
index 2294decf0afc..93bb704bbf0e 100644
--- a/Makefile
+++ b/Makefile
@@ -957,6 +957,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)
endif
endif

+ifdef CONFIG_TYPED_KMALLOC_CACHES
+# PARTITION_KMALLOC_CACHES_NR + 1
+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 85bf8dd9f087..271b41be314d 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..c0bf00ee6025 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -612,10 +612,10 @@ static inline unsigned int arch_slab_minalign(void)
#define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \
(KMALLOC_MIN_SIZE) : 16)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define PARTITION_KMALLOC_CACHES_NR 15 // # of cache copies
#else
-#define RANDOM_KMALLOC_CACHES_NR 0
+#define PARTITION_KMALLOC_CACHES_NR 0
#endif

/*
@@ -634,8 +634,8 @@ enum kmalloc_cache_type {
#ifndef CONFIG_MEMCG
KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
- KMALLOC_RANDOM_START = KMALLOC_NORMAL,
- KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
+ KMALLOC_PARTITION_START = KMALLOC_NORMAL,
+ KMALLOC_PARTITION_END = KMALLOC_PARTITION_START + PARTITION_KMALLOC_CACHES_NR,
#ifdef CONFIG_SLUB_TINY
KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else
@@ -662,9 +662,20 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];
(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))

+#ifdef CONFIG_RANDOM_KMALLOC_CACHES
extern unsigned long random_kmalloc_seed;
+typedef struct { unsigned long ip; } kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t) { .ip = _RET_IP_ })
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+typedef struct { unsigned long v; } kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })
+#else
+/* no-op */
+typedef struct {} kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){})
+#endif

-static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
+static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
{
/*
* The most common case is KMALLOC_NORMAL, so test for it
@@ -672,9 +683,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
- return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
- ilog2(RANDOM_KMALLOC_CACHES_NR + 1));
+ /* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
+ return KMALLOC_PARTITION_START + hash_64(token.ip ^ random_kmalloc_seed,
+ ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+ return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;
#endif
@@ -864,10 +877,10 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)
@@ -938,7 +951,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;
@@ -948,14 +961,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}
- return __kmalloc_noprof(size, flags);
+ return __kmalloc_noprof(size, flags, token);
}
+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_token_t token);
+#define kmalloc_nolock_noprof(...) _kmalloc_nolock_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1075,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))
+ alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, __kmalloc_token(_size)))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_, __kmalloc_token(_size)))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;
@@ -1075,11 +1090,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);
+ return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, token);
}
+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**
@@ -1088,14 +1104,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf
* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
- return kmalloc_noprof(bytes, flags);
+ return _kmalloc_noprof(bytes, flags, token);
}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**
@@ -1138,9 +1155,9 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi
#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,
- unsigned long caller) __alloc_size(1);
+ unsigned long caller, kmalloc_token_t token) __alloc_size(1);
#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \
- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller)
+ __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller, __kmalloc_token(size))
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1157,17 +1174,18 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag
#define kmalloc_track_caller_noprof(...) \
kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_)

-static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
+ int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
if (__builtin_constant_p(n) && __builtin_constant_p(size))
- return kmalloc_node_noprof(bytes, flags, node);
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node);
+ return _kmalloc_node_noprof(bytes, flags, node, token);
+ return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node, token);
}
+#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

#define kcalloc_node(_n, _size, _flags, _node) \
@@ -1183,39 +1201,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_
* @size: how many bytes of memory are required.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags)
+static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return kmalloc_noprof(size, flags | __GFP_ZERO);
+ return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}
+#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node) __alloc_size(1);
+ gfp_t flags, int node, kmalloc_token_t token) __alloc_size(1);
#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \
- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node)
+ __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node, __kmalloc_token(_size))
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)
+#define kvmalloc_node_noprof(size, flags, node) \
+ kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)
+#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE)
#define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO)

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \
- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE))
+ alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE, __kmalloc_token(_size)))

static inline __alloc_size(1, 2) void *
-kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
+_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kvmalloc_node_align_noprof(bytes, 1, flags, node);
+ return __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), 1, flags, node, token);
}
-
+#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)
#define kvcalloc_node_noprof(_n,_s,_f,_node) kvmalloc_array_node_noprof(_n,_s,(_f)|__GFP_ZERO,_node)
#define kvcalloc_noprof(...) kvcalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE)
diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config
index 7c3924614e01..2963b6bd890f 100644
--- a/kernel/configs/hardening.config
+++ b/kernel/configs/hardening.config
@@ -22,7 +22,7 @@ CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
CONFIG_SLAB_BUCKETS=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-CONFIG_RANDOM_KMALLOC_CACHES=y
+CONFIG_PARTITION_KMALLOC_CACHES=y

# Sanity check userspace page table mappings.
CONFIG_PAGE_TABLE_CHECK=y
diff --git a/mm/Kconfig b/mm/Kconfig
index ebd8ea353687..fa4ffc1fcb80 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -247,22 +247,47 @@ config SLUB_STATS
out which slabs are relevant to a particular load.
Try running: slabinfo -DA

-config RANDOM_KMALLOC_CACHES
- default n
+config PARTITION_KMALLOC_CACHES
depends on !SLUB_TINY
- bool "Randomize slab caches for normal kmalloc"
+ bool "Partitioned slab caches for normal kmalloc"
help
- A hardening feature that creates multiple copies of slab caches for
- normal kmalloc allocation and makes kmalloc randomly pick one based
- on code address, which makes the attackers more difficult to spray
- vulnerable memory objects on the heap for the purpose of exploiting
- memory vulnerabilities.
+ A hardening feature that creates multiple isolated copies of slab
+ caches for normal kmalloc allocations. This makes it more difficult
+ to exploit memory-safety vulnerabilities by attacking vulnerable
+ co-located memory objects. Several modes are provided.

Currently the number of copies is set to 16, a reasonably large value
that effectively diverges the memory objects allocated for different
subsystems or modules into different caches, at the expense of a
- limited degree of memory and CPU overhead that relates to hardware and
- system workload.
+ limited degree of memory and CPU overhead that relates to hardware
+ and system workload.
+
+choice
+ prompt "Partitioned slab cache mode"
+ depends on PARTITION_KMALLOC_CACHES
+ default RANDOM_KMALLOC_CACHES
+ help
+ Selects the slab cache partitioning mode.
+
+config RANDOM_KMALLOC_CACHES
+ bool "Randomize slab caches for normal kmalloc"
+ help
+ Randomly pick a slab cache based on code address.
+
+config TYPED_KMALLOC_CACHES
+ bool "Type based slab cache selection for normal kmalloc"
+ depends on $(cc-option,-falloc-token-max=123)
+ help
+ Rely on Clang's allocation tokens to choose a slab cache, where token
+ IDs are derived from the allocated type.
+
+ The current effectiveness of Clang's type inference can be judged by
+ -Rpass=alloc-token, which provides diagnostics where (after dead-code
+ elimination) type inference failed.
+
+ Requires Clang 22 or later.
+
+endchoice

endmenu # Slab allocator options

diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
index 5725a367246d..8807ea8ed0d3 100644
--- a/mm/kfence/kfence_test.c
+++ b/mm/kfence/kfence_test.c
@@ -214,7 +214,7 @@ static void test_cache_destroy(void)
static inline size_t kmalloc_cache_alignment(size_t size)
{
/* just to get ->align so no need to pass in the real caller */
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, 0);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(0));
return kmalloc_caches[type][__kmalloc_index(size, false)]->align;
}

@@ -285,7 +285,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat

if (is_kfence_address(alloc)) {
struct slab *slab = virt_to_slab(alloc);
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, _RET_IP_);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(size));
struct kmem_cache *s = test_cache ?:
kmalloc_caches[type][__kmalloc_index(size, false)];

diff --git a/mm/slab.h b/mm/slab.h
index e9ab292acd22..dd49d37e253d 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -361,12 +361,12 @@ static inline unsigned int size_index_elem(unsigned int bytes)
* KMALLOC_MAX_CACHE_SIZE and the caller must check that.
*/
static inline struct kmem_cache *
-kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller)
+kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, kmalloc_token_t token)
{
unsigned int index;

if (!b)
- b = &kmalloc_caches[kmalloc_type(flags, caller)];
+ b = &kmalloc_caches[kmalloc_type(flags, token)];
if (size <= 192)
index = kmalloc_size_index[size_index_elem(size)];
else
diff --git a/mm/slab_common.c b/mm/slab_common.c
index d5a70a831a2a..21ab7dd79b5e 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -787,7 +787,7 @@ size_t kmalloc_size_roundup(size_t size)
* The flags don't matter since size_index is common to all.
* Neither does the caller for just getting ->object_size.
*/
- return kmalloc_slab(size, NULL, GFP_KERNEL, 0)->object_size;
+ return kmalloc_slab(size, NULL, GFP_KERNEL, __kmalloc_token(0))->object_size;
}

/* Above the smaller buckets, size is a multiple of page size. */
@@ -821,26 +821,26 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
#define KMALLOC_RCL_NAME(sz)
#endif

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define __KMALLOC_RANDOM_CONCAT(a, b) a ## b
-#define KMALLOC_RANDOM_NAME(N, sz) __KMALLOC_RANDOM_CONCAT(KMA_RAND_, N)(sz)
-#define KMA_RAND_1(sz) .name[KMALLOC_RANDOM_START + 1] = "kmalloc-rnd-01-" #sz,
-#define KMA_RAND_2(sz) KMA_RAND_1(sz) .name[KMALLOC_RANDOM_START + 2] = "kmalloc-rnd-02-" #sz,
-#define KMA_RAND_3(sz) KMA_RAND_2(sz) .name[KMALLOC_RANDOM_START + 3] = "kmalloc-rnd-03-" #sz,
-#define KMA_RAND_4(sz) KMA_RAND_3(sz) .name[KMALLOC_RANDOM_START + 4] = "kmalloc-rnd-04-" #sz,
-#define KMA_RAND_5(sz) KMA_RAND_4(sz) .name[KMALLOC_RANDOM_START + 5] = "kmalloc-rnd-05-" #sz,
-#define KMA_RAND_6(sz) KMA_RAND_5(sz) .name[KMALLOC_RANDOM_START + 6] = "kmalloc-rnd-06-" #sz,
-#define KMA_RAND_7(sz) KMA_RAND_6(sz) .name[KMALLOC_RANDOM_START + 7] = "kmalloc-rnd-07-" #sz,
-#define KMA_RAND_8(sz) KMA_RAND_7(sz) .name[KMALLOC_RANDOM_START + 8] = "kmalloc-rnd-08-" #sz,
-#define KMA_RAND_9(sz) KMA_RAND_8(sz) .name[KMALLOC_RANDOM_START + 9] = "kmalloc-rnd-09-" #sz,
-#define KMA_RAND_10(sz) KMA_RAND_9(sz) .name[KMALLOC_RANDOM_START + 10] = "kmalloc-rnd-10-" #sz,
-#define KMA_RAND_11(sz) KMA_RAND_10(sz) .name[KMALLOC_RANDOM_START + 11] = "kmalloc-rnd-11-" #sz,
-#define KMA_RAND_12(sz) KMA_RAND_11(sz) .name[KMALLOC_RANDOM_START + 12] = "kmalloc-rnd-12-" #sz,
-#define KMA_RAND_13(sz) KMA_RAND_12(sz) .name[KMALLOC_RANDOM_START + 13] = "kmalloc-rnd-13-" #sz,
-#define KMA_RAND_14(sz) KMA_RAND_13(sz) .name[KMALLOC_RANDOM_START + 14] = "kmalloc-rnd-14-" #sz,
-#define KMA_RAND_15(sz) KMA_RAND_14(sz) .name[KMALLOC_RANDOM_START + 15] = "kmalloc-rnd-15-" #sz,
-#else // CONFIG_RANDOM_KMALLOC_CACHES
-#define KMALLOC_RANDOM_NAME(N, sz)
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define __KMALLOC_PARTITION_CONCAT(a, b) a ## b
+#define KMALLOC_PARTITION_NAME(N, sz) __KMALLOC_PARTITION_CONCAT(KMA_PART_, N)(sz)
+#define KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 1] = "kmalloc-part-01-" #sz,
+#define KMA_PART_2(sz) KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 2] = "kmalloc-part-02-" #sz,
+#define KMA_PART_3(sz) KMA_PART_2(sz) .name[KMALLOC_PARTITION_START + 3] = "kmalloc-part-03-" #sz,
+#define KMA_PART_4(sz) KMA_PART_3(sz) .name[KMALLOC_PARTITION_START + 4] = "kmalloc-part-04-" #sz,
+#define KMA_PART_5(sz) KMA_PART_4(sz) .name[KMALLOC_PARTITION_START + 5] = "kmalloc-part-05-" #sz,
+#define KMA_PART_6(sz) KMA_PART_5(sz) .name[KMALLOC_PARTITION_START + 6] = "kmalloc-part-06-" #sz,
+#define KMA_PART_7(sz) KMA_PART_6(sz) .name[KMALLOC_PARTITION_START + 7] = "kmalloc-part-07-" #sz,
+#define KMA_PART_8(sz) KMA_PART_7(sz) .name[KMALLOC_PARTITION_START + 8] = "kmalloc-part-08-" #sz,
+#define KMA_PART_9(sz) KMA_PART_8(sz) .name[KMALLOC_PARTITION_START + 9] = "kmalloc-part-09-" #sz,
+#define KMA_PART_10(sz) KMA_PART_9(sz) .name[KMALLOC_PARTITION_START + 10] = "kmalloc-part-10-" #sz,
+#define KMA_PART_11(sz) KMA_PART_10(sz) .name[KMALLOC_PARTITION_START + 11] = "kmalloc-part-11-" #sz,
+#define KMA_PART_12(sz) KMA_PART_11(sz) .name[KMALLOC_PARTITION_START + 12] = "kmalloc-part-12-" #sz,
+#define KMA_PART_13(sz) KMA_PART_12(sz) .name[KMALLOC_PARTITION_START + 13] = "kmalloc-part-13-" #sz,
+#define KMA_PART_14(sz) KMA_PART_13(sz) .name[KMALLOC_PARTITION_START + 14] = "kmalloc-part-14-" #sz,
+#define KMA_PART_15(sz) KMA_PART_14(sz) .name[KMALLOC_PARTITION_START + 15] = "kmalloc-part-15-" #sz,
+#else // CONFIG_PARTITION_KMALLOC_CACHES
+#define KMALLOC_PARTITION_NAME(N, sz)
#endif

#define INIT_KMALLOC_INFO(__size, __short_size) \
@@ -849,7 +849,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
KMALLOC_RCL_NAME(__short_size) \
KMALLOC_CGROUP_NAME(__short_size) \
KMALLOC_DMA_NAME(__short_size) \
- KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size) \
+ KMALLOC_PARTITION_NAME(PARTITION_KMALLOC_CACHES_NR, __short_size) \
.size = __size, \
}

@@ -961,8 +961,8 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
flags |= SLAB_CACHE_DMA;
}

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- if (type >= KMALLOC_RANDOM_START && type <= KMALLOC_RANDOM_END)
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+ if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
flags |= SLAB_NO_MERGE;
#endif

diff --git a/mm/slub.c b/mm/slub.c
index 2b2d33cc735c..b13d1117c87d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2125,7 +2125,7 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,
if (!is_kmalloc_normal(s))
return sz;

- obj_exts_cache = kmalloc_slab(sz, NULL, gfp, 0);
+ obj_exts_cache = kmalloc_slab(sz, NULL, gfp, __kmalloc_token(0));
/*
* We can't simply compare s with obj_exts_cache, because random kmalloc
* caches have multiple caches per size, selected by caller address.
@@ -5239,7 +5239,7 @@ EXPORT_SYMBOL(__kmalloc_large_node_noprof);

static __always_inline
void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
- unsigned long caller)
+ unsigned long caller, kmalloc_token_t token)
{
struct kmem_cache *s;
void *ret;
@@ -5254,22 +5254,22 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
if (unlikely(!size))
return ZERO_SIZE_PTR;

- s = kmalloc_slab(size, b, flags, caller);
+ s = kmalloc_slab(size, b, flags, token);

ret = slab_alloc_node(s, NULL, flags, node, caller, size);
ret = kasan_kmalloc(s, ret, size, flags);
trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}
-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_, token);
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_);
+ return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_, token);
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5284,7 +5284,7 @@ EXPORT_SYMBOL(__kmalloc_noprof);
* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
* There is no reason to call it again and expect !NULL.
*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_token_t token)
{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
struct kmem_cache *s;
@@ -5307,7 +5307,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);
+ s = kmalloc_slab(size, NULL, alloc_gfp, token);

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*
@@ -5360,12 +5360,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)
ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
- int node, unsigned long caller)
+ int node, unsigned long caller, kmalloc_token_t token)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller, token);

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
@@ -6726,6 +6726,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
* @align: desired alignment.
* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
* @node: numa node to allocate from
+ * @token: allocation token.
*
* Only alignments up to those guaranteed by kmalloc() will be honored. Please see
* Documentation/core-api/memory-allocation.rst for more details.
@@ -6740,7 +6741,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
* Return: pointer to the allocated memory of %NULL in case of failure
*/
void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node)
+ gfp_t flags, int node, kmalloc_token_t token)
{
bool allow_block;
void *ret;
@@ -6751,7 +6752,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),
- node, _RET_IP_);
+ node, _RET_IP_, token);
if (ret || size <= PAGE_SIZE)
return ret;

@@ -8351,7 +8352,7 @@ static void __init bootstrap_kmalloc_sheaves(void)
{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--
2.53.0.1018.g2bb0e51243-goog

[Dan Carpenter]

Hi Marco,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url: https://github.com/intel-lab-lkp/linux/commits/Marco-Elver/slab-support-for-compiler-assisted-type-based-slab-cache-partitioning/20260401-035608
base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link: https://lore.kernel.org/r/20260331111240.153913-1-elver%40google.com
patch subject: [PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning
config: um-randconfig-r072-20260401
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
smatch: v0.5.0-9004-gb810ac53

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <l...@intel.com>
| Reported-by: Dan Carpenter <err...@gmail.com>
| Closes: https://lore.kernel.org/r/202604020400...@intel.com/

New smatch warnings:
drivers/misc/lkdtm/heap.c:118 lkdtm_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
drivers/misc/lkdtm/heap.c:169 lkdtm_KFENCE_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)

vim +118 drivers/misc/lkdtm/heap.c

73f62e60d80c2d drivers/misc/lkdtm/heap.c Kees Cook 2022-03-03 92 static void lkdtm_READ_AFTER_FREE(void)
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 93 {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 94 int *base, *val, saw;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 95 size_t len = 1024;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 96 /*
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 97 * The slub allocator will use the either the first word or
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 98 * the middle of the allocation to store the free pointer,
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 99 * depending on configurations. Store in the second word to
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 100 * avoid running into the freelist.
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 101 */
e12145cf1c3a80 drivers/misc/lkdtm/heap.c Kees Cook 2020-06-25 102 size_t offset = sizeof(*base);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 103
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 104 base = kmalloc(len, GFP_KERNEL);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 105 if (!base) {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 106 pr_info("Unable to allocate base memory.\n");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 107 return;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 108 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 109
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 110 val = kmalloc(len, GFP_KERNEL);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 111 if (!val) {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 112 pr_info("Unable to allocate val memory.\n");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 113 kfree(base);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 114 return;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 115 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 116
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 117 *val = 0x12345678;
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 @118 base[offset] = *val;

This doesn't really matter, but the comment says we are writing to the
second word. The base[] array holds 256 integers, so:

base[sizeof(int)] = *val;

would be the third word, right? A word is unsigned long, right? All
of a sudden I am unsure. :P

ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 119 pr_info("Value in memory before free: %x\n", base[offset]);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 120
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 121 kfree(base);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 122
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 123 pr_info("Attempting bad read from freed memory\n");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 124 saw = base[offset];
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 125 if (saw != *val) {
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 126 /* Good! Poisoning happened, so declare a win. */
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 127 pr_info("Memory correctly poisoned (%x)\n", saw);
5b777131bd8005 drivers/misc/lkdtm/heap.c Kees Cook 2021-06-23 128 } else {
5b777131bd8005 drivers/misc/lkdtm/heap.c Kees Cook 2021-06-23 129 pr_err("FAIL: Memory was not poisoned!\n");
5b777131bd8005 drivers/misc/lkdtm/heap.c Kees Cook 2021-06-23 130 pr_expected_config_param(CONFIG_INIT_ON_FREE_DEFAULT_ON, "init_on_free");
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 131 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 132
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 133 kfree(val);
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 134 }
ffc514f3fcac4a drivers/misc/lkdtm_heap.c Kees Cook 2016-06-26 135
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 136 static void lkdtm_KFENCE_READ_AFTER_FREE(void)
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 137 {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 138 int *base, val, saw;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 139 unsigned long timeout, resched_after;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 140 size_t len = 1024;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 141 /*
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 142 * The slub allocator will use the either the first word or
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 143 * the middle of the allocation to store the free pointer,
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 144 * depending on configurations. Store in the second word to
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 145 * avoid running into the freelist.
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 146 */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 147 size_t offset = sizeof(*base);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 148
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 149 /*
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 150 * 100x the sample interval should be more than enough to ensure we get
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 151 * a KFENCE allocation eventually.
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 152 */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 153 timeout = jiffies + msecs_to_jiffies(100 * kfence_sample_interval);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 154 /*
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 155 * Especially for non-preemption kernels, ensure the allocation-gate
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 156 * timer can catch up: after @resched_after, every failed allocation
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 157 * attempt yields, to ensure the allocation-gate timer is scheduled.
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 158 */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 159 resched_after = jiffies + msecs_to_jiffies(kfence_sample_interval);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 160 do {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 161 base = kmalloc(len, GFP_KERNEL);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 162 if (!base) {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 163 pr_err("FAIL: Unable to allocate kfence memory!\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 164 return;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 165 }
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 166
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 167 if (is_kfence_address(base)) {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 168 val = 0x12345678;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 @169 base[offset] = val;
^^^^^^^^^^^^^^^^^^
Same here.

aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 170 pr_info("Value in memory before free: %x\n", base[offset]);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 171
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 172 kfree(base);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 173
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 174 pr_info("Attempting bad read from freed memory\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 175 saw = base[offset];
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 176 if (saw != val) {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 177 /* Good! Poisoning happened, so declare a win. */
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 178 pr_info("Memory correctly poisoned (%x)\n", saw);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 179 } else {
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 180 pr_err("FAIL: Memory was not poisoned!\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 181 pr_expected_config_param(CONFIG_INIT_ON_FREE_DEFAULT_ON, "init_on_free");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 182 }
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 183 return;
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 184 }
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 185
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 186 kfree(base);
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 187 if (time_after(jiffies, resched_after))
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 188 cond_resched();
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 189 } while (time_before(jiffies, timeout));
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 190
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 191 pr_err("FAIL: kfence memory never allocated!\n");
aabf7c37dfbce3 drivers/misc/lkdtm/heap.c Stephen Boyd 2023-11-29 192 }

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

[Marco Elver]

On Thu, 2 Apr 2026 at 15:33, Dan Carpenter <err...@gmail.com> wrote:
>
> Hi Marco,
>
> kernel test robot noticed the following build warnings:
>
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url: https://github.com/intel-lab-lkp/linux/commits/Marco-Elver/slab-support-for-compiler-assisted-type-based-slab-cache-partitioning/20260401-035608
> base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
> patch link: https://lore.kernel.org/r/20260331111240.153913-1-elver%40google.com
> patch subject: [PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning
> config: um-randconfig-r072-20260401
> compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
> smatch: v0.5.0-9004-gb810ac53
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <l...@intel.com>
> | Reported-by: Dan Carpenter <err...@gmail.com>
> | Closes: https://lore.kernel.org/r/202604020400...@intel.com/
>
> New smatch warnings:
> drivers/misc/lkdtm/heap.c:118 lkdtm_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
> drivers/misc/lkdtm/heap.c:169 lkdtm_KFENCE_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)

How is this related to the patch I sent? Did the <linux/slab.h> change
force rechecking of all these files and it found latent issues?

???

So this is clearly a minor defect in this existing code (comment wrong
or code might want to match what the comment said), but "slab: support
for compiler-assisted type-based slab cache partitioning" didn't touch
that.

Unrelated to "slab: support for compiler-assisted type-based slab
cache partitioning".

[Dan Carpenter]

On Thu, Apr 02, 2026 at 03:48:20PM +0200, Marco Elver wrote:
> On Thu, 2 Apr 2026 at 15:33, Dan Carpenter <err...@gmail.com> wrote:
> >
> > Hi Marco,
> >
> > kernel test robot noticed the following build warnings:
> >
> > https://git-scm.com/docs/git-format-patch#_base_tree_information]
> >
> > url: https://github.com/intel-lab-lkp/linux/commits/Marco-Elver/slab-support-for-compiler-assisted-type-based-slab-cache-partitioning/20260401-035608
> > base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
> > patch link: https://lore.kernel.org/r/20260331111240.153913-1-elver%40google.com
> > patch subject: [PATCH v1] slab: support for compiler-assisted type-based slab cache partitioning
> > config: um-randconfig-r072-20260401
> > compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
> > smatch: v0.5.0-9004-gb810ac53
> >
> > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > the same patch/commit), kindly add following tags
> > | Reported-by: kernel test robot <l...@intel.com>
> > | Reported-by: Dan Carpenter <err...@gmail.com>
> > | Closes: https://lore.kernel.org/r/202604020400...@intel.com/
> >
> > New smatch warnings:
> > drivers/misc/lkdtm/heap.c:118 lkdtm_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
> > drivers/misc/lkdtm/heap.c:169 lkdtm_KFENCE_READ_AFTER_FREE() warn: potential pointer math issue ('base' is a 32 bit pointer)
>
> How is this related to the patch I sent? Did the <linux/slab.h> change
> force rechecking of all these files and it found latent issues?
>

Oh, crud. It turns out that for this check Smatch allows
integer_array[sizeof()]so long as we know that the index is within
bounds. What happened is that your patch renamed the kmalloc()
function so Smatch stopped knowing the size of the buffer.

For these zero day bot warnings, the emails are automatically generated
so I don't have any context outside what's in the email. I saw that
Kees wrote the code, but I figured maybe you forwarded it or something.
Sorry about that.

regards,
dan carpenter

[Marco Elver]

Oh, I see. Smatch doesn't respect the __alloc_size attribute then?

> For these zero day bot warnings, the emails are automatically generated
> so I don't have any context outside what's in the email. I saw that
> Kees wrote the code, but I figured maybe you forwarded it or something.
> Sorry about that.

No worries and thanks!

[Harry Yoo (Oracle)]

Presumably because only the latest Clang supports it?

> RFC: https://lore.kernel.org/all/20250825154505....@google.com/
> ---
> Makefile | 5 ++
> include/linux/percpu.h | 2 +-
> include/linux/slab.h | 94 ++++++++++++++++++++-------------
> kernel/configs/hardening.config | 2 +-
> mm/Kconfig | 45 ++++++++++++----
> mm/kfence/kfence_test.c | 4 +-
> mm/slab.h | 4 +-
> mm/slab_common.c | 48 ++++++++---------
> mm/slub.c | 31 +++++------
> 9 files changed, 144 insertions(+), 91 deletions(-)
>

> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 15a60b501b95..c0bf00ee6025 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h

> @@ -864,10 +877,10 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
> * with the exception of kunit tests
> */
>
> -void *__kmalloc_noprof(size_t size, gfp_t flags)
> +void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
> __assume_kmalloc_alignment __alloc_size(1);
>
> -void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
> +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
> __assume_kmalloc_alignment __alloc_size(1);

So the @token parameter is unused when CONFIG_PARTITION_KMALLOC_CACHES is
disabled but still increases the kernel size by a few kilobytes...
but yeah I'm not sure if we can get avoid it without hurting readability.

Just saying. (does anybody care?)

> void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

Assuming not all people building the kernel are security experts...
(including myself) could you please add some insights/guidance on how to
decide between RANDOM_KMALLOC_CACHES and TYPED_KMALLOC_CACHES?

Something like what Florent wrote [1]:
| One more perspective on this: in a data center environment, attackers
| typically get a first foothold by compromising a userspace network
| service. If they can do that once, they can do that a bunch of times,
| and gain code execution on different machines every time.
|
| Before trying to exploit a kernel memory corruption to elevate
| privileges on a machine, they can test the SLAB properties of the
| running kernel to make sure it's as they wish (eg: with timing side
| channels like in the SLUBStick paper). So with RANDOM_KMALLOC_CACHES,
| attackers can just keep retrying their attacks until they land on a
| machine where the types T and S are collocated and only then proceed
| with their exploit.
|
| With TYPED_KMALLOC_CACHES (and with SLAB_VIRTUAL hopefully someday),
| they are simply never able to cross the "objects without pointers" to
| "objects with pointers" boundary which really gets in the way of many
| exploitation techniques and feels at least to me like a much stronger
| security boundary.
|
| This limit of RANDOM_KMALLOC_CACHES may not be as relevant in other
| deployments (eg: on a smartphone) but it makes me strongly prefer
| TYPED_KMALLOC_CACHES for server use cases at least.

[1] https://lore.kernel.org/all/CALGbS4U6fox7SwmdHfDuawmO...@mail.gmail.com

Otherwise the patch is really straightforward and looks good to me.

Thanks!

--
Cheers,
Harry / Hyeonggon

[Harry Yoo (Oracle)]

Now somewhat out-of-scope (or at least pre-existing) review comments
from Sashiko that I think are still worth mentioning...

> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h

> @@ -662,9 +662,20 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];

> -static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
> +static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
> {
> /*
> * The most common case is KMALLOC_NORMAL, so test for it
> @@ -672,9 +683,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
> */
> if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))

Sashiko pointed out KMALLOC_CGROUP caches are not partitioned [1]:
| Do allocations with the __GFP_ACCOUNT flag completely bypass typed
| and random partitioning? KMALLOC_NOT_NORMAL_BITS includes __GFP_ACCOUNT.

Right.

| If this bit is set, the code bypasses the partitioning logic and routes
| the allocation to the KMALLOC_CGROUP cache.

Right.

| Since user-controllable objects
| like msg_msg, file descriptors, and pipes are allocated with __GFP_ACCOUNT,

Right.

| they will all be clustered in the exact same unpartitioned cache.

Right.

From security perspective do you think it'd be worthwhile to partition
KMALLOC_CGROUP caches? (I see at least few hundreds of users, unlike
KMALLOC_RECLAIM where there are only few users).

Another valid concern from Sashiko [1]:
| Does this leave reallocation functions like krealloc() and kvrealloc()
| without allocation token propagation?
|
| When an object is reallocated and requires memory expansion, the underlying
| generic SLUB code allocates a new buffer. Because the token macro is not
| applied to these realloc paths, __builtin_infer_alloc_token() evaluates
| locally on a generic size_t variable rather than the original type.

I think this is a valid point and worth addressing.

| This causes it to return the fallback token (0), which silently migrates the
| object from its isolated typed cache to the shared fallback cache
| (kmalloc-part-00) when resized.

[1] https://sashiko.dev/#/patchset/20260331111240.153913-1-elver%40google.com

[Vlastimil Babka (SUSE)]

On 4/3/26 08:27, Harry Yoo (Oracle) wrote:
>> diff --git a/include/linux/slab.h b/include/linux/slab.h
>> index 15a60b501b95..c0bf00ee6025 100644
>> --- a/include/linux/slab.h
>> +++ b/include/linux/slab.h
>> @@ -864,10 +877,10 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
>> * with the exception of kunit tests
>> */
>>
>> -void *__kmalloc_noprof(size_t size, gfp_t flags)
>> +void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
>> __assume_kmalloc_alignment __alloc_size(1);
>>
>> -void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
>> +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
>> __assume_kmalloc_alignment __alloc_size(1);
>
> So the @token parameter is unused when CONFIG_PARTITION_KMALLOC_CACHES is
> disabled but still increases the kernel size by a few kilobytes...
> but yeah I'm not sure if we can get avoid it without hurting readability.
>
> Just saying. (does anybody care?)

Well we did care enough with CONFIG_SLAB_BUCKETS to hide the unused param
using DECL_BUCKET_PARAMS(), so maybe extend that idea?
I think it's not just kernel size, but increased register pressure etc.

[Harry Yoo (Oracle)]

Hmm yeah.

I wasn't sure if we could do this without hurting readability,
but perhaps we could...

> so maybe extend that idea?
> I think it's not just kernel size, but increased register pressure etc.

Something like this should work? (diff on top of this patch)

--
Cheers,
Harry / Hyeonggon

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c0bf00ee6025..0496d2e63f5e 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -871,16 +871,32 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
#define PASS_BUCKET_PARAM(_b) NULL
#endif

+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
+#define _PASS_TOKEN_PARAM(_token) , (_token)
+#define PASS_TOKEN_PARAM(_token) (_token)
+#else
+#define DECL_TOKEN_PARAM(_token)
+#define _PASS_TOKEN_PARAM(_token)
+#define PASS_TOKEN_PARAM(_token) ((kmalloc_token_t){})
+#endif
+
+#define DECL_KMALLOC_PARAMS(_size, _b, _token) DECL_BUCKET_PARAMS(_size, _b) \
+ DECL_TOKEN_PARAM(_token)
+
+#define PASS_KMALLOC_PARAMS(_size, _b, _token) PASS_BUCKET_PARAMS(_size, _b) \
+ _PASS_TOKEN_PARAM(_token)
+
/*
* The following functions are not to be used directly and are intended only
* for internal use from kmalloc() and kmalloc_node()

* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

@@ -964,7 +980,7 @@ static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t

kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}

- return __kmalloc_noprof(size, flags, token);
+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);

}
#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

@@ -1075,10 +1091,10 @@ void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_tok

__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \

- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, __kmalloc_token(_size)))
+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_, __kmalloc_token(_size)))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
@@ -1093,7 +1109,7 @@ static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, g

kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}

- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, token);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);

}
#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

@@ -1154,10 +1170,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi
*/

#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,
- unsigned long caller, kmalloc_token_t token) __alloc_size(1);
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node,
+ unsigned long caller) __alloc_size(1);

#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \

- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller, __kmalloc_token(size))
+ __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(size, NULL, __kmalloc_token(size)), flags, node, caller)
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1183,7 +1199,7 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size

return NULL;
if (__builtin_constant_p(n) && __builtin_constant_p(size))

return _kmalloc_node_noprof(bytes, flags, node, token);

- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node, token);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), flags, node);

}
#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

@@ -1209,10 +1225,10 @@ static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, km

#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node, kmalloc_token_t token) __alloc_size(1);
+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
+ gfp_t flags, int node) __alloc_size(1);

#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \

- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node, __kmalloc_token(_size))
+ __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)

#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)

@@ -1225,7 +1241,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \

- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE, __kmalloc_token(_size)))
+ alloc_hooks(__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), 1, _flags, NUMA_NO_NODE))

static inline __alloc_size(1, 2) void *

_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
@@ -1235,7 +1251,7 @@ _kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmallo

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), 1, flags, node, token);
+ return __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), 1, flags, node);

}
#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/mm/slub.c b/mm/slub.c
index 5630dde94df0..84f129d79c99 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5293,15 +5293,17 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, kmalloc_token_t token)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_, token);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,

+ _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
{
- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_, token);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,

+ NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5394,10 +5396,11 @@ void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node, kmalloc_tok
}
EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
- int node, unsigned long caller, kmalloc_token_t token)
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,
+ int node, unsigned long caller)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller, token);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,

+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);
@@ -6812,8 +6815,8 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
*

* Return: pointer to the allocated memory of %NULL in case of failure
*/

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,
- gfp_t flags, int node, kmalloc_token_t token)
+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token),
+ unsigned long align, gfp_t flags, int node)
{
bool allow_block;
void *ret;
@@ -6824,7 +6827,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),

- node, _RET_IP_, token);
+ node, _RET_IP_, PASS_TOKEN_PARAM(token));

if (ret || size <= PAGE_SIZE)
return ret;

--
2.43.0

[Marco Elver]

I'll take a closer look at generated code. In some cases the compiler
ought to omit zero-sized arguments, so I want to be sure we're not
prematurely optimizing and the size increase is not some other effect.

> Something like this should work? (diff on top of this patch)

Thanks, I'll consider it.

Re your other comments:

> Assuming not all people building the kernel are security experts...
> (including myself) could you please add some insights/guidance on how to
> decide between RANDOM_KMALLOC_CACHES and TYPED_KMALLOC_CACHES?

You can find different arguments for either, and in the original RFC
that was part of the discussion. However, my biased view is that
type-based partitioning in general is the stronger security boundary.
Because it creates a deterministic separation; specifically isolating
pointer-containing objects from pointerless ones: it effectively kills
certain classes of exploit techniques that probabilistic defenses
(like randomization) only delay, especially in environments where
attackers can retry or use side-channels.

The current pointer/non-pointer scheme is relatively intuitive with
well-understood properties, and a good start. However, an open
research question is if better alloc-token ID schemes exist: one that
is tailored to kernel data structures that would meaningfully increase
exploitation difficulty further without increasing the number of
partitions. Since an improved scheme could simply be activated with a
compiler flag, having the baseline infrastructure available and
maintained is the first step.

> Now somewhat out-of-scope (or at least pre-existing) review comments
> from Sashiko that I think are still worth mentioning...

Indeed, these are pre-existing issues with RANDOM_KMALLOC_CACHES.
Worth follow-up patches, but this patch here wants to just get the
TYPED_KMALLOC_CACHES infrastructure in place so we can build on top of
it.

Thanks,
-- Marco

[Harry Yoo (Oracle)]

Oh, didn't know that was a thing.

Not sure if it's safe to do that for exported functions though (since
modules can be built w/ a different compiler).

> so I want to be sure we're not
> prematurely optimizing and the size increase is not some other effect.

Great, thanks for looking into it.

> > Something like this should work? (diff on top of this patch)
>
> Thanks, I'll consider it.

Thanks.

> Re your other comments:
>
> > Assuming not all people building the kernel are security experts...
> > (including myself) could you please add some insights/guidance on how to
> > decide between RANDOM_KMALLOC_CACHES and TYPED_KMALLOC_CACHES?
>
> You can find different arguments for either, and in the original RFC
> that was part of the discussion.

Yeah, I had fun time following the discussions :)

> However, my biased view is that type-based partitioning in general
> is the stronger security boundary.
> Because it creates a deterministic separation; specifically isolating
> pointer-containing objects from pointerless ones: it effectively kills
> certain classes of exploit techniques that probabilistic defenses
> (like randomization) only delay, especially in environments where
> attackers can retry or use side-channels.

That's a fair argument.

Could we possibly put some of those arguments (in a neutral/technical
sense) in help text to make people's lives easier?

Err, trying to provide unbiased (tm) help text...
Something like:

config RANDOM_KMALLOC_CACHES

bool "Randomize slab caches for normal kmalloc"

help
Randomly pick a slab cache based on code address and a per-boot
random seed.

This makes it harder for attackers to predict object co-location.
The placement is random: while attackers don't know which kmalloc
cache an object will be allocated from, they might circumvent
the randomization by retrying attacks across multiple machines until
the target objects are co-located.

config TYPED_KMALLOC_CACHES

bool "Type based slab cache selection for normal kmalloc"

depends on $(cc-option,-falloc-token-max=123)
help

Rely on Clang's allocation tokens to choose a slab cache, where token

IDs are derived from the allocated type.

Unlike RANDOM_KMALLOC_CACHES, the cache assignment is deterministic
and based on type, which guarantees that objects of certain types
are not placed in the same cache. This effectively mitigates
certain classes of exploits that probabilistic defenses like
RANDOM_KMALLOC_CACHES only make harder but not impossible.
However, this also means the cache assignment is predictable.

For example, a token ID scheme might separate pointer-containing
objects and pointerless objects to prevent buffer overflows on
primitive buffers from directly corrupting pointer-containing objects.

The current effectiveness of Clang's type inference can be judged by

-Rpass=alloc-token, which provides diagnostics where (after dead-code

elimination) type inference failed.

Requires Clang 22 or later.

endchoice

> The current pointer/non-pointer scheme is relatively intuitive with
> well-understood properties, and a good start.
>
> However, an open research question is if better alloc-token ID schemes exist:
> one that is tailored to kernel data structures that would meaningfully
> increase exploitation difficulty further without increasing the number of
> partitions.

Haha, that's too adventurous research question for me :P

> Since an improved scheme could simply be activated with a
> compiler flag, having the baseline infrastructure available and
> maintained is the first step.

Agreed.

> > Now somewhat out-of-scope (or at least pre-existing) review comments
> > from Sashiko that I think are still worth mentioning...
>
> Indeed, these are pre-existing issues with RANDOM_KMALLOC_CACHES.
> Worth follow-up patches, but this patch here wants to just get the
> TYPED_KMALLOC_CACHES infrastructure in place so we can build on top of
> it.

Yeah, that's totally fine!

[Marco Elver]

On Tue, 7 Apr 2026 at 14:54, 'Harry Yoo (Oracle)' via kasan-dev

So I checked with Clang and GCC. Looks like Clang does omit the
zero-sized struct argument, i.e. generated code is identical to
before. Whereas GCC wastes a few bytes of stack space at callsites.

Which is sad, because that means we need the macro workaround.

Do you want to be credited with Co-authored-by - in which case I need
your Signed-off-by.

> Not sure if it's safe to do that for exported functions though (since
> modules can be built w/ a different compiler).

Kernel modules built with a different config (implicit if different
compiler) are not supported, and never have been. If it works, it's
just luck (I know people do this, but it's just a disaster waiting to
happen).

Thanks,
-- Marco

[Harry Yoo (Oracle)]

Thanks for confirming that.

> Which is sad, because that means we need the macro workaround.
>
> Do you want to be credited with Co-authored-by

I'd appreciate that. (I guess you meant Co-developed-by)

> - in which case I need your Signed-off-by.

Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>

> > Not sure if it's safe to do that for exported functions though (since
> > modules can be built w/ a different compiler).
>
> Kernel modules built with a different config (implicit if different
> compiler) are not supported, and never have been. If it works, it's
> just luck (I know people do this, but it's just a disaster waiting to
> happen).

And if GCC folks somehow fix this at some point, even kernel modules
built with a different version of GCC might not be supported?

[Marco Elver]

On Fri, 10 Apr 2026 at 04:10, Harry Yoo (Oracle) <ha...@kernel.org> wrote:
[...]

> > Which is sad, because that means we need the macro workaround.
> >
> > Do you want to be credited with Co-authored-by
>
> I'd appreciate that. (I guess you meant Co-developed-by)

Thanks (and yes).

> > - in which case I need your Signed-off-by.
>
> Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>
>
> > > Not sure if it's safe to do that for exported functions though (since
> > > modules can be built w/ a different compiler).
> >
> > Kernel modules built with a different config (implicit if different
> > compiler) are not supported, and never have been. If it works, it's
> > just luck (I know people do this, but it's just a disaster waiting to
> > happen).
>
> And if GCC folks somehow fix this at some point, even kernel modules
> built with a different version of GCC might not be supported?

Yes, and different compiler version also implies different compiler
(so even GCC A.B != GCC X.Y), because newer compiler versions may
activate different codegen options (some of which we auto-enable if
available).

[GONG Ruiqi]

On 3/31/2026 7:12 PM, Marco Elver wrote:
> ...

> @@ -662,9 +662,20 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];
> (IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
> (IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))
>
> +#ifdef CONFIG_RANDOM_KMALLOC_CACHES
> extern unsigned long random_kmalloc_seed;
> +typedef struct { unsigned long ip; } kmalloc_token_t;
> +#define __kmalloc_token(...) ((kmalloc_token_t) { .ip = _RET_IP_ })
> +#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
> +typedef struct { unsigned long v; } kmalloc_token_t;
> +#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

One tiny suggestion: we could use the same name for kmalloc_token_t's
member in both cases, which would make the code a bit more concise.

Acked-by: GONG Ruiqi <gongr...@huawei.com>

> +#else
> +/* no-op */
> +typedef struct {} kmalloc_token_t;
> +#define __kmalloc_token(...) ((kmalloc_token_t){})
> +#endif
>
> -static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
> +static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
> {
> /*
> * The most common case is KMALLOC_NORMAL, so test for it
> @@ -672,9 +683,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne
> */
> if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
> #ifdef CONFIG_RANDOM_KMALLOC_CACHES
> - /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
> - return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
> - ilog2(RANDOM_KMALLOC_CACHES_NR + 1));
> + /* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
> + return KMALLOC_PARTITION_START + hash_64(token.ip ^ random_kmalloc_seed,
> + ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
> +#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
> + return KMALLOC_PARTITION_START + token.v;
> #else
> return KMALLOC_NORMAL;
> #endif

> ...

[Marco Elver]

also does not prevent cross-cache attacks: while waiting for future
features like SLAB_VIRTUAL [3] to provide physical page isolation, this
feature should be deployed alongside SHUFFLE_PAGE_ALLOCATOR and
init_on_free=1 to mitigate cross-cache attacks and page-reuse attacks as
much as possible today.

With all that, my kernel (x86 defconfig) shows me a histogram of slab
cache object distribution per /proc/slabinfo (after boot):

<slab cache> <objs> <hist>

kmalloc-part-15 1465 ++++++++++++++
kmalloc-part-14 2988 +++++++++++++++++++++++++++++
kmalloc-part-13 1656 ++++++++++++++++
kmalloc-part-12 1045 ++++++++++
kmalloc-part-11 1697 ++++++++++++++++
kmalloc-part-10 1489 ++++++++++++++
kmalloc-part-09 965 +++++++++

kmalloc-part-08 710 +++++++
kmalloc-part-07 100 +
kmalloc-part-06 217 ++
kmalloc-part-05 105 +
kmalloc-part-04 4047 ++++++++++++++++++++++++++++++++++++++++

kmalloc-part-03 183 +

kmalloc-part-02 283 ++
kmalloc-part-01 316 +++

kmalloc 1422 ++++++++++++++

The above /proc/slabinfo snapshot shows me there are 6673 allocated

objects (slabs 00 - 07) that the compiler claims contain no pointers or

it was unable to infer the type of, and 12015 objects that contain

pointers (slabs 08 - 15). On a whole, this looks relatively sane.

Additionally, when I compile my kernel with -Rpass=alloc-token, which

provides diagnostics where (after dead-code elimination) type inference

failed, I see 186 allocation sites where the compiler failed to identify

a type (down from 966 when I sent the RFC [4]). Some initial review
confirms these are mostly variable sized buffers, but also include
structs with trailing flexible length arrays.

Link: https://clang.llvm.org/docs/AllocToken.html [1]
Link: https://blog.dfsec.com/ios/2025/05/30/blasting-past-ios-18/ [2]
Link: https://lwn.net/Articles/944647/ [3]
Link: https://lore.kernel.org/all/20250825154505....@google.com/ [4]
Link: https://discourse.llvm.org/t/rfc-a-framework-for-allocator-partitioning-hints/87434

Acked-by: GONG Ruiqi <gongr...@huawei.com>
Co-developed-by: Harry Yoo (Oracle) <ha...@kernel.org>

Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>

Signed-off-by: Marco Elver <el...@google.com>
---

v2:
* Avoid empty function argument if !PARTITION_KMALLOC_CACHES
(co-developed-by Harry). While Clang does optimize out the empty
struct argument (and generated code is identical to before if
PARTITION_KMALLOC_CACHES is disabled), GCC doesn't do so. So we need
to fully remove the argument if not actually required.
* Cover krealloc() which was missed before, resulting in ~100 additional
objects in the pointer-containing caches in above histogram.
* Unify kmalloc_token_t definition.
* Expand Kconfig help text.

v1: https://lore.kernel.org/all/20260331111240...@google.com/

* Rebase and switch to builtin name that was released in Clang 22.
* Keep RANDOM_KMALLOC_CACHES the default.

RFC: https://lore.kernel.org/all/20250825154505....@google.com/
---
Makefile | 5 ++
include/linux/percpu.h | 2 +-

include/linux/slab.h | 127 ++++++++++++++++++++------------
kernel/configs/hardening.config | 2 +-
mm/Kconfig | 65 +++++++++++++---

mm/kfence/kfence_test.c | 4 +-
mm/slab.h | 4 +-

mm/slab_common.c | 48 ++++++------
mm/slub.c | 54 +++++++-------
9 files changed, 200 insertions(+), 111 deletions(-)

diff --git a/Makefile b/Makefile
index 54e1ae602000..f70170ed1522 100644
--- a/Makefile
+++ b/Makefile
@@ -989,6 +989,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)

endif
endif

+ifdef CONFIG_TYPED_KMALLOC_CACHES
+# PARTITION_KMALLOC_CACHES_NR + 1
+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 85bf8dd9f087..271b41be314d 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..3bd0db0ec95f 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -499,14 +499,33 @@ int kmem_cache_shrink(struct kmem_cache *s);
.usersize = sizeof_field(struct __struct, __field), \
}, (__flags))

+#ifdef CONFIG_PARTITION_KMALLOC_CACHES

+typedef struct { unsigned long v; } kmalloc_token_t;

+#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+extern unsigned long random_kmalloc_seed;
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)

+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

+#endif

+#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
+#define _PASS_TOKEN_PARAM(_token) , (_token)
+#define PASS_TOKEN_PARAM(_token) (_token)

+#else /* !CONFIG_PARTITION_KMALLOC_CACHES */
+typedef struct {} kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){}) /* no-op */

+#define DECL_TOKEN_PARAM(_token)
+#define _PASS_TOKEN_PARAM(_token)
+#define PASS_TOKEN_PARAM(_token) ((kmalloc_token_t){})

+#endif /* CONFIG_PARTITION_KMALLOC_CACHES */
+
/*
* Common kmalloc functions provided by all allocators
*/
void * __must_check krealloc_node_align_noprof(const void *objp, size_t new_size,
unsigned long align,
- gfp_t flags, int nid) __realloc_size(2);
-#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE)
-#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__))
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token)) __realloc_size(2);
+#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE _PASS_TOKEN_PARAM(__kmalloc_token(_o, _s, _f)))
+#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__ _PASS_TOKEN_PARAM(__kmalloc_token(__VA_ARGS__))))
#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -612,10 +631,10 @@ static inline unsigned int arch_slab_minalign(void)

#define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \
(KMALLOC_MIN_SIZE) : 16)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies
+#ifdef CONFIG_PARTITION_KMALLOC_CACHES
+#define PARTITION_KMALLOC_CACHES_NR 15 // # of cache copies
#else
-#define RANDOM_KMALLOC_CACHES_NR 0
+#define PARTITION_KMALLOC_CACHES_NR 0
#endif

/*

@@ -634,8 +653,8 @@ enum kmalloc_cache_type {

#ifndef CONFIG_MEMCG
KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
- KMALLOC_RANDOM_START = KMALLOC_NORMAL,
- KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
+ KMALLOC_PARTITION_START = KMALLOC_NORMAL,
+ KMALLOC_PARTITION_END = KMALLOC_PARTITION_START + PARTITION_KMALLOC_CACHES_NR,
#ifdef CONFIG_SLUB_TINY
KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else

@@ -662,9 +681,7 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];

(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))

-extern unsigned long random_kmalloc_seed;
-

-static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
+static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
{
/*
* The most common case is KMALLOC_NORMAL, so test for it

@@ -672,9 +689,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigne

*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
- return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
- ilog2(RANDOM_KMALLOC_CACHES_NR + 1));
+ /* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */

+ return KMALLOC_PARTITION_START + hash_64(token.v ^ random_kmalloc_seed,

+ ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
+#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+ return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;
#endif

@@ -858,16 +877,22 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
#define PASS_BUCKET_PARAM(_b) NULL
#endif

+#define DECL_KMALLOC_PARAMS(_size, _b, _token) DECL_BUCKET_PARAMS(_size, _b) \
+ DECL_TOKEN_PARAM(_token)
+
+#define PASS_KMALLOC_PARAMS(_size, _b, _token) PASS_BUCKET_PARAMS(_size, _b) \
+ _PASS_TOKEN_PARAM(_token)
+
/*
* The following functions are not to be used directly and are intended only
* for internal use from kmalloc() and kmalloc_node()

* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags)

+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)

__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)

+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

@@ -938,7 +963,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -948,14 +973,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],

+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}
- return __kmalloc_noprof(size, flags);

+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
}

+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node DECL_TOKEN_PARAM(token));
+#define kmalloc_nolock_noprof(...) _kmalloc_nolock_noprof(__VA_ARGS__ _PASS_TOKEN_PARAM(__kmalloc_token(__VA_ARGS__)))
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1087,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \

- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))

+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \

- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))

+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -1075,11 +1102,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],

+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);

+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);
}

+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**
@@ -1088,14 +1116,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)

+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kmalloc_noprof(bytes, flags);
+ return _kmalloc_noprof(bytes, flags, token);
}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**

@@ -1115,18 +1144,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz
* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/
-static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p,
+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
size_t new_n,
size_t new_size,
- gfp_t flags)
+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;

- return krealloc_noprof(p, bytes, flags);
+ return krealloc_node_align_noprof(p, bytes, 1, flags, NUMA_NO_NODE _PASS_TOKEN_PARAM(token));
}
+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))

/**
@@ -1137,10 +1167,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi

*/
#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,

+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node,

unsigned long caller) __alloc_size(1);
#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \

- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller)

+ __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(size, NULL, __kmalloc_token(size)), flags, node, caller)
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1157,17 +1187,18 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag

#define kmalloc_track_caller_noprof(...) \
kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_)

-static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
+ int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

if (__builtin_constant_p(n) && __builtin_constant_p(size))
- return kmalloc_node_noprof(bytes, flags, node);

- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node);
+ return _kmalloc_node_noprof(bytes, flags, node, token);

+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), flags, node);
}

+#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

#define kcalloc_node(_n, _size, _flags, _node) \

@@ -1183,39 +1214,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_

* @size: how many bytes of memory are required.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags)
+static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
- return kmalloc_noprof(size, flags | __GFP_ZERO);
+ return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}
+#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

gfp_t flags, int node) __alloc_size(1);
#define kvmalloc_node_align_noprof(_size, _align, _flags, _node) \

- __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, NULL), _align, _flags, _node)

+ __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))
#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)

+#define kvmalloc_node_noprof(size, flags, node) \
+ kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)
+#define kvmalloc_noprof(_size, _flags) kvmalloc_node_noprof(_size, _flags, NUMA_NO_NODE)
#define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO)

#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

#define kmem_buckets_valloc(_b, _size, _flags) \

- alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), 1, _flags, NUMA_NO_NODE))

+ alloc_hooks(__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), 1, _flags, NUMA_NO_NODE))

static inline __alloc_size(1, 2) void *

-kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)

+_kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;

- return kvmalloc_node_align_noprof(bytes, 1, flags, node);

+ return __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), 1, flags, node);
}

-
+#define kvmalloc_array_node_noprof(...) _kvmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kvmalloc_array_noprof(...) kvmalloc_array_node_noprof(__VA_ARGS__, NUMA_NO_NODE)

#define kvcalloc_node_noprof(_n,_s,_f,_node) kvmalloc_array_node_noprof(_n,_s,(_f)|__GFP_ZERO,_node)
#define kvcalloc_noprof(...) kvcalloc_node_noprof(__VA_ARGS__, NUMA_NO_NODE)

@@ -1225,9 +1260,9 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
#define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__))

void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
- gfp_t flags, int nid) __realloc_size(2);
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token)) __realloc_size(2);
#define kvrealloc_node_align(...) \
- alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__))
+ alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__ _PASS_TOKEN_PARAM(__kmalloc_token(__VA_ARGS__))))
#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config
index 7c3924614e01..2963b6bd890f 100644
--- a/kernel/configs/hardening.config
+++ b/kernel/configs/hardening.config
@@ -22,7 +22,7 @@ CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
CONFIG_SLAB_BUCKETS=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-CONFIG_RANDOM_KMALLOC_CACHES=y
+CONFIG_PARTITION_KMALLOC_CACHES=y

# Sanity check userspace page table mappings.
CONFIG_PAGE_TABLE_CHECK=y

diff --git a/mm/Kconfig b/mm/Kconfig
index ebd8ea353687..47f323816de7 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -247,22 +247,67 @@ config SLUB_STATS

out which slabs are relevant to a particular load.
Try running: slabinfo -DA

-config RANDOM_KMALLOC_CACHES
- default n
+config PARTITION_KMALLOC_CACHES
depends on !SLUB_TINY

- bool "Randomize slab caches for normal kmalloc"
+ bool "Partitioned slab caches for normal kmalloc"
help

+ bool "Randomize slab caches for normal kmalloc"
+ help
+ Randomly pick a slab cache based on code address and a per-boot
+ random seed.
+
+ This makes it harder for attackers to predict object co-location.
+ The placement is random: while attackers don't know which kmalloc
+ cache an object will be allocated from, they might circumvent
+ the randomization by retrying attacks across multiple machines until
+ the target objects are co-located.

+
+config TYPED_KMALLOC_CACHES
+ bool "Type based slab cache selection for normal kmalloc"
+ depends on $(cc-option,-falloc-token-max=123)
+ help

+ Rely on Clang's allocation tokens to choose a slab cache, where token
+ IDs are derived from the allocated type.
+
+ Unlike RANDOM_KMALLOC_CACHES, cache assignment is deterministic based
+ on type, which guarantees that objects of certain types are not
+ placed in the same cache. This effectively mitigates certain classes
+ of exploits that probabilistic defenses like RANDOM_KMALLOC_CACHES
+ only make harder but not impossible. However, this also means the
+ cache assignment is predictable.
+
+ Clang's default token ID calculation returns a bounded hash with
+ disjoint ranges for pointer-containing and pointerless objects: when
+ used as the slab cache index, this prevents buffer overflows on
+ primitive buffers from directly corrupting pointer-containing
+ objects.

+
+ The current effectiveness of Clang's type inference can be judged by
+ -Rpass=alloc-token, which provides diagnostics where (after dead-code
+ elimination) type inference failed.
+

diff --git a/mm/slub.c b/mm/slub.c
index 2b2d33cc735c..f033aae90bdc 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2125,11 +2125,11 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,

if (!is_kmalloc_normal(s))
return sz;

- obj_exts_cache = kmalloc_slab(sz, NULL, gfp, 0);
+ obj_exts_cache = kmalloc_slab(sz, NULL, gfp, __kmalloc_token(0));
/*

- * We can't simply compare s with obj_exts_cache, because random kmalloc
- * caches have multiple caches per size, selected by caller address.
- * Since caller address may differ between kmalloc_slab() and actual
+ * We can't simply compare s with obj_exts_cache, because partitioned kmalloc
+ * caches have multiple caches per size, selected by caller address or type.
+ * Since caller address or type may differ between kmalloc_slab() and actual
* allocation, bump size when sizes are equal.
*/
if (s->object_size == obj_exts_cache->object_size)

@@ -5239,7 +5239,7 @@ EXPORT_SYMBOL(__kmalloc_large_node_noprof);

static __always_inline

void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

- unsigned long caller)
+ unsigned long caller, kmalloc_token_t token)
{
struct kmem_cache *s;
void *ret;

@@ -5254,22 +5254,24 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

if (unlikely(!size))
return ZERO_SIZE_PTR;

- s = kmalloc_slab(size, b, flags, caller);
+ s = kmalloc_slab(size, b, flags, token);

ret = slab_alloc_node(s, NULL, flags, node, caller, size);
ret = kasan_kmalloc(s, ret, size, flags);

trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)

+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
{

- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags)

+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
{

- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,
+ NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5284,7 +5286,7 @@ EXPORT_SYMBOL(__kmalloc_noprof);

* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
* There is no reason to call it again and expect !NULL.
*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

+void *_kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node DECL_TOKEN_PARAM(token))

{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
struct kmem_cache *s;

@@ -5307,7 +5309,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);

+ s = kmalloc_slab(size, NULL, alloc_gfp, PASS_TOKEN_PARAM(token));

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*

@@ -5360,12 +5362,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,

+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,

int node, unsigned long caller)
{

- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);

+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);

@@ -6555,7 +6558,7 @@ void kfree_nolock(const void *object)
EXPORT_SYMBOL_GPL(kfree_nolock);

static __always_inline __realloc_size(2) void *
-__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid)
+__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid, kmalloc_token_t token)
{
void *ret;
size_t ks = 0;
@@ -6627,7 +6630,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
return (void *)p;

alloc_new:
- ret = kmalloc_node_track_caller_noprof(new_size, flags, nid, _RET_IP_);
+ ret = __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(new_size, NULL, token), flags, nid, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();
@@ -6677,7 +6680,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
* Return: pointer to the allocated memory or %NULL in case of error
*/
void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long align,
- gfp_t flags, int nid)
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token))
{
void *ret;

@@ -6686,7 +6689,7 @@ void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long a
return ZERO_SIZE_PTR;
}

- ret = __do_krealloc(p, new_size, align, flags, nid);
+ ret = __do_krealloc(p, new_size, align, flags, nid, PASS_TOKEN_PARAM(token));
if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret))
kfree(p);

@@ -6723,6 +6726,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
* failure, fall back to non-contiguous (vmalloc) allocation.
* @size: size of the request.
* @b: which set of kmalloc buckets to allocate from.

+ * @token: allocation token.
* @align: desired alignment.
* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
* @node: numa node to allocate from

@@ -6739,7 +6743,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

*
* Return: pointer to the allocated memory of %NULL in case of failure
*/
-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

gfp_t flags, int node)
{
bool allow_block;

@@ -6751,7 +6755,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),

- node, _RET_IP_);

+ node, _RET_IP_, PASS_TOKEN_PARAM(token));
if (ret || size <= PAGE_SIZE)
return ret;

@@ -6848,17 +6852,17 @@ EXPORT_SYMBOL(kvfree_sensitive);
* Return: pointer to the allocated memory or %NULL in case of error
*/
void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
- gfp_t flags, int nid)
+ gfp_t flags, int nid DECL_TOKEN_PARAM(token))
{
void *n;

if (is_vmalloc_addr(p))
return vrealloc_node_align_noprof(p, size, align, flags, nid);

- n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid);
+ n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid _PASS_TOKEN_PARAM(token));
if (!n) {
/* We failed to krealloc(), fall back to kvmalloc(). */
- n = kvmalloc_node_align_noprof(size, align, flags, nid);
+ n = __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), align, flags, nid);
if (!n)
return NULL;

@@ -8351,7 +8355,7 @@ static void __init bootstrap_kmalloc_sheaves(void)

{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--

2.54.0.rc1.513.gad8abe7a5a-goog

[Marco Elver]

Sashiko found 2 latent issues in the diff context :
https://sashiko.dev/#/patchset/20260415143735.2974230-1-elver%40google.com
The fix is here:
https://lore.kernel.org/all/20260416132837....@google.com/

The irony is that TYPED_KMALLOC_CACHES would have helped mitigate this
kind of overflow bug.

[Harry Yoo (Oracle)]

[CC'ing RUST ALLOC folks for rust bindings]

Thanks. I find the help text much more useful.

>
> v1: https://lore.kernel.org/all/20260331111240...@google.com/
> * Rebase and switch to builtin name that was released in Clang 22.
> * Keep RANDOM_KMALLOC_CACHES the default.
>
> RFC: https://lore.kernel.org/all/20250825154505....@google.com/
> ---

A few comments on V2:

# comment 1

I'm not a big fan of how k[v]realloc_node_align()
and kmalloc_nolock() define and pass the token parameter.

IMHO it'll be fine to use {DECL,PASS}_KMALLOC_PARAMS() in those
functions, since SLAB_BUCKETS users already passes NULL bucket
to most of __kmalloc*() calls anyway.

# comment 2

This breaks Documentation/.

Problems:

- The document generator doesn't handle DECL_KMALLOC_PARAMS() well.

- The signature of the function that users call (krealloc_node_align())
and the function that has kerneldoc (krealloc_node_align_noprof())
don't match.

- Even worse, moving kerneldoc to the macro doesn't work because
it uses variable arguments (...)

# comment 3

Looking at how rust generates helper functions,
in rust/helpers/slab.c:
| // SPDX-License-Identifier: GPL-2.0
|
| #include <linux/slab.h>
|
| __rust_helper void *__must_check __realloc_size(2)
| rust_helper_krealloc_node_align(const void *objp, size_t new_size, unsigned long align,
| gfp_t flags, int node)
| {
| return krealloc_node_align(objp, new_size, align, flags, node);
| }
|
| __rust_helper void *__must_check __realloc_size(2)
| rust_helper_kvrealloc_node_align(const void *p, size_t size, unsigned long align,
| gfp_t flags, int node)
| {
| return kvrealloc_node_align(p, size, align, flags, node);
| }

Rust code probably won't pass any meaningful token?
(something you may want to address in the future)

--
Cheers,
Harry / Hyeonggon

[Not trimming the rest of the patch for those added to Cc]

[Marco Elver]

I'm not sure I agree. 2 reasons:

1. Even though it's "just" k[v]realloc_node_align() and
kmalloc_nolock() - despite their relatively less frequent use - just
put one of them in a hot path and you're sacrificing performance even
further. There are only so many arguments that can be passed in
registers (depending on arch), and may cause more stack spilling.

2. We'd misleadingly declare that these functions do something with
the bucket arg. This is wrong.

Both feels wrong, and would only make this change if you confirm both
are trade-offs that you strongly prefer.

> # comment 2
>
> This breaks Documentation/.
>
> Problems:
>
> - The document generator doesn't handle DECL_KMALLOC_PARAMS() well.
>
> - The signature of the function that users call (krealloc_node_align())
> and the function that has kerneldoc (krealloc_node_align_noprof())
> don't match.
>
> - Even worse, moving kerneldoc to the macro doesn't work because
> it uses variable arguments (...)

Well, some were broken before, now it's just broken more. :-)

We could move the documentation to macros and switch to explicit args
instead of (...).

Otherwise, I don't see any way to fix this. Preferences?

> # comment 3
>
> Looking at how rust generates helper functions,
> in rust/helpers/slab.c:
> | // SPDX-License-Identifier: GPL-2.0
> |
> | #include <linux/slab.h>
> |
> | __rust_helper void *__must_check __realloc_size(2)
> | rust_helper_krealloc_node_align(const void *objp, size_t new_size, unsigned long align,
> | gfp_t flags, int node)
> | {
> | return krealloc_node_align(objp, new_size, align, flags, node);
> | }
> |
> | __rust_helper void *__must_check __realloc_size(2)
> | rust_helper_kvrealloc_node_align(const void *p, size_t size, unsigned long align,
> | gfp_t flags, int node)
> | {
> | return kvrealloc_node_align(p, size, align, flags, node);
> | }
>
> Rust code probably won't pass any meaningful token?
> (something you may want to address in the future)

Yes, it'll just pass '0' by default. We could force Rust's allocation
to be in the pointer-containing range - if we assume Rust code is less
prone to contain bugs, but the assumption is that such allocations
both originate and are confined to the Rust side. One easy way to do
this is to write:

return kvrealloc_node_align(p, size + 0 * sizeof(long*), align,
flags, node);

But I'd defer that for now, until we're sure the above assumption
holds (Rust originated + confined).

[Harry Yoo (Oracle)]

On Mon, Apr 20, 2026 at 11:30:23AM +0200, Marco Elver wrote:
> On Mon, 20 Apr 2026 at 09:25, Harry Yoo (Oracle) <ha...@kernel.org> wrote:
> > [CC'ing RUST ALLOC folks for rust bindings]
> > On Wed, Apr 15, 2026 at 04:37:05PM +0200, Marco Elver wrote:
> > A few comments on V2:
> >
> > # comment 1
> >
> > I'm not a big fan of how k[v]realloc_node_align()
> > and kmalloc_nolock() define and pass the token parameter.
> >
> > IMHO it'll be fine to use {DECL,PASS}_KMALLOC_PARAMS() in those
> > functions, since SLAB_BUCKETS users already passes NULL bucket
> > to most of __kmalloc*() calls anyway.
>
> I'm not sure I agree. 2 reasons:
>
> 1. Even though it's "just" k[v]realloc_node_align() and
> kmalloc_nolock() - despite their relatively less frequent use - just
> put one of them in a hot path and you're sacrificing performance even
> further. There are only so many arguments that can be passed in
> registers (depending on arch), and may cause more stack spilling.
>
> 2. We'd misleadingly declare that these functions do something with
> the bucket arg. This is wrong.

Both are valid points. But it still feels wrong to have:

void *krealloc_node_align_noprof(const void *objp, size_t new_size,
unsigned long align,
gfp_t flags, int nid DECL_TOKEN_PARAM(token));

n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid _PASS_TOKEN_PARAM(token));

Actually the problem here is that some of parameters in
DECL_KMALLOC_PARAMS() are not necessary in some functions.

Perhaps we could have

DECL_KMALLOC_PARAMS(size, b, token) # declare size, bucket, token

DECL_BUCKET_PARAMS(size, token) # declare size, bucket;
# but, actually, we don't need this!

DECL_TOKEN_PARAMS(size, b) # declare size, token only;
# for kmalloc_nolock and k[v]realloc_node_align()

and use DECL_TOKEN_PARAMS(), PASS_TOKEN_PARAMS() for those functions?
(just like how DECL_BUCKET_PARAMS() worked before)

What do you think?

> Both feels wrong, and would only make this change if you confirm both
> are trade-offs that you strongly prefer.
>
> > # comment 2
> >
> > This breaks Documentation/.
> >
> > Problems:
> >
> > - The document generator doesn't handle DECL_KMALLOC_PARAMS() well.
> >
> > - The signature of the function that users call (krealloc_node_align())
> > and the function that has kerneldoc (krealloc_node_align_noprof())
> > don't match.
> >
> > - Even worse, moving kerneldoc to the macro doesn't work because
> > it uses variable arguments (...)
>
> Well, some were broken before, now it's just broken more. :-)

Ouch... ;-)

> We could move the documentation to macros and switch to explicit args
> instead of (...).

That works for me!

Ack.

By the way, since Allocator trait uses realloc() to allocate new memory,
IIUC all k[v]malloc, k[v]realloc usage from Rust will be affected.

[Kees Cook]

On Wed, Apr 15, 2026 at 04:37:05PM +0200, Marco Elver wrote:

> The builtin __builtin_infer_alloc_token(<malloc-args>, ...) instructs
> the compiler to infer an allocation type from arguments commonly passed
> to memory-allocating functions and returns a type-derived token ID. The
> implementation passes kmalloc-args to the builtin: the compiler performs
> best-effort type inference, and then recognizes common patterns such as
> `kmalloc(sizeof(T), ...)`, `kmalloc(sizeof(T) * n, ...)`, but also
> `(T *)kmalloc(...)`. Where the compiler fails to infer a type the
> fallback token (default: 0) is chosen.
>
> Note: kmalloc_obj(..) APIs fix the pattern how size and result type are
> expressed, and therefore ensures there's not much drift in which
> patterns the compiler needs to recognize. Specifically, kmalloc_obj()
> and friends expand to `(TYPE *)KMALLOC(__obj_size, GFP)`, which the
> compiler recognizes via the cast to TYPE*.

Great! I'm glad this gets deterministically handled for the kmalloc_obj*
cases.

> Additionally, when I compile my kernel with -Rpass=alloc-token, which
> provides diagnostics where (after dead-code elimination) type inference
> failed, I see 186 allocation sites where the compiler failed to identify
> a type (down from 966 when I sent the RFC [4]). Some initial review
> confirms these are mostly variable sized buffers, but also include
> structs with trailing flexible length arrays.

For the call-site-partitioning series[1] I sent before, I had
per-site caches for fixed-sized allocations and size bucket caches for
variably-sized allocations. I'd like to see something similar for this
series. Specifically, I replaced "kmalloc_slab" with "choose_slab" that
did O(1) to find the dedicated cache/bucket for the allocation[2].

In this case, we now have a build-time-constant value that it should be
possible to use to look up a _single_ dedicated cache/bucket for the
given unique type: there is no need to do hashing.

> [...]

> -config RANDOM_KMALLOC_CACHES
> - default n
> +config PARTITION_KMALLOC_CACHES
> depends on !SLUB_TINY
> - bool "Randomize slab caches for normal kmalloc"
> + bool "Partitioned slab caches for normal kmalloc"
> help
> - A hardening feature that creates multiple copies of slab caches for
> - normal kmalloc allocation and makes kmalloc randomly pick one based
> - on code address, which makes the attackers more difficult to spray
> - vulnerable memory objects on the heap for the purpose of exploiting
> - memory vulnerabilities.
> + A hardening feature that creates multiple isolated copies of slab
> + caches for normal kmalloc allocations. This makes it more difficult
> + to exploit memory-safety vulnerabilities by attacking vulnerable
> + co-located memory objects. Several modes are provided.
>
> Currently the number of copies is set to 16, a reasonably large value

The "16" buckets seems to hold for TYPED_KMALLOC_CACHES too? My goal
with the earlier type-partitioning was to get _total_ isolation, not
simply bucketed: 1 cache (or sizes-bucket) for each type. The "16"
limitation from RANDOM_KMALLOC_CACHES was kind of arbitrary due to the
hashing.

> that effectively diverges the memory objects allocated for different
> subsystems or modules into different caches, at the expense of a
> - limited degree of memory and CPU overhead that relates to hardware and
> - system workload.
> + limited degree of memory and CPU overhead that relates to hardware
> + and system workload.
> +
> +choice
> + prompt "Partitioned slab cache mode"
> + depends on PARTITION_KMALLOC_CACHES
> + default RANDOM_KMALLOC_CACHES

I think this should be adjusted a bit:

config CC_HAS_ALLOC_TOKEN
def_bool $(cc-option,-falloc-token-max=123)

...
choice

prompt "Partitioned slab cache mode"

depends on PARTITION_KMALLOC_CACHES
default TYPED_KMALLOC_CACHES if CC_HAS_ALLOC_TOKEN
default RANDOM_KMALLOC_CACHES

And actually, perhaps a global rename of the options so the selection
naming is at the end of the CONFIG phrase, and bundle the on/off into
the choice:


choice

prompt "Partitioned slab cache mode"

depends on PARTITION_KMALLOC_CACHES
default KMALLOC_PARTITION_TYPED if !SLUB_TINY && CC_HAS_ALLOC_TOKEN
default KMALLOC_PARTITION_RANDOM if !SLUB_TINY
default KMALLOC_PARTITION_NONE

config KMALLOC_PARTITION_NONE
...
config KMALLOC_PARTITION_RANDOM
depends on !SLUB_TINY
...
config KMALLOC_PARTITION_TYPED
depends on !SLUB_TINY && CC_HAS_ALLOC_TOKEN

-Kees

[1] https://lore.kernel.org/lkml/20240809072532...@kernel.org/
[2] https://lore.kernel.org/lkml/20240809073309...@kernel.org/

--
Kees Cook

[Marco Elver]

That should be a separate series; I know what you're getting at, but
it's a significant rework and a different design with different
properties. This simpler patch is likely ready for the next merge
window (once I send v3), and in light of recent developments, I'd like
this to land sooner than later.

> > [...]
> > -config RANDOM_KMALLOC_CACHES
> > - default n
> > +config PARTITION_KMALLOC_CACHES
> > depends on !SLUB_TINY
> > - bool "Randomize slab caches for normal kmalloc"
> > + bool "Partitioned slab caches for normal kmalloc"
> > help
> > - A hardening feature that creates multiple copies of slab caches for
> > - normal kmalloc allocation and makes kmalloc randomly pick one based
> > - on code address, which makes the attackers more difficult to spray
> > - vulnerable memory objects on the heap for the purpose of exploiting
> > - memory vulnerabilities.
> > + A hardening feature that creates multiple isolated copies of slab
> > + caches for normal kmalloc allocations. This makes it more difficult
> > + to exploit memory-safety vulnerabilities by attacking vulnerable
> > + co-located memory objects. Several modes are provided.
> >
> > Currently the number of copies is set to 16, a reasonably large value
>
> The "16" buckets seems to hold for TYPED_KMALLOC_CACHES too? My goal
> with the earlier type-partitioning was to get _total_ isolation, not
> simply bucketed: 1 cache (or sizes-bucket) for each type. The "16"
> limitation from RANDOM_KMALLOC_CACHES was kind of arbitrary due to the
> hashing.

The token ID is also a hash, although it can be configured to be
unbounded to effectively give unique hash per type. As-is, limiting to
16 keeps it comparable to the RANDOM mode, albeit IMHO with better
isolation properties with the same overheads. As-is, performance
properties of RANDOM and TYPED are comparable, and the friction to
switch between them is minimal.

Unlike a completely new design, which will have comletely different
performance and memory usage properties - and wouldn't be comparable.

> > that effectively diverges the memory objects allocated for different
> > subsystems or modules into different caches, at the expense of a
> > - limited degree of memory and CPU overhead that relates to hardware and
> > - system workload.
> > + limited degree of memory and CPU overhead that relates to hardware
> > + and system workload.
> > +
> > +choice
> > + prompt "Partitioned slab cache mode"
> > + depends on PARTITION_KMALLOC_CACHES
> > + default RANDOM_KMALLOC_CACHES
>
> I think this should be adjusted a bit:
>
> config CC_HAS_ALLOC_TOKEN
> def_bool $(cc-option,-falloc-token-max=123)
>
> ...
> choice
> prompt "Partitioned slab cache mode"
> depends on PARTITION_KMALLOC_CACHES
> default TYPED_KMALLOC_CACHES if CC_HAS_ALLOC_TOKEN
> default RANDOM_KMALLOC_CACHES

Sure.

> And actually, perhaps a global rename of the options so the selection
> naming is at the end of the CONFIG phrase, and bundle the on/off into
> the choice:
>
>
> choice
> prompt "Partitioned slab cache mode"
> depends on PARTITION_KMALLOC_CACHES
> default KMALLOC_PARTITION_TYPED if !SLUB_TINY && CC_HAS_ALLOC_TOKEN
> default KMALLOC_PARTITION_RANDOM if !SLUB_TINY
> default KMALLOC_PARTITION_NONE
>
> config KMALLOC_PARTITION_NONE
> ...
> config KMALLOC_PARTITION_RANDOM
> depends on !SLUB_TINY
> ...
> config KMALLOC_PARTITION_TYPED
> depends on !SLUB_TINY && CC_HAS_ALLOC_TOKEN

There was a comment somewhere else that even introducing
PARTITION_KMALLOC_CACHES might confuse users of RANDOM_KMALLOC_CACHES.
I think completely getting rid of and renaming RANDOM_KMALLOC_CACHES
has marginal benefit, and will cause friction for existing users (even
moreso than already). I see little benefit here, and would prefer not
to break user configs more than needed: configs that already set
RANDOM_KMALLOC_CACHES, upon rebuild will be prompted to enable
PARTITION_KMALLOC_CACHES; if user says Y, then their previous
selection (RANDOM) would already be picked and they don't have to
rediscover that it exists under a new name.

I can make this change, but only if you're sure the benefit outweighs
the downsides here.

Thanks,
-- Marco

[Marco Elver]

On Tue, Apr 21, 2026 at 09:13PM +0200, Marco Elver wrote:
[...]

Upon further reflection, since the transition isn't smooth anyway, I'm
probably going to rename, but have them all use the PARTITION_KMALLOC_*
prefix so it's easy to just search for "CONFIG_PARTITION_KMALLOC_". I
don't see the need for a "NONE" variant - I've seen this pattern
elsewhere, and then you end up with users reading the .config and
concluding "CONFIG_PARTITION_KMALLOC_CACHES is enabled ... but oh never
mind actually it isn't" which I find confusing. This could be useful if
we had a dynamic on/off toggle and the default is NONE, but that's not
the case here.

diff --git a/Makefile b/Makefile
index f70170ed1522..d1f63ffb85f0 100644
--- a/Makefile
+++ b/Makefile
@@ -989,7 +989,7 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)
endif
endif

-ifdef CONFIG_TYPED_KMALLOC_CACHES
+ifdef CONFIG_PARTITION_KMALLOC_TYPED
# PARTITION_KMALLOC_CACHES_NR + 1
KBUILD_CFLAGS += -falloc-token-max=16
endif
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 0537c3596163..b46300037ca5 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -501,10 +501,10 @@ int kmem_cache_shrink(struct kmem_cache *s);

#ifdef CONFIG_PARTITION_KMALLOC_CACHES

typedef struct { unsigned long v; } kmalloc_token_t;

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM
extern unsigned long random_kmalloc_seed;

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })

-#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+#elif defined(CONFIG_PARTITION_KMALLOC_TYPED)

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })

#endif
#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
@@ -732,11 +732,11 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc
* with a single branch for all the relevant flags.

*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM

/* PARTITION_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */

return KMALLOC_PARTITION_START + hash_64(token.v ^ random_kmalloc_seed,

ilog2(PARTITION_KMALLOC_CACHES_NR + 1));
-#elif defined(CONFIG_TYPED_KMALLOC_CACHES)
+#elif defined(CONFIG_PARTITION_KMALLOC_TYPED)

return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;

diff --git a/mm/Kconfig b/mm/Kconfig
index 6d44bd2633bb..d8510913fbeb 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -265,12 +265,12 @@ config PARTITION_KMALLOC_CACHES

choice
prompt "Partitioned slab cache mode"
depends on PARTITION_KMALLOC_CACHES

- default TYPED_KMALLOC_CACHES if CC_HAS_ALLOC_TOKEN
- default RANDOM_KMALLOC_CACHES
+ default PARTITION_KMALLOC_TYPED if CC_HAS_ALLOC_TOKEN
+ default PARTITION_KMALLOC_RANDOM
help

Selects the slab cache partitioning mode.

-config RANDOM_KMALLOC_CACHES
+config PARTITION_KMALLOC_RANDOM

bool "Randomize slab caches for normal kmalloc"

help

Randomly pick a slab cache based on code address and a per-boot

@@ -282,17 +282,17 @@ config RANDOM_KMALLOC_CACHES

the randomization by retrying attacks across multiple machines until

the target objects are co-located.

-config TYPED_KMALLOC_CACHES
+config PARTITION_KMALLOC_TYPED

bool "Type based slab cache selection for normal kmalloc"

depends on CC_HAS_ALLOC_TOKEN
help

Rely on Clang's allocation tokens to choose a slab cache, where token

IDs are derived from the allocated type.

- Unlike RANDOM_KMALLOC_CACHES, cache assignment is deterministic based
+ Unlike PARTITION_KMALLOC_RANDOM, cache assignment is deterministic based

on type, which guarantees that objects of certain types are not

placed in the same cache. This effectively mitigates certain classes

- of exploits that probabilistic defenses like RANDOM_KMALLOC_CACHES
+ of exploits that probabilistic defenses like PARTITION_KMALLOC_RANDOM

only make harder but not impossible. However, this also means the

cache assignment is predictable.

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 21ab7dd79b5e..ca5e2a6d4e46 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -742,7 +742,7 @@ kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init =
{ /* initialization for https://llvm.org/pr42570 */ };
EXPORT_SYMBOL(kmalloc_caches);

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM
unsigned long random_kmalloc_seed __ro_after_init;
EXPORT_SYMBOL(random_kmalloc_seed);
#endif
@@ -1010,7 +1010,7 @@ void __init create_kmalloc_caches(void)
for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
new_kmalloc_cache(i, type);
}
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#ifdef CONFIG_PARTITION_KMALLOC_RANDOM
random_kmalloc_seed = get_random_u64();
#endif

[Kees Cook]

There is now the "transitional" Kconfig keyword too, which should make it easy to move to the new name.

For the naming, I tend to prefer the noun-verb (e.g. timer_start()) and noun-feature-option ordering (e.g. CONFIG_SLAB_FREELIST_RANDOM). This feature isn't about partition tables, so I don't think the first word should be "partition". :)

As far as doing the full isolation later, yeah, that's fine, and I think with this Kconfig change it's easy to add a new mode.

-Kees

--
Kees Cook

[Marco Elver]

Rework the general infrastructure around RANDOM_KMALLOC_CACHES into more

flexible KMALLOC_PARTITION_CACHES, with the former being a partitioning
mode of the latter.

Introduce a new mode, KMALLOC_PARTITION_TYPED, which leverages a feature

available in Clang 22 and later, called "allocation tokens" via

__builtin_infer_alloc_token() [1]. Unlike KMALLOC_PARTITION_RANDOM
(formerly RANDOM_KMALLOC_CACHES), this mode deterministically assigns a

slab cache to an allocation of type T, regardless of allocation site.

The builtin __builtin_infer_alloc_token(<malloc-args>, ...) instructs
the compiler to infer an allocation type from arguments commonly passed
to memory-allocating functions and returns a type-derived token ID. The
implementation passes kmalloc-args to the builtin: the compiler performs
best-effort type inference, and then recognizes common patterns such as
`kmalloc(sizeof(T), ...)`, `kmalloc(sizeof(T) * n, ...)`, but also
`(T *)kmalloc(...)`. Where the compiler fails to infer a type the
fallback token (default: 0) is chosen.

Note: kmalloc_obj(..) APIs fix the pattern how size and result type are
expressed, and therefore ensures there's not much drift in which
patterns the compiler needs to recognize. Specifically, kmalloc_obj()
and friends expand to `(TYPE *)KMALLOC(__obj_size, GFP)`, which the
compiler recognizes via the cast to TYPE*.

Additionally, when I compile my kernel with -Rpass=alloc-token, which
provides diagnostics where (after dead-code elimination) type inference
failed, I see 186 allocation sites where the compiler failed to identify
a type (down from 966 when I sent the RFC [4]). Some initial review
confirms these are mostly variable sized buffers, but also include
structs with trailing flexible length arrays.

Link: https://clang.llvm.org/docs/AllocToken.html [1]
Link: https://blog.dfsec.com/ios/2025/05/30/blasting-past-ios-18/ [2]
Link: https://lwn.net/Articles/944647/ [3]
Link: https://lore.kernel.org/all/20250825154505....@google.com/ [4]
Link: https://discourse.llvm.org/t/rfc-a-framework-for-allocator-partitioning-hints/87434
Acked-by: GONG Ruiqi <gongr...@huawei.com>
Co-developed-by: Harry Yoo (Oracle) <ha...@kernel.org>
Signed-off-by: Harry Yoo (Oracle) <ha...@kernel.org>
Signed-off-by: Marco Elver <el...@google.com>
---

v3:
* Introduce DECL_TOKEN_PARAMS / PASS_TOKEN_PARAMS (suggested-by Harry).
* Rename {RANDOM,TYPED}_KMALLOC_CACHES to KMALLOC_PARTITION_{RANDOM,TYPED},
so it's more easily searchable and consistent with the top-level name
with the variant at the end of the CONFIG phrase (reported-by Kees).
* Make KMALLOC_PARTITION_TYPED default if available (suggested-by Kees).
* Add RANDOM_KMALLOC_CACHES transitional option (suggested-by Kees).
* Add CC_HAS_ALLOC_TOKEN and use it (reported-by Kees).

v2: https://lore.kernel.org/all/20260415143735....@google.com/
* Avoid empty function argument if !KMALLOC_PARTITION_CACHES

(co-developed-by Harry). While Clang does optimize out the empty
struct argument (and generated code is identical to before if

KMALLOC_PARTITION_CACHES is disabled), GCC doesn't do so. So we need

to fully remove the argument if not actually required.
* Cover krealloc() which was missed before, resulting in ~100 additional
objects in the pointer-containing caches in above histogram.
* Unify kmalloc_token_t definition.
* Expand Kconfig help text.

v1: https://lore.kernel.org/all/20260331111240...@google.com/
* Rebase and switch to builtin name that was released in Clang 22.
* Keep RANDOM_KMALLOC_CACHES the default.

RFC: https://lore.kernel.org/all/20250825154505....@google.com/
---

Makefile | 5 ++
include/linux/percpu.h | 2 +-

include/linux/slab.h | 136 +++++++++++++++++++++-----------
init/Kconfig | 3 +
kernel/configs/hardening.config | 2 +-
mm/Kconfig | 73 ++++++++++++++---

mm/kfence/kfence_test.c | 4 +-
mm/slab.h | 4 +-

mm/slab_common.c | 52 ++++++------
mm/slub.c | 54 +++++++------
10 files changed, 220 insertions(+), 115 deletions(-)

diff --git a/Makefile b/Makefile
index 54e1ae602000..afe663bdb9c0 100644
--- a/Makefile
+++ b/Makefile
@@ -989,6 +989,11 @@ KBUILD_CFLAGS += $(CC_AUTO_VAR_INIT_ZERO_ENABLER)
endif
endif

+ifdef CONFIG_KMALLOC_PARTITION_TYPED
+# KMALLOC_PARTITION_CACHES_NR + 1

+KBUILD_CFLAGS += -falloc-token-max=16
+endif
+
ifdef CONFIG_CC_IS_CLANG
ifdef CONFIG_CC_HAS_COUNTED_BY_PTR
KBUILD_CFLAGS += -fexperimental-late-parse-attributes
diff --git a/include/linux/percpu.h b/include/linux/percpu.h

index 85bf8dd9f087..bdb721dac0e3 100644

--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -36,7 +36,7 @@
#define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \
PCPU_MIN_ALLOC_SHIFT)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES

# if defined(CONFIG_LOCKDEP) && !defined(CONFIG_PAGE_SIZE_4KB)
# define PERCPU_DYNAMIC_SIZE_SHIFT 13
# else

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 15a60b501b95..c232f8a10af6 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -499,14 +499,38 @@ int kmem_cache_shrink(struct kmem_cache *s);

.usersize = sizeof_field(struct __struct, __field), \
}, (__flags))

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES
+typedef struct { unsigned long v; } kmalloc_token_t;
+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
+extern unsigned long random_kmalloc_seed;
+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })
+#elif defined(CONFIG_KMALLOC_PARTITION_TYPED)

+#define __kmalloc_token(...) ((kmalloc_token_t){ .v = __builtin_infer_alloc_token(__VA_ARGS__) })
+#endif
+#define DECL_TOKEN_PARAM(_token) , kmalloc_token_t (_token)
+#define _PASS_TOKEN_PARAM(_token) , (_token)
+#define PASS_TOKEN_PARAM(_token) (_token)

+#define DECL_TOKEN_PARAMS(_size, _token) size_t (_size), kmalloc_token_t (_token)
+#define PASS_TOKEN_PARAMS(_size, _token) (_size), (_token)
+#else /* !CONFIG_KMALLOC_PARTITION_CACHES */

+typedef struct {} kmalloc_token_t;
+#define __kmalloc_token(...) ((kmalloc_token_t){}) /* no-op */
+#define DECL_TOKEN_PARAM(_token)
+#define _PASS_TOKEN_PARAM(_token)
+#define PASS_TOKEN_PARAM(_token) ((kmalloc_token_t){})

+#define DECL_TOKEN_PARAMS(_size, _token) size_t (_size)
+#define PASS_TOKEN_PARAMS(_size, _token) (_size)
+#endif /* CONFIG_KMALLOC_PARTITION_CACHES */

+
/*
* Common kmalloc functions provided by all allocators
*/

-void * __must_check krealloc_node_align_noprof(const void *objp, size_t new_size,
+void * __must_check krealloc_node_align_noprof(const void *objp,
+ DECL_TOKEN_PARAMS(new_size, token),
unsigned long align,

gfp_t flags, int nid) __realloc_size(2);
-#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, _s, 1, _f, NUMA_NO_NODE)
-#define krealloc_node_align(...) alloc_hooks(krealloc_node_align_noprof(__VA_ARGS__))

+#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)
+#define krealloc_node_align(_o, _s, _a, _f, _n) alloc_hooks(krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -612,10 +636,10 @@ static inline unsigned int arch_slab_minalign(void)

#define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \
(KMALLOC_MIN_SIZE) : 16)

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
-#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES
+#define KMALLOC_PARTITION_CACHES_NR 15 // # of cache copies
#else
-#define RANDOM_KMALLOC_CACHES_NR 0
+#define KMALLOC_PARTITION_CACHES_NR 0
#endif

/*
@@ -634,8 +658,8 @@ enum kmalloc_cache_type {

#ifndef CONFIG_MEMCG
KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
- KMALLOC_RANDOM_START = KMALLOC_NORMAL,
- KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
+ KMALLOC_PARTITION_START = KMALLOC_NORMAL,

+ KMALLOC_PARTITION_END = KMALLOC_PARTITION_START + KMALLOC_PARTITION_CACHES_NR,

#ifdef CONFIG_SLUB_TINY
KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else

@@ -662,19 +686,19 @@ extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES];

(IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \
(IS_ENABLED(CONFIG_MEMCG) ? __GFP_ACCOUNT : 0))

-extern unsigned long random_kmalloc_seed;
-
-static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, unsigned long caller)
+static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags, kmalloc_token_t token)
{
/*
* The most common case is KMALLOC_NORMAL, so test for it

* with a single branch for all the relevant flags.
*/
if (likely((flags & KMALLOC_NOT_NORMAL_BITS) == 0))
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

- /* RANDOM_KMALLOC_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
- return KMALLOC_RANDOM_START + hash_64(caller ^ random_kmalloc_seed,
- ilog2(RANDOM_KMALLOC_CACHES_NR + 1));

+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
+ /* KMALLOC_PARTITION_CACHES_NR (=15) copies + the KMALLOC_NORMAL */
+ return KMALLOC_PARTITION_START + hash_64(token.v ^ random_kmalloc_seed,
+ ilog2(KMALLOC_PARTITION_CACHES_NR + 1));
+#elif defined(CONFIG_KMALLOC_PARTITION_TYPED)
+ return KMALLOC_PARTITION_START + token.v;
#else
return KMALLOC_NORMAL;
#endif
@@ -858,16 +882,22 @@ unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);

#define PASS_BUCKET_PARAM(_b) NULL
#endif

+#define DECL_KMALLOC_PARAMS(_size, _b, _token) DECL_BUCKET_PARAMS(_size, _b) \
+ DECL_TOKEN_PARAM(_token)
+
+#define PASS_KMALLOC_PARAMS(_size, _b, _token) PASS_BUCKET_PARAMS(_size, _b) \
+ _PASS_TOKEN_PARAM(_token)
+
/*
* The following functions are not to be used directly and are intended only
* for internal use from kmalloc() and kmalloc_node()
* with the exception of kunit tests
*/

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
__assume_kmalloc_alignment __alloc_size(1);

-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
__assume_kmalloc_alignment __alloc_size(1);

void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size)

@@ -938,7 +968,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -948,14 +978,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f

index = kmalloc_index(size);
return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, size);
}
- return __kmalloc_noprof(size, flags);
+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
}
+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))

-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

+void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);
+#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)
#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))

/**
@@ -1060,12 +1092,12 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);

__alloc_flex(kvzalloc, default_gfp(__VA_ARGS__), typeof(P), FAM, COUNT)

#define kmem_buckets_alloc(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE))
+ alloc_hooks(__kmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE))

#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \
- alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_))
+ alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(_size, _b, __kmalloc_token(_size)), _flags, NUMA_NO_NODE, _RET_IP_))

-static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, gfp_t flags, int node, kmalloc_token_t token)
{
if (__builtin_constant_p(size) && size) {
unsigned int index;

@@ -1075,11 +1107,12 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

index = kmalloc_index(size);
return __kmalloc_cache_node_noprof(
- kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
+ kmalloc_caches[kmalloc_type(flags, token)][index],
flags, node, size);
}
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags, node);
}
+#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

/**

@@ -1088,14 +1121,15 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
- return kmalloc_noprof(bytes, flags);
+ return _kmalloc_noprof(bytes, flags, token);
}
+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

/**

@@ -1115,18 +1149,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/
-static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p,
+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
size_t new_n,
size_t new_size,
- gfp_t flags)
+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;

- return krealloc_noprof(p, bytes, flags);

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);

}
+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))

/**

@@ -1137,10 +1172,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi

*/
#define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO)

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node,
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node,
unsigned long caller) __alloc_size(1);
#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \
- __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller)
+ __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(size, NULL, __kmalloc_token(size)), flags, node, caller)
#define kmalloc_node_track_caller(...) \
alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_))

@@ -1157,17 +1192,18 @@ void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flag

#define kmalloc_track_caller_noprof(...) \
kmalloc_node_track_caller_noprof(__VA_ARGS__, NUMA_NO_NODE, _RET_IP_)

-static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags,
+ int node, kmalloc_token_t token)
{
size_t bytes;

if (unlikely(check_mul_overflow(n, size, &bytes)))
return NULL;
if (__builtin_constant_p(n) && __builtin_constant_p(size))
- return kmalloc_node_noprof(bytes, flags, node);
- return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node);
+ return _kmalloc_node_noprof(bytes, flags, node, token);
+ return __kmalloc_node_noprof(PASS_KMALLOC_PARAMS(bytes, NULL, token), flags, node);
}
+#define kmalloc_array_node_noprof(...) _kmalloc_array_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__))

#define kcalloc_node(_n, _size, _flags, _node) \

@@ -1183,39 +1219,43 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_

@@ -1224,10 +1264,10 @@ kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node)
#define kvcalloc_node(...) alloc_hooks(kvcalloc_node_noprof(__VA_ARGS__))
#define kvcalloc(...) alloc_hooks(kvcalloc_noprof(__VA_ARGS__))

-void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
+void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid) __realloc_size(2);
-#define kvrealloc_node_align(...) \
- alloc_hooks(kvrealloc_node_align_noprof(__VA_ARGS__))
+#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
+ alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))

#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/init/Kconfig b/init/Kconfig
index 2937c4d308ae..dba97bc8e2ce 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -170,6 +170,9 @@ config CC_HAS_BROKEN_COUNTED_BY_REF
# https://github.com/llvm/llvm-project/issues/182575
default y if CC_IS_CLANG && CLANG_VERSION < 220100

+config CC_HAS_ALLOC_TOKEN
+ def_bool $(cc-option,-falloc-token-max=123)
+
config CC_HAS_MULTIDIMENSIONAL_NONSTRING
def_bool $(success,echo 'char tag[][4] __attribute__((__nonstring__)) = { };' | $(CC) $(CLANG_FLAGS) -x c - -c -o /dev/null -Werror)

diff --git a/kernel/configs/hardening.config b/kernel/configs/hardening.config
index 7c3924614e01..26831a2a5739 100644

--- a/kernel/configs/hardening.config
+++ b/kernel/configs/hardening.config
@@ -22,7 +22,7 @@ CONFIG_SLAB_FREELIST_RANDOM=y
CONFIG_SLAB_FREELIST_HARDENED=y
CONFIG_SLAB_BUCKETS=y
CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-CONFIG_RANDOM_KMALLOC_CACHES=y

+CONFIG_KMALLOC_PARTITION_CACHES=y

# Sanity check userspace page table mappings.
CONFIG_PAGE_TABLE_CHECK=y

diff --git a/mm/Kconfig b/mm/Kconfig
index e8bf1e9e6ad9..4f187b07eb48 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -248,22 +248,75 @@ config SLUB_STATS

out which slabs are relevant to a particular load.
Try running: slabinfo -DA

-config RANDOM_KMALLOC_CACHES
- default n

+config KMALLOC_PARTITION_CACHES
depends on !SLUB_TINY
- bool "Randomize slab caches for normal kmalloc"
+ bool "Partitioned slab caches for normal kmalloc"
+ default RANDOM_KMALLOC_CACHES

help
- A hardening feature that creates multiple copies of slab caches for
- normal kmalloc allocation and makes kmalloc randomly pick one based
- on code address, which makes the attackers more difficult to spray
- vulnerable memory objects on the heap for the purpose of exploiting
- memory vulnerabilities.
+ A hardening feature that creates multiple isolated copies of slab
+ caches for normal kmalloc allocations. This makes it more difficult
+ to exploit memory-safety vulnerabilities by attacking vulnerable
+ co-located memory objects. Several modes are provided.

Currently the number of copies is set to 16, a reasonably large value

that effectively diverges the memory objects allocated for different
subsystems or modules into different caches, at the expense of a
- limited degree of memory and CPU overhead that relates to hardware and
- system workload.
+ limited degree of memory and CPU overhead that relates to hardware
+ and system workload.
+
+choice

+ prompt "Partitioned slab cache mode"
+ depends on KMALLOC_PARTITION_CACHES
+ default KMALLOC_PARTITION_TYPED if CC_HAS_ALLOC_TOKEN
+ default KMALLOC_PARTITION_RANDOM
+ help
+ Selects the slab cache partitioning mode.
+
+config KMALLOC_PARTITION_RANDOM

+ bool "Randomize slab caches for normal kmalloc"
+ help
+ Randomly pick a slab cache based on code address and a per-boot
+ random seed.
+
+ This makes it harder for attackers to predict object co-location.
+ The placement is random: while attackers don't know which kmalloc
+ cache an object will be allocated from, they might circumvent

+ the randomization by retrying attacks across multiple machines until
+ the target objects are co-located.
+
+config KMALLOC_PARTITION_TYPED
+ bool "Type based slab cache selection for normal kmalloc"
+ depends on CC_HAS_ALLOC_TOKEN
+ help
+ Rely on Clang's allocation tokens to choose a slab cache, where token
+ IDs are derived from the allocated type.
+
+ Unlike KMALLOC_PARTITION_RANDOM, cache assignment is deterministic based
+ on type, which guarantees that objects of certain types are not
+ placed in the same cache. This effectively mitigates certain classes
+ of exploits that probabilistic defenses like KMALLOC_PARTITION_RANDOM

+ only make harder but not impossible. However, this also means the
+ cache assignment is predictable.
+
+ Clang's default token ID calculation returns a bounded hash with
+ disjoint ranges for pointer-containing and pointerless objects: when
+ used as the slab cache index, this prevents buffer overflows on
+ primitive buffers from directly corrupting pointer-containing
+ objects.
+
+ The current effectiveness of Clang's type inference can be judged by
+ -Rpass=alloc-token, which provides diagnostics where (after dead-code
+ elimination) type inference failed.
+
+ Requires Clang 22 or later.
+
+endchoice

+
+config RANDOM_KMALLOC_CACHES
+ bool

+ transitional
+ help
+ Transitional config for migration to KMALLOC_PARTITION_CACHES.

endmenu # Slab allocator options

diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c
index 5725a367246d..8807ea8ed0d3 100644
--- a/mm/kfence/kfence_test.c
+++ b/mm/kfence/kfence_test.c
@@ -214,7 +214,7 @@ static void test_cache_destroy(void)
static inline size_t kmalloc_cache_alignment(size_t size)
{
/* just to get ->align so no need to pass in the real caller */
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, 0);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(0));
return kmalloc_caches[type][__kmalloc_index(size, false)]->align;
}

@@ -285,7 +285,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat

if (is_kfence_address(alloc)) {
struct slab *slab = virt_to_slab(alloc);
- enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, _RET_IP_);
+ enum kmalloc_cache_type type = kmalloc_type(GFP_KERNEL, __kmalloc_token(size));
struct kmem_cache *s = test_cache ?:
kmalloc_caches[type][__kmalloc_index(size, false)];

diff --git a/mm/slab.h b/mm/slab.h

index bf2f87acf5e3..1bf9c3021ae3 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -362,12 +362,12 @@ static inline unsigned int size_index_elem(unsigned int bytes)

* KMALLOC_MAX_CACHE_SIZE and the caller must check that.
*/
static inline struct kmem_cache *
-kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller)
+kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, kmalloc_token_t token)
{
unsigned int index;

if (!b)
- b = &kmalloc_caches[kmalloc_type(flags, caller)];
+ b = &kmalloc_caches[kmalloc_type(flags, token)];
if (size <= 192)
index = kmalloc_size_index[size_index_elem(size)];
else

diff --git a/mm/slab_common.c b/mm/slab_common.c
index d5a70a831a2a..388eb5980859 100644

--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -742,7 +742,7 @@ kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init =
{ /* initialization for https://llvm.org/pr42570 */ };
EXPORT_SYMBOL(kmalloc_caches);

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM

unsigned long random_kmalloc_seed __ro_after_init;
EXPORT_SYMBOL(random_kmalloc_seed);
#endif

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES

+#define __KMALLOC_PARTITION_CONCAT(a, b) a ## b
+#define KMALLOC_PARTITION_NAME(N, sz) __KMALLOC_PARTITION_CONCAT(KMA_PART_, N)(sz)
+#define KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 1] = "kmalloc-part-01-" #sz,
+#define KMA_PART_2(sz) KMA_PART_1(sz) .name[KMALLOC_PARTITION_START + 2] = "kmalloc-part-02-" #sz,
+#define KMA_PART_3(sz) KMA_PART_2(sz) .name[KMALLOC_PARTITION_START + 3] = "kmalloc-part-03-" #sz,
+#define KMA_PART_4(sz) KMA_PART_3(sz) .name[KMALLOC_PARTITION_START + 4] = "kmalloc-part-04-" #sz,
+#define KMA_PART_5(sz) KMA_PART_4(sz) .name[KMALLOC_PARTITION_START + 5] = "kmalloc-part-05-" #sz,
+#define KMA_PART_6(sz) KMA_PART_5(sz) .name[KMALLOC_PARTITION_START + 6] = "kmalloc-part-06-" #sz,
+#define KMA_PART_7(sz) KMA_PART_6(sz) .name[KMALLOC_PARTITION_START + 7] = "kmalloc-part-07-" #sz,
+#define KMA_PART_8(sz) KMA_PART_7(sz) .name[KMALLOC_PARTITION_START + 8] = "kmalloc-part-08-" #sz,
+#define KMA_PART_9(sz) KMA_PART_8(sz) .name[KMALLOC_PARTITION_START + 9] = "kmalloc-part-09-" #sz,
+#define KMA_PART_10(sz) KMA_PART_9(sz) .name[KMALLOC_PARTITION_START + 10] = "kmalloc-part-10-" #sz,
+#define KMA_PART_11(sz) KMA_PART_10(sz) .name[KMALLOC_PARTITION_START + 11] = "kmalloc-part-11-" #sz,
+#define KMA_PART_12(sz) KMA_PART_11(sz) .name[KMALLOC_PARTITION_START + 12] = "kmalloc-part-12-" #sz,
+#define KMA_PART_13(sz) KMA_PART_12(sz) .name[KMALLOC_PARTITION_START + 13] = "kmalloc-part-13-" #sz,
+#define KMA_PART_14(sz) KMA_PART_13(sz) .name[KMALLOC_PARTITION_START + 14] = "kmalloc-part-14-" #sz,
+#define KMA_PART_15(sz) KMA_PART_14(sz) .name[KMALLOC_PARTITION_START + 15] = "kmalloc-part-15-" #sz,

+#else // CONFIG_KMALLOC_PARTITION_CACHES

+#define KMALLOC_PARTITION_NAME(N, sz)
#endif

#define INIT_KMALLOC_INFO(__size, __short_size) \
@@ -849,7 +849,7 @@ EXPORT_SYMBOL(kmalloc_size_roundup);
KMALLOC_RCL_NAME(__short_size) \
KMALLOC_CGROUP_NAME(__short_size) \
KMALLOC_DMA_NAME(__short_size) \
- KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size) \

+ KMALLOC_PARTITION_NAME(KMALLOC_PARTITION_CACHES_NR, __short_size) \

.size = __size, \
}

@@ -961,8 +961,8 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
flags |= SLAB_CACHE_DMA;
}

-#ifdef CONFIG_RANDOM_KMALLOC_CACHES
- if (type >= KMALLOC_RANDOM_START && type <= KMALLOC_RANDOM_END)

+#ifdef CONFIG_KMALLOC_PARTITION_CACHES

+ if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
flags |= SLAB_NO_MERGE;

#endif

@@ -1010,7 +1010,7 @@ void __init create_kmalloc_caches(void)
for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
new_kmalloc_cache(i, type);
}
-#ifdef CONFIG_RANDOM_KMALLOC_CACHES

+#ifdef CONFIG_KMALLOC_PARTITION_RANDOM
random_kmalloc_seed = get_random_u64();

#endif

diff --git a/mm/slub.c b/mm/slub.c

index 161079ac5ba1..ccb208cfbecd 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2129,11 +2129,11 @@ static inline size_t obj_exts_alloc_size(struct kmem_cache *s,

if (!is_kmalloc_normal(s))
return sz;

- obj_exts_cache = kmalloc_slab(sz, NULL, gfp, 0);
+ obj_exts_cache = kmalloc_slab(sz, NULL, gfp, __kmalloc_token(0));
/*
- * We can't simply compare s with obj_exts_cache, because random kmalloc
- * caches have multiple caches per size, selected by caller address.
- * Since caller address may differ between kmalloc_slab() and actual
+ * We can't simply compare s with obj_exts_cache, because partitioned kmalloc
+ * caches have multiple caches per size, selected by caller address or type.
+ * Since caller address or type may differ between kmalloc_slab() and actual
* allocation, bump size when sizes are equal.
*/
if (s->object_size == obj_exts_cache->object_size)

@@ -5274,7 +5274,7 @@ EXPORT_SYMBOL(__kmalloc_large_node_noprof);

static __always_inline
void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,
- unsigned long caller)
+ unsigned long caller, kmalloc_token_t token)
{
struct kmem_cache *s;
void *ret;

@@ -5289,22 +5289,24 @@ void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node,

if (unlikely(!size))
return ZERO_SIZE_PTR;

- s = kmalloc_slab(size, b, flags, caller);
+ s = kmalloc_slab(size, b, flags, token);

ret = slab_alloc_node(s, NULL, flags, node, caller, size);
ret = kasan_kmalloc(s, ret, size, flags);
trace_kmalloc(caller, ret, size, s->size, flags, node);
return ret;
}
-void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node)
+void *__kmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags, int node)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_node_noprof);

-void *__kmalloc_noprof(size_t size, gfp_t flags)
+void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
{
- return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags,
+ NUMA_NO_NODE, _RET_IP_, PASS_TOKEN_PARAM(token));
}
EXPORT_SYMBOL(__kmalloc_noprof);

@@ -5319,7 +5321,7 @@ EXPORT_SYMBOL(__kmalloc_noprof);

* NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
* There is no reason to call it again and expect !NULL.
*/
-void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

+void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)

{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;
struct kmem_cache *s;

@@ -5342,7 +5344,7 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

retry:
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return NULL;
- s = kmalloc_slab(size, NULL, alloc_gfp, _RET_IP_);
+ s = kmalloc_slab(size, NULL, alloc_gfp, PASS_TOKEN_PARAM(token));

if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s))
/*

@@ -5395,12 +5397,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node)

ret = kasan_kmalloc(s, ret, size, alloc_gfp);
return ret;
}
-EXPORT_SYMBOL_GPL(kmalloc_nolock_noprof);
+EXPORT_SYMBOL_GPL(_kmalloc_nolock_noprof);

-void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags,
+void *__kmalloc_node_track_caller_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags,
int node, unsigned long caller)
{
- return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller);
+ return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node,
+ caller, PASS_TOKEN_PARAM(token));

}
EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof);

@@ -6631,7 +6634,7 @@ void kfree_nolock(const void *object)

EXPORT_SYMBOL_GPL(kfree_nolock);

static __always_inline __realloc_size(2) void *
-__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid)
+__do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags, int nid, kmalloc_token_t token)
{
void *ret;
size_t ks = 0;

@@ -6703,7 +6706,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,

return (void *)p;

alloc_new:
- ret = kmalloc_node_track_caller_noprof(new_size, flags, nid, _RET_IP_);
+ ret = __kmalloc_node_track_caller_noprof(PASS_KMALLOC_PARAMS(new_size, NULL, token), flags, nid, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();

@@ -6752,7 +6755,7 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
*

* Return: pointer to the allocated memory or %NULL in case of error
*/

-void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long align,
+void *krealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(new_size, token), unsigned long align,
gfp_t flags, int nid)
{
void *ret;
@@ -6762,7 +6765,7 @@ void *krealloc_node_align_noprof(const void *p, size_t new_size, unsigned long a

return ZERO_SIZE_PTR;
}

- ret = __do_krealloc(p, new_size, align, flags, nid);
+ ret = __do_krealloc(p, new_size, align, flags, nid, PASS_TOKEN_PARAM(token));
if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret))
kfree(p);

@@ -6799,6 +6802,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

* failure, fall back to non-contiguous (vmalloc) allocation.
* @size: size of the request.
* @b: which set of kmalloc buckets to allocate from.
+ * @token: allocation token.
* @align: desired alignment.
* @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
* @node: numa node to allocate from

@@ -6815,7 +6819,7 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)

*
* Return: pointer to the allocated memory of %NULL in case of failure
*/
-void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

+void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node)
{
bool allow_block;
@@ -6827,7 +6831,7 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), unsigned long align,

*/
ret = __do_kmalloc_node(size, PASS_BUCKET_PARAM(b),
kmalloc_gfp_adjust(flags, size),
- node, _RET_IP_);
+ node, _RET_IP_, PASS_TOKEN_PARAM(token));
if (ret || size <= PAGE_SIZE)
return ret;

@@ -6923,7 +6927,7 @@ EXPORT_SYMBOL(kvfree_sensitive);
*

* Return: pointer to the allocated memory or %NULL in case of error
*/

-void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long align,
+void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid)
{
void *n;
@@ -6931,10 +6935,10 @@ void *kvrealloc_node_align_noprof(const void *p, size_t size, unsigned long alig

if (is_vmalloc_addr(p))
return vrealloc_node_align_noprof(p, size, align, flags, nid);

- n = krealloc_node_align_noprof(p, size, align, kmalloc_gfp_adjust(flags, size), nid);

+ n = krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(size, token), align, kmalloc_gfp_adjust(flags, size), nid);

if (!n) {
/* We failed to krealloc(), fall back to kvmalloc(). */
- n = kvmalloc_node_align_noprof(size, align, flags, nid);
+ n = __kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), align, flags, nid);
if (!n)
return NULL;

@@ -8449,7 +8453,7 @@ static void __init bootstrap_kmalloc_sheaves(void)

{
enum kmalloc_cache_type type;

- for (type = KMALLOC_NORMAL; type <= KMALLOC_RANDOM_END; type++) {
+ for (type = KMALLOC_NORMAL; type <= KMALLOC_PARTITION_END; type++) {
for (int idx = 0; idx < KMALLOC_SHIFT_HIGH + 1; idx++) {
if (kmalloc_caches[type][idx])
bootstrap_cache_sheaves(kmalloc_caches[type][idx]);
--

2.54.0.545.g6539524ca2-goog

[Marco Elver]

The mm-api kernel-doc comments have been broken for a while, as many
documented symbols shifted from being direct function definitions to
macros wrapping _noprof implementations during the introduction of
allocation tagging (starting with commit 7bd230a26648 "mm/slab: enable
slab allocation tagging for kmalloc and friends").

When the kernel-doc block remains above the internal implementation
function but uses the public API name, the documentation generator fails
to associate the documented symbol and generates warnings and fails to
emit the documentation.

Fix this by:

1. Moving the kernel-doc comment blocks from slub.c to slab.h, placing
them directly above the user-facing macros.

2. Converting the variadic macros for the documented APIs to use
explicit arguments.

No functional change intended.

Signed-off-by: Marco Elver <el...@google.com>
---

include/linux/slab.h | 192 ++++++++++++++++++++++++++++++++-----------
mm/slub.c | 98 ----------------------
2 files changed, 144 insertions(+), 146 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c232f8a10af6..dcff0da1c01c 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -530,7 +530,46 @@ void * __must_check krealloc_node_align_noprof(const void *objp,

unsigned long align,
gfp_t flags, int nid) __realloc_size(2);

#define krealloc_noprof(_o, _s, _f) krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), 1, _f, NUMA_NO_NODE)

-#define krealloc_node_align(_o, _s, _a, _f, _n) alloc_hooks(krealloc_node_align_noprof(_o, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))
+/**
+ * krealloc_node_align - reallocate memory. The contents will remain unchanged.
+ * @p: object to reallocate memory for.
+ * @new_size: how many bytes of memory are required.
+ * @align: desired alignment.
+ * @flags: the type of memory to allocate.
+ * @nid: NUMA node or NUMA_NO_NODE
+ *
+ * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size
+ * is 0 and @p is not a %NULL pointer, the object pointed to is freed.
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
+ * initial memory allocation, every subsequent call to this API for the same
+ * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
+ * __GFP_ZERO is not fully honored by this API.
+ *
+ * When slub_debug_orig_size() is off, krealloc() only knows about the bucket
+ * size of an allocation (but not the exact size it was allocated with) and
+ * hence implements the following semantics for shrinking and growing buffers
+ * with __GFP_ZERO::
+ *
+ * new bucket
+ * 0 size size
+ * |--------|----------------|
+ * | keep | zero |
+ *
+ * Otherwise, the original allocation size 'orig_size' could be used to
+ * precisely clear the requested size, and the new size will also be stored
+ * as the new 'orig_size'.
+ *
+ * In any case, the contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes.
+ *
+ * Return: pointer to the allocated memory or %NULL in case of error
+ */
+#define krealloc_node_align(p, new_size, align, flags, nid) \
+ alloc_hooks(krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(new_size, __kmalloc_token(new_size)), align, flags, nid))

#define krealloc_node(_o, _s, _f, _n) krealloc_node_align(_o, _s, 1, _f, _n)
#define krealloc(...) krealloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -913,6 +952,22 @@ void *__kmalloc_large_noprof(size_t size, gfp_t flags)

void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

__assume_page_alignment __alloc_size(1);

+static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)

+{
+ if (__builtin_constant_p(size) && size) {
+ unsigned int index;
+
+ if (size > KMALLOC_MAX_CACHE_SIZE)
+ return __kmalloc_large_noprof(size, flags);
+
+ index = kmalloc_index(size);
+ return __kmalloc_cache_noprof(

+ kmalloc_caches[kmalloc_type(flags, token)][index],

+ flags, size);
+ }

+ return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
+}

+#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
/**
* kmalloc - allocate kernel memory

* @size: how many bytes of memory are required.

@@ -968,27 +1023,22 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)

* Try really hard to succeed the allocation but fail
* eventually.
*/

-static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
-{
- if (__builtin_constant_p(size) && size) {
- unsigned int index;
-
- if (size > KMALLOC_MAX_CACHE_SIZE)
- return __kmalloc_large_noprof(size, flags);
-
- index = kmalloc_index(size);
- return __kmalloc_cache_noprof(
- kmalloc_caches[kmalloc_type(flags, token)][index],
- flags, size);
- }
- return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);
-}
-#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))
+#define kmalloc(size, flags) alloc_hooks(kmalloc_noprof(size, flags))

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);

#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)

-#define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))
+/**
+ * kmalloc_nolock - Allocate an object of given size from any context.
+ * @size: size to allocate
+ * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
+ * allowed.
+ * @node: node number of the target node.
+ *
+ * Return: pointer to the new object or NULL in case of error.
+ * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
+ * There is no reason to call it again and expect !NULL.
+ */
+#define kmalloc_nolock(size, gfp_flags, node) alloc_hooks(kmalloc_nolock_noprof(size, gfp_flags, node))

/**
* __alloc_objs - Allocate objects of a given type using
@@ -1115,23 +1165,36 @@ static __always_inline __alloc_size(1) void *_kmalloc_node_noprof(size_t size, g

#define kmalloc_node_noprof(...) _kmalloc_node_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__))

+static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)

+{
+ size_t bytes;
+
+ if (unlikely(check_mul_overflow(n, size, &bytes)))
+ return NULL;

+ return _kmalloc_noprof(bytes, flags, token);
+}

+#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
/**
* kmalloc_array - allocate memory for an array.
* @n: number of elements.

* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/

-static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t size, gfp_t flags, kmalloc_token_t token)
+#define kmalloc_array(n, size, flags) alloc_hooks(kmalloc_array_noprof(n, size, flags))
+

+static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,

+ size_t new_n,
+ size_t new_size,

+ gfp_t flags, kmalloc_token_t token)
{
size_t bytes;

- if (unlikely(check_mul_overflow(n, size, &bytes)))
+ if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
return NULL;
- return _kmalloc_noprof(bytes, flags, token);
-}
-#define kmalloc_array_noprof(...) _kmalloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__))

+ return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
+}

+#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
/**
* krealloc_array - reallocate memory for an array.
* @p: pointer to the memory chunk to reallocate
@@ -1149,20 +1212,7 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_noprof(size_t n, size_t si

* In any case, the contents of the object pointed to are preserved up to the
* lesser of the new and old sizes.
*/

-static inline __realloc_size(2, 3) void * __must_check _krealloc_array_noprof(void *p,
- size_t new_n,
- size_t new_size,
- gfp_t flags, kmalloc_token_t token)
-{
- size_t bytes;
-
- if (unlikely(check_mul_overflow(new_n, new_size, &bytes)))
- return NULL;
-
- return krealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(bytes, token), 1, flags, NUMA_NO_NODE);
-}
-#define krealloc_array_noprof(...) _krealloc_array_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__))
+#define krealloc_array(p, new_n, new_size, flags) alloc_hooks(krealloc_array_noprof(p, new_n, new_size, flags))

/**
* kcalloc - allocate memory for an array. The memory is set to zero.
@@ -1214,17 +1264,17 @@ static inline __alloc_size(1, 2) void *_kmalloc_array_node_noprof(size_t n, size
*/
#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO)

-/**
- * kzalloc - allocate memory. The memory is set to zero.
- * @size: how many bytes of memory are required.
- * @flags: the type of memory to allocate (see kmalloc).
- */

static inline __alloc_size(1) void *_kzalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
{

return _kmalloc_noprof(size, flags | __GFP_ZERO, token);
}

#define kzalloc_noprof(...) _kzalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))
-#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__))
+/**
+ * kzalloc - allocate memory. The memory is set to zero.
+ * @size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate (see kmalloc).
+ */
+#define kzalloc(size, flags) alloc_hooks(kzalloc_noprof(size, flags))

#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node)

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,

@@ -1233,7 +1283,26 @@ void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long

__kvmalloc_node_noprof(PASS_KMALLOC_PARAMS(_size, NULL, __kmalloc_token(_size)), _align, _flags, _node)
#define kvmalloc_node_align(...) \
alloc_hooks(kvmalloc_node_align_noprof(__VA_ARGS__))

-#define kvmalloc_node(_s, _f, _n) kvmalloc_node_align(_s, 1, _f, _n)
+/**
+ * kvmalloc_node - attempt to allocate physically contiguous memory, but upon
+ * failure, fall back to non-contiguous (vmalloc) allocation.
+ * @size: size of the request.
+ * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
+ * @node: numa node to allocate from
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * Uses kmalloc to get the memory but if the allocation fails then falls back
+ * to the vmalloc allocator. Use kvfree for freeing the memory.
+ *
+ * GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not.
+ * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is
+ * preferable to the vmalloc fallback, due to visible performance drawbacks.
+ *
+ * Return: pointer to the allocated memory of %NULL in case of failure
+ */
+#define kvmalloc_node(size, flags, node) kvmalloc_node_align(size, 1, flags, node)
#define kvmalloc_node_noprof(size, flags, node) \

kvmalloc_node_align_noprof(size, 1, flags, node)
#define kvmalloc(...) kvmalloc_node(__VA_ARGS__, NUMA_NO_NODE)

@@ -1266,8 +1335,35 @@ _kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node, kmallo

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid) __realloc_size(2);

-#define kvrealloc_node_align(_p, _s, _a, _f, _n) \
- alloc_hooks(kvrealloc_node_align_noprof(_p, PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _a, _f, _n))
+/**
+ * kvrealloc_node_align - reallocate memory; contents remain unchanged
+ * @p: object to reallocate memory for
+ * @size: the size to reallocate
+ * @align: desired alignment
+ * @flags: the flags for the page level allocator
+ * @nid: NUMA node id
+ *
+ * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0
+ * and @p is not a %NULL pointer, the object pointed to is freed.
+ *
+ * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
+ * Documentation/core-api/memory-allocation.rst for more details.
+ *
+ * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
+ * initial memory allocation, every subsequent call to this API for the same
+ * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
+ * __GFP_ZERO is not fully honored by this API.
+ *
+ * In any case, the contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes.
+ *
+ * This function must not be called concurrently with itself or kvfree() for the
+ * same memory allocation.
+ *
+ * Return: pointer to the allocated memory or %NULL in case of error
+ */
+#define kvrealloc_node_align(p, size, align, flags, nid) \
+ alloc_hooks(kvrealloc_node_align_noprof(p, PASS_TOKEN_PARAMS(size, __kmalloc_token(size)), align, flags, nid))

#define kvrealloc_node(_p, _s, _f, _n) kvrealloc_node_align(_p, _s, 1, _f, _n)
#define kvrealloc(...) kvrealloc_node(__VA_ARGS__, NUMA_NO_NODE)

diff --git a/mm/slub.c b/mm/slub.c
index ccb208cfbecd..dbc3c947e5be 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5310,17 +5310,6 @@ void *__kmalloc_noprof(DECL_KMALLOC_PARAMS(size, b, token), gfp_t flags)
}
EXPORT_SYMBOL(__kmalloc_noprof);

-/**
- * kmalloc_nolock - Allocate an object of given size from any context.
- * @size: size to allocate
- * @gfp_flags: GFP flags. Only __GFP_ACCOUNT, __GFP_ZERO, __GFP_NO_OBJ_EXT
- * allowed.
- * @node: node number of the target node.
- *
- * Return: pointer to the new object or NULL in case of error.
- * NULL does not mean EBUSY or EAGAIN. It means ENOMEM.
- * There is no reason to call it again and expect !NULL.
- */

void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node)
{
gfp_t alloc_gfp = __GFP_NOWARN | __GFP_NOMEMALLOC | gfp_flags;

@@ -6717,44 +6706,6 @@ __do_krealloc(const void *p, size_t new_size, unsigned long align, gfp_t flags,
return ret;
}

-/**
- * krealloc_node_align - reallocate memory. The contents will remain unchanged.
- * @p: object to reallocate memory for.
- * @new_size: how many bytes of memory are required.
- * @align: desired alignment.
- * @flags: the type of memory to allocate.
- * @nid: NUMA node or NUMA_NO_NODE
- *
- * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size
- * is 0 and @p is not a %NULL pointer, the object pointed to is freed.
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
- * initial memory allocation, every subsequent call to this API for the same
- * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
- * __GFP_ZERO is not fully honored by this API.
- *
- * When slub_debug_orig_size() is off, krealloc() only knows about the bucket
- * size of an allocation (but not the exact size it was allocated with) and
- * hence implements the following semantics for shrinking and growing buffers
- * with __GFP_ZERO::
- *
- * new bucket
- * 0 size size
- * |--------|----------------|
- * | keep | zero |
- *
- * Otherwise, the original allocation size 'orig_size' could be used to
- * precisely clear the requested size, and the new size will also be stored
- * as the new 'orig_size'.
- *
- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */

void *krealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(new_size, token), unsigned long align,
gfp_t flags, int nid)
{

@@ -6797,28 +6748,6 @@ static gfp_t kmalloc_gfp_adjust(gfp_t flags, size_t size)
return flags;
}

-/**
- * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon
- * failure, fall back to non-contiguous (vmalloc) allocation.
- * @size: size of the request.
- * @b: which set of kmalloc buckets to allocate from.
- * @token: allocation token.
- * @align: desired alignment.
- * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL.
- * @node: numa node to allocate from
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * Uses kmalloc to get the memory but if the allocation fails then falls back
- * to the vmalloc allocator. Use kvfree for freeing the memory.
- *
- * GFP_NOWAIT and GFP_ATOMIC are supported, the __GFP_NORETRY modifier is not.
- * __GFP_RETRY_MAYFAIL is supported, and it should be used only if kmalloc is
- * preferable to the vmalloc fallback, due to visible performance drawbacks.
- *
- * Return: pointer to the allocated memory of %NULL in case of failure
- */

void *__kvmalloc_node_noprof(DECL_KMALLOC_PARAMS(size, b, token), unsigned long align,
gfp_t flags, int node)

{
@@ -6900,33 +6829,6 @@ void kvfree_sensitive(const void *addr, size_t len)
}
EXPORT_SYMBOL(kvfree_sensitive);

-/**
- * kvrealloc_node_align - reallocate memory; contents remain unchanged
- * @p: object to reallocate memory for
- * @size: the size to reallocate
- * @align: desired alignment
- * @flags: the flags for the page level allocator
- * @nid: NUMA node id
- *
- * If @p is %NULL, kvrealloc() behaves exactly like kvmalloc(). If @size is 0
- * and @p is not a %NULL pointer, the object pointed to is freed.
- *
- * Only alignments up to those guaranteed by kmalloc() will be honored. Please see
- * Documentation/core-api/memory-allocation.rst for more details.
- *
- * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
- * initial memory allocation, every subsequent call to this API for the same
- * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
- * __GFP_ZERO is not fully honored by this API.
- *
- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * This function must not be called concurrently with itself or kvfree() for the
- * same memory allocation.
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */

void *kvrealloc_node_align_noprof(const void *p, DECL_TOKEN_PARAMS(size, token), unsigned long align,
gfp_t flags, int nid)

{
--
2.54.0.545.g6539524ca2-goog

[Vlastimil Babka (SUSE)]

On 4/24/26 15:24, Marco Elver wrote:

> @@ -938,7 +968,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node)
> * Try really hard to succeed the allocation but fail
> * eventually.
> */
> -static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags)
> +static __always_inline __alloc_size(1) void *_kmalloc_noprof(size_t size, gfp_t flags, kmalloc_token_t token)
> {
> if (__builtin_constant_p(size) && size) {
> unsigned int index;
> @@ -948,14 +978,16 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f
>
> index = kmalloc_index(size);
> return __kmalloc_cache_noprof(
> - kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index],
> + kmalloc_caches[kmalloc_type(flags, token)][index],

While reviewing this, it occured to me we might have been using _RET_IP_
here in a suboptimal way ever since this was introduced. Since this is all
inlined, shouldn't have we been using _THIS_IP_ to really randomize using
the kmalloc() callsite, and not its parent?

And after this patch, we get the token passed to _kmalloc_noprof()...

> flags, size);
> }
> - return __kmalloc_noprof(size, flags);
> + return __kmalloc_noprof(PASS_KMALLOC_PARAMS(size, NULL, token), flags);

... and used also here for the non-constant-size, where previously
__kmalloc_noprof() (not inline function) would correctly use _RET_IP_ on its
own ...

> }
> +#define kmalloc_noprof(...) _kmalloc_noprof(__VA_ARGS__, __kmalloc_token(__VA_ARGS__))

... and the token comes from here. With random partitioning that's

#define __kmalloc_token(...) ((kmalloc_token_t){ .v = _RET_IP_ })

so that AFAIK makes the situation worse as now the cases without constant
size also start randomizing by the parent callsite and not the kmalloc callsite.

But there are many users of __kmalloc_token() and maybe some are corrent in
using _RET_IP_, I haven't checked, maybe we'll need two variants, or further
change things around.

> #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__))
>
> -void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_flags, int node);
> +void *_kmalloc_nolock_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t gfp_flags, int node);
> +#define kmalloc_nolock_noprof(_s, _f, _n) _kmalloc_nolock_noprof(PASS_TOKEN_PARAMS(_s, __kmalloc_token(_s)), _f, _n)
> #define kmalloc_nolock(...) alloc_hooks(kmalloc_nolock_noprof(__VA_ARGS__))
>
> /**

<snip>

We're renaming the caches visible in /proc/slabinfo. Maybe it's fine as
people who care will just adapt their tools without complaining. Not the
first time we've changes something here.

[Vlastimil Babka (SUSE)]

On 4/24/26 15:24, Marco Elver wrote:

> The mm-api kernel-doc comments have been broken for a while, as many
> documented symbols shifted from being direct function definitions to
> macros wrapping _noprof implementations during the introduction of
> allocation tagging (starting with commit 7bd230a26648 "mm/slab: enable
> slab allocation tagging for kmalloc and friends").
>
> When the kernel-doc block remains above the internal implementation
> function but uses the public API name, the documentation generator fails
> to associate the documented symbol and generates warnings and fails to
> emit the documentation.
>
> Fix this by:
>
> 1. Moving the kernel-doc comment blocks from slub.c to slab.h, placing
> them directly above the user-facing macros.
>
> 2. Converting the variadic macros for the documented APIs to use
> explicit arguments.
>
> No functional change intended.
>
> Signed-off-by: Marco Elver <el...@google.com>

+Cc Jon

I thought it was supposed to work because the kernel-doc scripts were at the
time taught by commit 51a7bf0238c2 ("scripts/kernel-doc: drop "_noprof" on
function prototypes") to handle _noprof. In the current form git grep finds:

tools/lib/python/kdoc/kdoc_parser.py: suffixes = [ '_noprof' ]
tools/lib/python/kdoc/xforms_lists.py: (KernRe("_noprof"), ""),

Doesn't it work for you then?

[Marco Elver]

Ah, I see. So it doesn't work anymore because we add the '_' prefix, too.

I guess the question is if we want to proliferate more kdoc parser
special cases, or just move the docs to the macros. The downside of
macros is that they lose the types in the displayed function
signature.

Preferences?