[PATCH 0/3] KMSAN: Move per-page metadata out of struct page

1 view
Skip to first unread message

Muchun Song

unread,
Nov 14, 2026, 5:42:15 AM (6 days ago) Nov 14
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
This series fixes a KMSAN metadata-loss bug triggered when boot-time
gigantic HugeTLB pages initialize their page descriptors after
kmsan_init_shadow(). KMSAN currently stores each page's shadow and origin
pointers in struct page, so the later descriptor initialization loses the
metadata mappings and leaks their backing pages.

Preserving the pointers around __init_single_page() would either add
KMSAN-specific save and restore logic to the HugeTLB loop or make the
generic memmap initializer retain sanitizer-private state. Moving
kmsan_init_shadow() after HugeTLB setup is not viable either. Its metadata
comes from memblock, while gigantic page descriptors are initialized later
in page_alloc_init_late(), after memblock teardown and after the KMSAN
runtime is enabled.

Moving the metadata to page_ext is therefore part of the fix, not a
standalone refactoring. page_ext storage is independent of struct page
contents, so later descriptor initialization cannot erase the metadata and
the HugeTLB and generic memmap paths remain unaware of KMSAN.

The first two patches provide the metadata-access and early-allocation
groundwork; the final patch moves the metadata to page_ext.

Muchun Song (3):
kmsan: encapsulate per-page metadata access
mm/page_ext: support early allocation for SPARSEMEM
kmsan: fix metadata loss during HugeTLB struct page reinitialization

include/linux/kmsan.h | 3 +
include/linux/mm_types.h | 12 ----
include/linux/page_ext.h | 27 +++-----
lib/Kconfig.kmsan | 1 +
mm/Makefile | 8 +++
mm/kmsan/shadow.c | 120 +++++++++++++++++++++++++---------
mm/mm_init.c | 11 +---
mm/page_ext.c | 136 ++++++++++++++++++++++++++++++---------
8 files changed, 219 insertions(+), 99 deletions(-)


base-commit: 6ec534d64f85e949a5d58070678dcd7711875a7c
--
2.54.0

Muchun Song

unread,
Nov 14, 2026, 5:42:29 AM (6 days ago) Nov 14
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN accesses the shadow and origin page pointers directly through
struct page. This spreads knowledge of the metadata storage across the
shadow implementation and makes changing that storage unnecessarily
invasive.

Introduce helpers that get and set both metadata page pointers together
and convert the existing users to them. Keeping the pair together also
lets callers avoid duplicate lookups once the storage moves out of struct
page.

No functional change is intended.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
mm/kmsan/shadow.c | 86 +++++++++++++++++++++++++++++++----------------
1 file changed, 57 insertions(+), 29 deletions(-)

diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 0c88d89bf0d6..8922d1ba9a0a 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -20,29 +20,43 @@
#include "../vmalloc.h"
#include "kmsan.h"

-#define shadow_page_for(page) ((page)->kmsan_shadow)
+struct page_metadata {
+ struct page *shadow;
+ struct page *origin;
+};

-#define origin_page_for(page) ((page)->kmsan_origin)
+static inline struct page_metadata get_page_metadata(struct page *page)
+{
+ return (struct page_metadata) {
+ .shadow = page->kmsan_shadow,
+ .origin = page->kmsan_origin,
+ };
+}
+
+static inline void set_page_metadata(struct page *page, struct page_metadata meta)
+{
+ page->kmsan_shadow = meta.shadow;
+ page->kmsan_origin = meta.origin;
+}

-static void *shadow_ptr_for(struct page *page)
+static inline void *shadow_ptr_for(struct page_metadata meta)
{
- return page_address(shadow_page_for(page));
+ return page_address(meta.shadow);
}

-static void *origin_ptr_for(struct page *page)
+static inline void *origin_ptr_for(struct page_metadata meta)
{
- return page_address(origin_page_for(page));
+ return page_address(meta.origin);
}

-static bool page_has_metadata(struct page *page)
+static inline bool page_has_metadata(struct page_metadata meta)
{
- return shadow_page_for(page) && origin_page_for(page);
+ return meta.shadow && meta.origin;
}

static void set_no_shadow_origin_page(struct page *page)
{
- shadow_page_for(page) = NULL;
- origin_page_for(page) = NULL;
+ set_page_metadata(page, (struct page_metadata) {});
}

/*
@@ -124,6 +138,7 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
*/
void *kmsan_get_metadata(void *address, bool is_origin)
{
+ struct page_metadata meta;
u64 addr = (u64)address, off;
struct page *page;
void *ret;
@@ -142,28 +157,36 @@ void *kmsan_get_metadata(void *address, bool is_origin)
page = virt_to_page_or_null(address);
if (!page)
return NULL;
- if (!page_has_metadata(page))
+ meta = get_page_metadata(page);
+ if (!page_has_metadata(meta))
return NULL;
off = offset_in_page(addr);

- return (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) + off;
+ return (is_origin ? origin_ptr_for(meta) : shadow_ptr_for(meta)) + off;
}

void kmsan_copy_page_meta(struct page *dst, struct page *src)
{
+ struct page_metadata dst_meta, src_meta;
+
if (!kmsan_enabled || kmsan_in_runtime())
return;
- if (!dst || !page_has_metadata(dst))
+ if (!dst)
return;
- if (!src || !page_has_metadata(src)) {
+ dst_meta = get_page_metadata(dst);
+ if (!page_has_metadata(dst_meta))
+ return;
+ if (src)
+ src_meta = get_page_metadata(src);
+ if (!src || !page_has_metadata(src_meta)) {
kmsan_internal_unpoison_memory(page_address(dst), PAGE_SIZE,
/*checked*/ false);
return;
}

kmsan_enter_runtime();
- __memcpy(shadow_ptr_for(dst), shadow_ptr_for(src), PAGE_SIZE);
- __memcpy(origin_ptr_for(dst), origin_ptr_for(src), PAGE_SIZE);
+ __memcpy(shadow_ptr_for(dst_meta), shadow_ptr_for(src_meta), PAGE_SIZE);
+ __memcpy(origin_ptr_for(dst_meta), origin_ptr_for(src_meta), PAGE_SIZE);
kmsan_leave_runtime();
}
EXPORT_SYMBOL(kmsan_copy_page_meta);
@@ -171,19 +194,18 @@ EXPORT_SYMBOL(kmsan_copy_page_meta);
void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
{
bool initialized = (flags & __GFP_ZERO) || !kmsan_enabled;
- struct page *shadow, *origin;
+ struct page_metadata meta;
depot_stack_handle_t handle;
int pages = 1 << order;

if (!page)
return;

- shadow = shadow_page_for(page);
- origin = origin_page_for(page);
+ meta = get_page_metadata(page);

if (initialized) {
- __memset(page_address(shadow), 0, PAGE_SIZE * pages);
- __memset(page_address(origin), 0, PAGE_SIZE * pages);
+ __memset(shadow_ptr_for(meta), 0, PAGE_SIZE * pages);
+ __memset(origin_ptr_for(meta), 0, PAGE_SIZE * pages);
return;
}

@@ -191,7 +213,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
if (kmsan_in_runtime())
return;

- __memset(page_address(shadow), -1, PAGE_SIZE * pages);
+ __memset(shadow_ptr_for(meta), -1, PAGE_SIZE * pages);
kmsan_enter_runtime();
handle = kmsan_save_stack_with_flags(flags, /*extra_bits*/ 0);
kmsan_leave_runtime();
@@ -200,7 +222,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
* to just fill the origin pages with @handle.
*/
for (int i = 0; i < PAGE_SIZE * pages / sizeof(handle); i++)
- ((depot_stack_handle_t *)page_address(origin))[i] = handle;
+ ((depot_stack_handle_t *)origin_ptr_for(meta))[i] = handle;
}

void kmsan_free_page(struct page *page, unsigned int order)
@@ -238,8 +260,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
goto ret;
}
for (int i = 0; i < nr; i++) {
- s_pages[i] = shadow_page_for(pages[i]);
- o_pages[i] = origin_page_for(pages[i]);
+ struct page_metadata meta = get_page_metadata(pages[i]);
+
+ s_pages[i] = meta.shadow;
+ o_pages[i] = meta.origin;
}
prot = PAGE_KERNEL;

@@ -289,10 +313,12 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end)
page = virt_to_page_or_null((char *)start + addr);
shadow_p = virt_to_page((char *)shadow + addr);
set_no_shadow_origin_page(shadow_p);
- shadow_page_for(page) = shadow_p;
origin_p = virt_to_page((char *)origin + addr);
set_no_shadow_origin_page(origin_p);
- origin_page_for(page) = origin_p;
+ set_page_metadata(page, (struct page_metadata) {
+ .shadow = shadow_p,
+ .origin = origin_p,
+ });
}
}

@@ -302,7 +328,9 @@ void kmsan_setup_meta(struct page *page, struct page *shadow,
for (int i = 0; i < (1 << order); i++) {
set_no_shadow_origin_page(&shadow[i]);
set_no_shadow_origin_page(&origin[i]);
- shadow_page_for(&page[i]) = &shadow[i];
- origin_page_for(&page[i]) = &origin[i];
+ set_page_metadata(&page[i], (struct page_metadata) {
+ .shadow = &shadow[i],
+ .origin = &origin[i],
+ });
}
}
--
2.54.0

Muchun Song

unread,
Nov 14, 2026, 5:42:42 AM (6 days ago) Nov 14
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
Moving KMSAN's per-page metadata into page_ext requires page extensions
to be available when kmsan_init_shadow() runs, before memblock is freed.
FLATMEM page extensions are available by then, but SPARSEMEM normally
allocates them later from the page or vmalloc allocators.

Add need_early so clients can request SPARSEMEM section backing from
memblock during page_ext_init_early(). Use page_ext_init() as the common
late path for client callbacks.

KMSAN starts requesting early page extensions in the following patch.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
include/linux/page_ext.h | 26 +++--------
mm/mm_init.c | 11 +----
mm/page_ext.c | 99 ++++++++++++++++++++++++++++++----------
3 files changed, 82 insertions(+), 54 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index 79c53ec45dfa..3b66720ddb18 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -16,6 +16,8 @@ struct pglist_data;
* @size: The size of the client data within page_ext.
* @need: Function that returns true if client requires page_ext.
* @init: (optional) Called to initialize client once page_exts are allocated.
+ * @need_early: True if the client requires page_ext during early boot,
+ * before normal memory allocation is available.
* @need_shared_flags: True when client is using shared page_ext->flags
* field.
*
@@ -27,6 +29,7 @@ struct page_ext_operations {
size_t size;
bool (*need)(void);
void (*init)(void);
+ bool need_early;
bool need_shared_flags;
};

@@ -61,15 +64,10 @@ static inline bool early_page_ext_enabled(void)
return early_page_ext;
}

-#ifdef CONFIG_SPARSEMEM
-static inline void page_ext_init_flatmem(void)
-{
-}
-extern void page_ext_init(void);
-static inline void page_ext_init_flatmem_late(void)
-{
-}
+void page_ext_init_early(void);
+void page_ext_init(void);

+#ifdef CONFIG_SPARSEMEM
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
/*
@@ -79,12 +77,6 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
return next_pfn % PAGES_PER_SECTION;
}
#else
-extern void page_ext_init_flatmem(void);
-extern void page_ext_init_flatmem_late(void);
-static inline void page_ext_init(void)
-{
-}
-
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
return true;
@@ -205,11 +197,7 @@ static inline void page_ext_init(void)
{
}

-static inline void page_ext_init_flatmem_late(void)
-{
-}
-
-static inline void page_ext_init_flatmem(void)
+static inline void page_ext_init_early(void)
{
}

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 2367af05570b..35cf799971c3 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2707,11 +2707,7 @@ void __init mm_core_init(void)
build_all_zonelists(NULL);
page_alloc_init_cpuhp();
alloc_tag_sec_init();
- /*
- * page_ext requires contiguous pages,
- * bigger than MAX_PAGE_ORDER unless SPARSEMEM.
- */
- page_ext_init_flatmem();
+ page_ext_init_early();
mem_debugging_and_hardening_init();
kfence_alloc_pool_and_metadata();
report_meminit();
@@ -2727,11 +2723,6 @@ void __init mm_core_init(void)
memblock_free_all();
mem_init();
kmem_cache_init();
- /*
- * page_owner must be initialized after buddy is ready, and also after
- * slab is ready so that stack_depot_init() works properly
- */
- page_ext_init_flatmem_late();
kmemleak_init();
ptlock_cache_init();
pgtable_cache_init();
diff --git a/mm/page_ext.c b/mm/page_ext.c
index b679a8c1f7d7..8578f52a32db 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/slab.h>

/*
* struct page extension
@@ -135,7 +136,12 @@ static bool __init invoke_need_callbacks(void)
if (page_ext_ops[i]->need()) {
page_ext_ops[i]->offset = page_ext_size;
page_ext_size += page_ext_ops[i]->size;
- need = true;
+ /*
+ * page_ext requires contiguous pages, bigger than
+ * MAX_PAGE_ORDER unless SPARSEMEM.
+ */
+ need |= !IS_ENABLED(CONFIG_SPARSEMEM) ||
+ page_ext_ops[i]->need_early;
}
}

@@ -159,7 +165,7 @@ static inline struct page_ext *get_entry(void *base, unsigned long index)
}

#ifndef CONFIG_SPARSEMEM
-void __init page_ext_init_flatmem_late(void)
+void __init page_ext_init(void)
{
invoke_init_callbacks();
}
@@ -217,9 +223,8 @@ static int __init alloc_node_page_ext(int nid)
return 0;
}

-void __init page_ext_init_flatmem(void)
+void __init page_ext_init_early(void)
{
-
int nid, fail;

if (!invoke_need_callbacks())
@@ -262,11 +267,19 @@ static struct page_ext *lookup_page_ext(const struct page *page)
return get_entry(page_ext, pfn);
}

-static void *__meminit alloc_page_ext(size_t size, int nid)
+static __ref void *alloc_page_ext(size_t size, int nid)
{
gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
void *addr = NULL;

+ if (!slab_is_available()) {
+ addr = memblock_alloc_try_nid(size, PAGE_SIZE, 0,
+ MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+ if (addr)
+ memmap_boot_pages_add(size / PAGE_SIZE);
+ return addr;
+ }
+
addr = alloc_pages_exact_nid(nid, size, flags);
if (addr)
kmemleak_alloc(addr, size, 1, flags);
@@ -274,7 +287,7 @@ static void *__meminit alloc_page_ext(size_t size, int nid)
addr = vzalloc_node(size, nid);

if (addr)
- memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));
+ memmap_pages_add(size / PAGE_SIZE);

return addr;
}
@@ -290,7 +303,7 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
if (section->page_ext)
return 0;

- table_size = page_ext_size * PAGES_PER_SECTION;
+ table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
base = alloc_page_ext(table_size, nid);

/*
@@ -317,20 +330,27 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)

static void free_page_ext(void *addr)
{
- size_t table_size;
+ size_t table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
+ long nr_pages = table_size / PAGE_SIZE;
struct page *page;

- table_size = page_ext_size * PAGES_PER_SECTION;
- memmap_pages_add(-1L * (DIV_ROUND_UP(table_size, PAGE_SIZE)));
-
if (is_vmalloc_addr(addr)) {
+ memmap_pages_add(-nr_pages);
vfree(addr);
- } else {
- page = virt_to_page(addr);
- BUG_ON(PageReserved(page));
- kmemleak_free(addr);
- free_pages_exact(addr, table_size);
+ return;
+ }
+
+ page = virt_to_page(addr);
+ if (PageReserved(page)) {
+ memmap_boot_pages_add(-nr_pages);
+ kmemleak_free_part_phys(__pa(addr), table_size);
+ free_reserved_area(addr, addr + table_size, -1, "page_ext");
+ return;
}
+
+ memmap_pages_add(-nr_pages);
+ kmemleak_free(addr);
+ free_pages_exact(addr, table_size);
}

static void __free_page_ext(unsigned long pfn)
@@ -444,14 +464,28 @@ static int __meminit page_ext_callback(struct notifier_block *self,
return notifier_from_errno(ret);
}

-void __init page_ext_init(void)
+static int __init page_ext_init_sections_early(void)
+{
+ unsigned long start, end, pfn;
+ int i, nid;
+
+ for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) {
+ for (pfn = start; pfn < end; pfn = SECTION_ALIGN_UP(pfn + 1)) {
+ if (!pfn_valid(pfn))
+ continue;
+ if (init_section_page_ext(pfn, nid))
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+static int __init page_ext_init_sections(void)
{
unsigned long pfn;
int nid;

- if (!invoke_need_callbacks())
- return;
-
for_each_node_state(nid, N_MEMORY) {
unsigned long start_pfn, end_pfn;

@@ -476,17 +510,32 @@ void __init page_ext_init(void)
if (pfn_to_nid(pfn) != nid)
continue;
if (init_section_page_ext(pfn, nid))
- goto oom;
+ return -ENOMEM;
cond_resched();
}
}
+
+ return 0;
+}
+
+void __init page_ext_init_early(void)
+{
+ if (!invoke_need_callbacks())
+ return;
+ if (page_ext_init_sections_early())
+ panic("Out of memory");
+}
+
+void __init page_ext_init(void)
+{
+ if (!page_ext_size)
+ return;
+ if (page_ext_init_sections())
+ panic("Out of memory");
+
hotplug_memory_notifier(page_ext_callback, DEFAULT_CALLBACK_PRI);
pr_info("allocated %ld bytes of page_ext\n", total_usage);
invoke_init_callbacks();
- return;
-
-oom:
- panic("Out of memory");
}

#endif
--
2.54.0

Muchun Song

unread,
Nov 14, 2026, 5:42:56 AM (6 days ago) Nov 14
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song, Sashiko
KMSAN stores each page's shadow and origin pointers in struct page. Late
initialization of boot-time gigantic HugeTLB pages rewrites page
descriptors after kmsan_init_shadow() has installed the metadata, losing
the mappings and leaking the metadata pages.

Store the pointers in page_ext, which survives struct page
reinitialization, and request early page extensions so they are available
when kmsan_init_shadow() runs.

Treat page_ext as part of the KMSAN runtime: use a raw lookup under notrace
preemption and compile page_ext without sanitizer, tracing, branch
profiling, or coverage instrumentation to avoid recursive metadata lookup.

Reported-by: Sashiko <sashi...@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-ac161ba4-30cc-4bd3-b93c-ab7af8a6f64f
Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song <songm...@bytedance.com>
---
include/linux/kmsan.h | 3 +++
include/linux/mm_types.h | 12 -----------
include/linux/page_ext.h | 1 +
lib/Kconfig.kmsan | 1 +
mm/Makefile | 8 +++++++
mm/kmsan/shadow.c | 46 ++++++++++++++++++++++++++++++++++------
mm/page_ext.c | 37 ++++++++++++++++++++++++++++----
7 files changed, 86 insertions(+), 22 deletions(-)

diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h
index 7da9fd506b39..d7498abe7d42 100644
--- a/include/linux/kmsan.h
+++ b/include/linux/kmsan.h
@@ -15,6 +15,7 @@
#include <linux/types.h>

struct page;
+struct page_ext_operations;
struct kmem_cache;
struct task_struct;
struct scatterlist;
@@ -22,6 +23,8 @@ struct urb;

#ifdef CONFIG_KMSAN

+extern struct page_ext_operations kmsan_ops;
+
/**
* kmsan_task_create() - Initialize KMSAN state for the task.
* @task: task to initialize.
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..dc73f9f5a51b 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -208,18 +208,6 @@ struct page {
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
int _last_cpupid;
#endif
-
-#ifdef CONFIG_KMSAN
- /*
- * KMSAN metadata for this page:
- * - shadow page: every bit indicates whether the corresponding
- * bit of the original page is initialized (0) or not (1);
- * - origin page: every 4 bytes contain an id of the stack trace
- * where the uninitialized value was created.
- */
- struct page *kmsan_shadow;
- struct page *kmsan_origin;
-#endif
} _struct_page_alignment;

/*
diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index 3b66720ddb18..4bca34450356 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -87,6 +87,7 @@ extern struct page_ext *page_ext_get(const struct page *page);
extern struct page_ext *page_ext_from_phys(phys_addr_t phys);
extern void page_ext_put(struct page_ext *page_ext);
extern struct page_ext *page_ext_lookup(unsigned long pfn);
+struct page_ext *page_ext_lookup_raw(unsigned long pfn);

static inline void *page_ext_data(struct page_ext *page_ext,
struct page_ext_operations *ops)
diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan
index cae1ddcc18e1..015e85c70793 100644
--- a/lib/Kconfig.kmsan
+++ b/lib/Kconfig.kmsan
@@ -10,6 +10,7 @@ config KMSAN
depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER
depends on DEBUG_KERNEL && !KASAN && !KCSAN
depends on !PREEMPT_RT
+ select PAGE_EXTENSION
select STACKDEPOT
select STACKDEPOT_ALWAYS_INIT
help
diff --git a/mm/Makefile b/mm/Makefile
index 277b97f7f8d7..7655f87fe974 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -17,6 +17,14 @@ KCSAN_SANITIZE_page_alloc.o := n
# But enable explicit instrumentation for memory barriers.
KCSAN_INSTRUMENT_BARRIERS := y

+ifdef CONFIG_KMSAN
+KMSAN_SANITIZE_page_ext.o := n
+KCOV_INSTRUMENT_page_ext.o := n
+UBSAN_SANITIZE_page_ext.o := n
+CFLAGS_page_ext.o += -DDISABLE_BRANCH_PROFILING
+CFLAGS_REMOVE_page_ext.o = $(CC_FLAGS_FTRACE)
+endif
+
# These files are disabled because they produce non-interesting and/or
# flaky coverage that is not a function of syscall inputs. E.g. slab is out of
# free pages, or a task is migrated between nodes.
diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 8922d1ba9a0a..cab8896e1fc4 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -12,6 +12,7 @@
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
+#include <linux/page_ext.h>
#include <linux/slab.h>
#include <linux/smp.h>
#include <linux/stddef.h>
@@ -20,23 +21,56 @@
#include "../vmalloc.h"
#include "kmsan.h"

+/*
+ * KMSAN metadata for this page:
+ * - shadow page: every bit indicates whether the corresponding
+ * bit of the original page is initialized (0) or not (1);
+ * - origin page: every 4 bytes contain an id of the stack trace
+ * where the uninitialized value was created.
+ */
struct page_metadata {
struct page *shadow;
struct page *origin;
};

+static __init bool need_page_metadata(void)
+{
+ return true;
+}
+
+struct page_ext_operations kmsan_ops = {
+ .size = sizeof(struct page_metadata),
+ .need = need_page_metadata,
+ .need_early = true,
+};
+
static inline struct page_metadata get_page_metadata(struct page *page)
{
- return (struct page_metadata) {
- .shadow = page->kmsan_shadow,
- .origin = page->kmsan_origin,
- };
+ struct page_metadata meta = {};
+ struct page_ext *page_ext;
+
+ /*
+ * Avoid instrumented preemption debug and tracing hooks while resolving
+ * KMSAN metadata, as they would recurse back into metadata lookup.
+ */
+ preempt_disable_notrace();
+ page_ext = page_ext_lookup_raw(page_to_pfn(page));
+ if (page_ext)
+ meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
+ preempt_enable_no_resched_notrace();
+
+ return meta;
}

static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- page->kmsan_shadow = meta.shadow;
- page->kmsan_origin = meta.origin;
+ struct page_ext *page_ext;
+
+ preempt_disable_notrace();
+ page_ext = page_ext_lookup_raw(page_to_pfn(page));
+ if (page_ext)
+ *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops) = meta;
+ preempt_enable_no_resched_notrace();
}

static inline void *shadow_ptr_for(struct page_metadata meta)
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 8578f52a32db..caffe29421a1 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/kmsan.h>
#include <linux/slab.h>

/*
@@ -94,6 +95,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {
#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
&page_iommu_debug_ops,
#endif
+#ifdef CONFIG_KMSAN
+ &kmsan_ops,
+#endif
};

unsigned long page_ext_size;
@@ -170,13 +174,12 @@ void __init page_ext_init(void)
invoke_init_callbacks();
}

-static struct page_ext *lookup_page_ext(const struct page *page)
+static struct page_ext *__lookup_page_ext(const struct page *page)
{
unsigned long pfn = page_to_pfn(page);
unsigned long index;
struct page_ext *base;

- WARN_ON_ONCE(!rcu_read_lock_held());
base = NODE_DATA(page_to_nid(page))->node_page_ext;
/*
* The sanity checks the page allocator does upon freeing a
@@ -191,6 +194,12 @@ static struct page_ext *lookup_page_ext(const struct page *page)
return get_entry(base, index);
}

+static struct page_ext *lookup_page_ext(const struct page *page)
+{
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ return __lookup_page_ext(page);
+}
+
static int __init alloc_node_page_ext(int nid)
{
struct page_ext *base;
@@ -249,13 +258,12 @@ static bool page_ext_invalid(struct page_ext *page_ext)
return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID);
}

-static struct page_ext *lookup_page_ext(const struct page *page)
+static struct page_ext *__lookup_page_ext(const struct page *page)
{
unsigned long pfn = page_to_pfn(page);
struct mem_section *section = __pfn_to_section(pfn);
struct page_ext *page_ext = READ_ONCE(section->page_ext);

- WARN_ON_ONCE(!rcu_read_lock_held());
/*
* The sanity checks the page allocator does upon freeing a
* page can reach here before the page_ext arrays are
@@ -267,6 +275,12 @@ static struct page_ext *lookup_page_ext(const struct page *page)
return get_entry(page_ext, pfn);
}

+static struct page_ext *lookup_page_ext(const struct page *page)
+{
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ return __lookup_page_ext(page);
+}
+
static __ref void *alloc_page_ext(size_t size, int nid)
{
gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
@@ -553,6 +567,21 @@ struct page_ext *page_ext_lookup(unsigned long pfn)
return lookup_page_ext(pfn_to_page(pfn));
}

+/**
+ * page_ext_lookup_raw() - Lookup a page extension without RCU checking.
+ * @pfn: PFN of the page we're interested in.
+ *
+ * The caller must prevent the page extension from being freed, for example by
+ * disabling preemption. Unlike page_ext_lookup(), this function does not run
+ * lockdep checks and is safe to call from low-level sanitizer runtime code.
+ *
+ * Return: NULL if no page_ext exists for this page.
+ */
+struct page_ext *page_ext_lookup_raw(unsigned long pfn)
+{
+ return __lookup_page_ext(pfn_to_page(pfn));
+}
+
/**
* page_ext_get() - Get the extended information for a page.
* @page: The page we're interested in.
--
2.54.0

Chris Down

unread,
Nov 15, 2026, 1:23:47 AM (5 days ago) Nov 15
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org
Reviewed-by: Chris Down <ch...@chrisdown.name>

Chris Down

unread,
Nov 15, 2026, 1:23:50 AM (5 days ago) Nov 15
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org
Hi Muchun,

This looks good to me in general, I only have a few small things.

Muchun Song writes:
>- /*
>- * page_owner must be initialized after buddy is ready, and also after
>- * slab is ready so that stack_depot_init() works properly
>- */
>- page_ext_init_flatmem_late();

Not a big deal, but this means that on FLATMEM the init callbacks now run
after vmalloc_init() rather than straight after kmem_cache_init(). I can't
see anything that depends on the old place, but FLATMEM isn't what the
patch is about, so it'd be good to say in the changelog that this moves.

>+ if (!slab_is_available()) {
>+ addr = memblock_alloc_try_nid(size, PAGE_SIZE, 0,
>+ MEMBLOCK_ALLOC_ACCESSIBLE, nid);

Maybe this should pass __pa(MAX_DMA_ADDRESS) as min_addr, like
alloc_node_page_ext() does, so that the tables stay out of ZONE_DMA when
there's room elsewhere?

>+ page = virt_to_page(addr);
>+ if (PageReserved(page)) {
>+ memmap_boot_pages_add(-nr_pages);
>+ kmemleak_free_part_phys(__pa(addr), table_size);
>+ free_reserved_area(addr, addr + table_size, -1, "page_ext");
>+ return;

free_reserved_area() prints "Freeing page_ext memory" when it's given a
name, and this runs once for every section, so offlining a few gigabytes
would put dozens of those in the log. I think it should pass NULL instead.

Did you manage to get this branch to run, by the way? KMSAN is the only
user of need_early, and from a quick check offlining a memory block on a
KMSAN kernel fails with EBUSY, presumably because the metadata pages can't
be migrated, so I couldn't exercise it myself.

Once the changelog and the log message are fixed, please feel free to add:

Reviewed-by: Chris Down <ch...@chrisdown.name>

Thanks!

Chris

Chris Down

unread,
Nov 15, 2026, 1:23:53 AM (5 days ago) Nov 15
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org, Sashiko
Hi Muchun,

I can reproduce the crash and the series fixes it, and I think page_ext is
the right place for these pointers. I do have a few comments, though.

Muchun Song writes:
>- struct page *kmsan_shadow;
>- struct page *kmsan_origin;

This takes struct page from 80 bytes to 64 on a KMSAN kernel, which means
that hugetlb_vmemmap_optimizable_size() no longer returns 0 and HVO can be
used there for the first time. From a quick check with
hugetlb_free_vmemmap=on it seems to work, but nobody would guess that from
the subject, so I think the changelog should mention it.

Documentation/dev-tools/kmsan.rst also still describes the two pointers as
members of struct page, under "Metadata allocation", so that needs
updating as well.

>+ifdef CONFIG_KMSAN
>+KMSAN_SANITIZE_page_ext.o := n
>+KCOV_INSTRUMENT_page_ext.o := n
>+UBSAN_SANITIZE_page_ext.o := n
>+CFLAGS_page_ext.o += -DDISABLE_BRANCH_PROFILING
>+CFLAGS_REMOVE_page_ext.o = $(CC_FLAGS_FTRACE)
>+endif

This takes the instrumentation away from all of page_ext.c, when the only
thing that needs it is the lookup. page_ext_size is already visible in
page_ext.h for page_ext_next(), so I think page_ext_lookup_raw() could be
a static inline there instead. That way it gets compiled into
mm/kmsan/shadow.c with the right flags already and needs none of the
finagling above, and it saves a call on a very hot path as well.

>+ preempt_disable_notrace();
>+ page_ext = page_ext_lookup_raw(page_to_pfn(page));
>+ if (page_ext)
>+ meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
>+ preempt_enable_no_resched_notrace();

This is quite a lot more work than the two loads from struct page that it
replaces, and kmsan_get_shadow_origin_ptr() has to go through it twice
for every instrumented access to direct-mapped memory, because it calls
kmsan_get_metadata() once for the shadow and then again for the origin.
You might think that's a bit academic, but it does add up. For example, on
a 2 CPU VM, perf bench sched pipe -l 30000 goes from about 1.37s to about
1.58s with the series applied, perf bench syscall basic -l 300000 from
about 0.66s to about 0.75s, and a dd of 300000 blocks of 512 bytes from
/dev/zero to /dev/null from about 1.63s to about 1.90s.

1/3 says that keeping the pair together lets callers avoid duplicate
lookups, but the caller that matters most doesn't make use of that yet.
From a quick hack that has kmsan_get_shadow_origin_ptr() look the page up
once and take both pointers from the result, the same three come out at
about 1.38s, 0.62s and 1.54s, which is no worse than before the series.
Could you do something like that here?

Thanks!

Chris

Muchun Song

unread,
Nov 15, 2026, 1:45:21 AM (5 days ago) Nov 15
to Chris Down, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org


> On Nov 15, 2026, at 14:23, Chris Down <ch...@chrisdown.name> wrote:
>
> Hi Muchun,

Hi,

>
> This looks good to me in general, I only have a few small things.

Thanks for taking a look.

>
> Muchun Song writes:
>> - /*
>> - * page_owner must be initialized after buddy is ready, and also after
>> - * slab is ready so that stack_depot_init() works properly
>> - */
>> - page_ext_init_flatmem_late();
>
> Not a big deal, but this means that on FLATMEM the init callbacks now run
> after vmalloc_init() rather than straight after kmem_cache_init(). I can't
> see anything that depends on the old place, but FLATMEM isn't what the
> patch is about, so it'd be good to say in the changelog that this moves.

Sure. The new behavior is aligned with the non-FLATMEM case. It should be fine.

>
>> + if (!slab_is_available()) {
>> + addr = memblock_alloc_try_nid(size, PAGE_SIZE, 0,
>> + MEMBLOCK_ALLOC_ACCESSIBLE, nid);
>
> Maybe this should pass __pa(MAX_DMA_ADDRESS) as min_addr, like
> alloc_node_page_ext() does, so that the tables stay out of ZONE_DMA when
> there's room elsewhere?

Make sense.

>
>> + page = virt_to_page(addr);
>> + if (PageReserved(page)) {
>> + memmap_boot_pages_add(-nr_pages);
>> + kmemleak_free_part_phys(__pa(addr), table_size);
>> + free_reserved_area(addr, addr + table_size, -1, "page_ext");
>> + return;
>
> free_reserved_area() prints "Freeing page_ext memory" when it's given a
> name, and this runs once for every section, so offlining a few gigabytes
> would put dozens of those in the log. I think it should pass NULL instead.

Good point. I'll pass NULL in v2.

>
> Did you manage to get this branch to run, by the way? KMSAN is the only

No. I only verified that the KMSCN kernel boots successfully.

> user of need_early, and from a quick check offlining a memory block on a
> KMSAN kernel fails with EBUSY, presumably because the metadata pages can't
> be migrated, so I couldn't exercise it myself.

I'll exercise this code on this path and report back.

>
> Once the changelog and the log message are fixed, please feel free to add:
>
> Reviewed-by: Chris Down <ch...@chrisdown.name>

Thanks for your review.

Muchun

>
> Thanks!
>
> Chris

Mike Rapoport

unread,
Nov 15, 2026, 2:31:52 AM (5 days ago) Nov 15
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org
Hi Muchun,

> Moving KMSAN's per-page metadata into page_ext requires page extensions
> to be available when kmsan_init_shadow() runs, before memblock is freed.
> FLATMEM page extensions are available by then, but SPARSEMEM normally
> allocates them later from the page or vmalloc allocators.
>
> Add need_early so clients can request SPARSEMEM section backing from
> memblock during page_ext_init_early().

Can't we just always allocate page_ext for early sections from memblock?

> Use page_ext_init() as the common late path for client callbacks.
> KMSAN starts requesting early page extensions in the following patch.
>
> Signed-off-by: Muchun Song <songm...@bytedance.com>
> ---
> include/linux/page_ext.h | 26 +++--------
> mm/mm_init.c | 11 +----
> mm/page_ext.c | 99 ++++++++++++++++++++++++++++++----------
> 3 files changed, 82 insertions(+), 54 deletions(-)
>
> +static __ref void *alloc_page_ext(size_t size, int nid)
> {
> gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
> void *addr = NULL;
>
> + if (!slab_is_available()) {
> + addr = memblock_alloc_try_nid(size, PAGE_SIZE, 0,
> + MEMBLOCK_ALLOC_ACCESSIBLE, nid);
> + if (addr)
> + memmap_boot_pages_add(size / PAGE_SIZE);
> + return addr;
> + }
> +
> addr = alloc_pages_exact_nid(nid, size, flags);

And unrelated to this patch, do we really need to try an allocation from
buddy rather than always use vmalloc?

--
Sincerely yours,
Mike.

Muchun Song

unread,
Nov 15, 2026, 2:46:35 AM (5 days ago) Nov 15
to Chris Down, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org, Sashiko


> On Nov 15, 2026, at 14:23, Chris Down <ch...@chrisdown.name> wrote:
>
> Hi Muchun,

Hi,

>
> I can reproduce the crash and the series fixes it, and I think page_ext is
> the right place for these pointers. I do have a few comments, though.
>
> Muchun Song writes:
>> - struct page *kmsan_shadow;
>> - struct page *kmsan_origin;
>
> This takes struct page from 80 bytes to 64 on a KMSAN kernel, which means
> that hugetlb_vmemmap_optimizable_size() no longer returns 0 and HVO can be
> used there for the first time. From a quick check with
> hugetlb_free_vmemmap=on it seems to work, but nobody would guess that from
> the subject, so I think the changelog should mention it.

Make sense. Going forward, KMSAN can now work with HVO.

>
> Documentation/dev-tools/kmsan.rst also still describes the two pointers as
> members of struct page, under "Metadata allocation", so that needs
> updating as well.

Right.

>
>> +ifdef CONFIG_KMSAN
>> +KMSAN_SANITIZE_page_ext.o := n
>> +KCOV_INSTRUMENT_page_ext.o := n
>> +UBSAN_SANITIZE_page_ext.o := n
>> +CFLAGS_page_ext.o += -DDISABLE_BRANCH_PROFILING
>> +CFLAGS_REMOVE_page_ext.o = $(CC_FLAGS_FTRACE)
>> +endif
>
> This takes the instrumentation away from all of page_ext.c, when the only
> thing that needs it is the lookup. page_ext_size is already visible in

Absolutely right!

> page_ext.h for page_ext_next(), so I think page_ext_lookup_raw() could be
> a static inline there instead. That way it gets compiled into
> mm/kmsan/shadow.c with the right flags already and needs none of the
> finagling above, and it saves a call on a very hot path as well.

I'll include this change in the next version.

>
>> + preempt_disable_notrace();
>> + page_ext = page_ext_lookup_raw(page_to_pfn(page));
>> + if (page_ext)
>> + meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
>> + preempt_enable_no_resched_notrace();
>
> This is quite a lot more work than the two loads from struct page that it
> replaces, and kmsan_get_shadow_origin_ptr() has to go through it twice
> for every instrumented access to direct-mapped memory, because it calls
> kmsan_get_metadata() once for the shadow and then again for the origin.
> You might think that's a bit academic, but it does add up. For example, on
> a 2 CPU VM, perf bench sched pipe -l 30000 goes from about 1.37s to about
> 1.58s with the series applied, perf bench syscall basic -l 300000 from
> about 0.66s to about 0.75s, and a dd of 300000 blocks of 512 bytes from
> /dev/zero to /dev/null from about 1.63s to about 1.90s.

Good catch, and the numbers are hard to argue with.

>
> 1/3 says that keeping the pair together lets callers avoid duplicate
> lookups, but the caller that matters most doesn't make use of that yet.

Sorry, that's my oversight.

> From a quick hack that has kmsan_get_shadow_origin_ptr() look the page up
> once and take both pointers from the result, the same three come out at
> about 1.38s, 0.62s and 1.54s, which is no worse than before the series.
> Could you do something like that here?

That's a fair point. I'll amend patch 1 so that kmsan_get_shadow_origin_ptr()
uses the shadow/origin pair introduced there and obtains both pointers from a
single page metadata lookup.

Thanks very much for your review.

Muchun

>
> Thanks!
>
> Chris

Muchun Song

unread,
Nov 15, 2026, 3:10:21 AM (5 days ago) Nov 15
to Mike Rapoport, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org


> On Nov 15, 2026, at 15:31, Mike Rapoport <rp...@kernel.org> wrote:
>
> Hi Muchun,

Hi Mike,

>
>> Moving KMSAN's per-page metadata into page_ext requires page extensions
>> to be available when kmsan_init_shadow() runs, before memblock is freed.
>> FLATMEM page extensions are available by then, but SPARSEMEM normally
>> allocates them later from the page or vmalloc allocators.
>>
>> Add need_early so clients can request SPARSEMEM section backing from
>> memblock during page_ext_init_early().
>
> Can't we just always allocate page_ext for early sections from memblock?

That would be simpler. I checked the page_ext history but could not find a
documented reason for treating boot-time SPARSEMEM sections differently or
for avoiding memblock allocation there.

I agree with your suggestion. I will remove need_early and allocate
page_ext from memblock for boot-time SPARSEMEM sections whenever any client
needs it.

>
>> Use page_ext_init() as the common late path for client callbacks.
>> KMSAN starts requesting early page extensions in the following patch.
>>
>> Signed-off-by: Muchun Song <songm...@bytedance.com>
>> ---
>> include/linux/page_ext.h | 26 +++--------
>> mm/mm_init.c | 11 +----
>> mm/page_ext.c | 99 ++++++++++++++++++++++++++++++----------
>> 3 files changed, 82 insertions(+), 54 deletions(-)
>>
>> +static __ref void *alloc_page_ext(size_t size, int nid)
>> {
>> gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
>> void *addr = NULL;
>>
>> + if (!slab_is_available()) {
>> + addr = memblock_alloc_try_nid(size, PAGE_SIZE, 0,
>> + MEMBLOCK_ALLOC_ACCESSIBLE, nid);
>> + if (addr)
>> + memmap_boot_pages_add(size / PAGE_SIZE);
>> + return addr;
>> + }
>> +
>> addr = alloc_pages_exact_nid(nid, size, flags);
>
> And unrelated to this patch, do we really need to try an allocation from
> buddy rather than always use vmalloc?
>

From my perspective, I don't think it's necessary at all. The only real
benefit to allocating from the buddy might be better performance compared
to vmalloc, but do we actually care about performance here? It doesn't look
like this is on the hot path, so I don't think it's necessary. We can remove
it to simplify the code. I can make that a separate cleanup.

Thanks,
Muchun

> --
> Sincerely yours,
> Mike.


Muchun Song

unread,
Nov 15, 2026, 3:56:42 AM (5 days ago) Nov 15
to Chris Down, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Sang-Heon Jeon, Mostafa Saleh, Ketan, Usama Arif, kasa...@googlegroups.com, linux-...@vger.kernel.org, linu...@kvack.org
I exercised this branch with a non-KMSAN kernel by temporarily setting
page_owner_ops.need_early = true. Offlining a boot memory block succeeded,
nr_memmap_boot_pages decreased by 576 pages, and the kernel logged:

Freeing page_ext memory: 2304K

A subsequent kmemleak scan completed without reports or kernel warnings.

I could not exercise the branch with KMSAN enabled. During the
memblock-to-buddy handoff, kmsan_memblock_free_pages() retains the first
two of every three same-order blocks as shadow and origin backing, and
only releases the third block to the buddy allocator. The retained metadata
pages are not migratable. On this x86 setup, every ordinary boot memory
block therefore contains KMSAN metadata pages, and memory offlining fails
with -EBUSY before the MEM_OFFLINE notification is sent. Consequently,
page_ext_callback() never reaches this branch.

Thanks,
Muchun

Muchun Song

unread,
Nov 17, 2026, 7:07:05 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
This series fixes a KMSAN metadata-loss bug triggered when boot-time
gigantic HugeTLB pages reinitialize their page descriptors after
kmsan_init_shadow(). KMSAN currently stores each page's shadow and
origin pointers in struct page, so the later initialization drops the
metadata mappings and leaves their backing pages unreachable.

Preserving the pointers around the HugeTLB initialization would add
KMSAN-specific state handling to the memmap path. Moving
kmsan_init_shadow() later is not viable either, because it allocates
metadata from memblock while gigantic page descriptors are initialized
after memblock teardown.

Move the metadata pointers to page_ext instead. page_ext storage is
independent of struct page contents and is made available before
kmsan_init_shadow(), so later page-descriptor initialization cannot
erase the mappings.

The series first decouples KMSAN metadata access from its storage, then
makes page_ext safe and available during early boot, and finally moves
the metadata pointers. The move also shrinks struct page from 80 to 64
bytes on x86-64 KMSAN builds, allowing HugeTLB Vmemmap Optimization to
be used with KMSAN.

v2:
- Fetch shadow and origin through one masked metadata resolver
(suggested by Chris Down).
- Add a fix for pre-existing page_ext client misalignment.
- Use vmalloc for runtime SPARSEMEM page_ext backing
(suggested by Mike Rapoport).
- Unify page_ext allocation and initialization entry points and document
the later FLATMEM callback timing (noted by Chris Down).
- Allocate all boot-time SPARSEMEM page_ext from memblock
(suggested by Mike Rapoport).
- Keep memblock allocations above DMA and suppress per-section free
messages (suggested by Chris Down).
- Inline the unchecked page_ext lookup while keeping page_ext.c
instrumented (suggested by Chris Down).
- Update the KMSAN metadata documentation and describe HVO enablement
(suggested by Chris Down).
- Expand the series from three to seven patches.
- Drop all v1 Reviewed-by tags because the affected patches were
substantially reworked and split.

v1: https://lore.kernel.org/20261114104150.5...@bytedance.com/

Muchun Song (7):
kmsan: encapsulate per-page metadata access
mm/page_ext: fix client data alignment
mm/page_ext: always use vmalloc for section page extensions
mm/page_ext: use common allocation and initialization entry points
mm/page_ext: allocate page extensions early for SPARSEMEM
mm/page_ext: inline unchecked page extension lookup
kmsan: fix metadata loss during HugeTLB struct page reinitialization

Documentation/dev-tools/kmsan.rst | 8 +-
include/linux/kmsan.h | 2 +
include/linux/mm.h | 12 +-
include/linux/mm_types.h | 12 --
include/linux/page_ext.h | 61 +++++++--
lib/Kconfig.kmsan | 1 +
mm/kmsan/shadow.c | 199 +++++++++++++++++++--------
mm/mm_init.c | 11 +-
mm/page_ext.c | 218 ++++++++++++------------------
9 files changed, 286 insertions(+), 238 deletions(-)


base-commit: 67f3b04b277373438f17300e056306adbd9edf7b
--
2.54.0

Muchun Song

unread,
Nov 17, 2026, 7:07:11 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN directly accesses page->kmsan_shadow and page->kmsan_origin
throughout the shadow implementation. This couples these users to
storing the metadata pointers in struct page, requiring every access
site to change when the storage moves elsewhere.

Introduce helpers that get and set both metadata page pointers as a pair
and convert the existing users to them.

kmsan_get_shadow_origin_ptr() is the main metadata lookup path for
compiler-instrumented memory accesses and requires both pointers. For
page-backed addresses, resolve the backing page once and obtain the
metadata pair through one lookup instead of calling the single-metadata
path twice. This also avoids duplicate lookups after moving the metadata
storage out of struct page.

No functional change is intended.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- Use one page metadata lookup for shadow and origin pointers (suggested
by Chris Down)
---
mm/kmsan/shadow.c | 166 +++++++++++++++++++++++++++++++---------------
1 file changed, 111 insertions(+), 55 deletions(-)

diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 0c88d89bf0d6..300b48818af1 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -9,6 +9,7 @@

#include <asm/kmsan.h>
#include <asm/tlbflush.h>
+#include <linux/bits.h>
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
@@ -20,29 +21,46 @@
#include "../vmalloc.h"
#include "kmsan.h"

-#define shadow_page_for(page) ((page)->kmsan_shadow)
+#define KMSAN_META_MASK_SHADOW BIT(0)
+#define KMSAN_META_MASK_ORIGIN BIT(1)

-#define origin_page_for(page) ((page)->kmsan_origin)
+struct page_metadata {
+ struct page *shadow;
+ struct page *origin;
+};

-static void *shadow_ptr_for(struct page *page)
+static inline struct page_metadata get_page_metadata(struct page *page)
{
- return page_address(shadow_page_for(page));
+ return (struct page_metadata) {
+ .shadow = page->kmsan_shadow,
+ .origin = page->kmsan_origin,
+ };
}

-static void *origin_ptr_for(struct page *page)
+static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- return page_address(origin_page_for(page));
+ page->kmsan_shadow = meta.shadow;
+ page->kmsan_origin = meta.origin;
}

-static bool page_has_metadata(struct page *page)
+static inline void *shadow_ptr_for(struct page_metadata meta)
{
- return shadow_page_for(page) && origin_page_for(page);
+ return page_address(meta.shadow);
+}
+
+static inline void *origin_ptr_for(struct page_metadata meta)
+{
+ return page_address(meta.origin);
+}
+
+static inline bool page_has_metadata(struct page_metadata meta)
+{
+ return meta.shadow && meta.origin;
}

static void set_no_shadow_origin_page(struct page *page)
{
- shadow_page_for(page) = NULL;
- origin_page_for(page) = NULL;
+ set_page_metadata(page, (struct page_metadata) {});
}

/*
@@ -79,11 +97,56 @@ static struct page *virt_to_page_or_null(void *vaddr)
return NULL;
}

+static __always_inline
+struct shadow_origin_ptr __kmsan_get_metadata(void *address, unsigned int mask)
+{
+ struct shadow_origin_ptr ret = {};
+ struct page_metadata meta;
+ struct page *page;
+ u64 addr = (u64)address;
+ u64 origin_addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE);
+
+ if (kmsan_internal_is_vmalloc_addr(address) ||
+ kmsan_internal_is_module_addr(address)) {
+ if (mask & KMSAN_META_MASK_SHADOW)
+ ret.shadow = (void *)vmalloc_meta(address, KMSAN_META_SHADOW);
+ if (mask & KMSAN_META_MASK_ORIGIN)
+ ret.origin = (void *)vmalloc_meta((void *)origin_addr,
+ KMSAN_META_ORIGIN);
+ return ret;
+ }
+
+ if (mask & KMSAN_META_MASK_SHADOW) {
+ ret.shadow = arch_kmsan_get_meta_or_null(address, KMSAN_META_SHADOW);
+ if (!ret.shadow)
+ goto lookup_page;
+ }
+ if (mask & KMSAN_META_MASK_ORIGIN)
+ ret.origin = arch_kmsan_get_meta_or_null((void *)origin_addr,
+ KMSAN_META_ORIGIN);
+ if (ret.shadow || ret.origin)
+ return ret;
+
+lookup_page:
+ page = virt_to_page_or_null(address);
+ if (!page)
+ return ret;
+ meta = get_page_metadata(page);
+ if (!page_has_metadata(meta))
+ return ret;
+
+ if (mask & KMSAN_META_MASK_SHADOW)
+ ret.shadow = shadow_ptr_for(meta) + offset_in_page(addr);
+ if (mask & KMSAN_META_MASK_ORIGIN)
+ ret.origin = origin_ptr_for(meta) + offset_in_page(origin_addr);
+
+ return ret;
+}
+
struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
bool store)
{
struct shadow_origin_ptr ret;
- void *shadow;

/*
* Even if we redirect this memory access to the dummy page, it will
@@ -95,12 +158,10 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
goto return_dummy;

KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(address, size));
- shadow = kmsan_get_metadata(address, KMSAN_META_SHADOW);
- if (!shadow)
+ ret = __kmsan_get_metadata(address, KMSAN_META_MASK_SHADOW |
+ KMSAN_META_MASK_ORIGIN);
+ if (!ret.shadow)
goto return_dummy;
-
- ret.shadow = shadow;
- ret.origin = kmsan_get_metadata(address, KMSAN_META_ORIGIN);
return ret;

return_dummy:
@@ -124,46 +185,36 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
*/
void *kmsan_get_metadata(void *address, bool is_origin)
{
- u64 addr = (u64)address, off;
- struct page *page;
- void *ret;
-
- if (is_origin)
- addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE);
- address = (void *)addr;
- if (kmsan_internal_is_vmalloc_addr(address) ||
- kmsan_internal_is_module_addr(address))
- return (void *)vmalloc_meta(address, is_origin);
-
- ret = arch_kmsan_get_meta_or_null(address, is_origin);
- if (ret)
- return ret;
-
- page = virt_to_page_or_null(address);
- if (!page)
- return NULL;
- if (!page_has_metadata(page))
- return NULL;
- off = offset_in_page(addr);
+ struct shadow_origin_ptr ret;
+ unsigned int mask;

- return (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) + off;
+ mask = is_origin ? KMSAN_META_MASK_ORIGIN : KMSAN_META_MASK_SHADOW;
+ ret = __kmsan_get_metadata(address, mask);
+ return is_origin ? ret.origin : ret.shadow;
@@ -171,19 +222,18 @@ EXPORT_SYMBOL(kmsan_copy_page_meta);
void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
{
bool initialized = (flags & __GFP_ZERO) || !kmsan_enabled;
- struct page *shadow, *origin;
+ struct page_metadata meta;
depot_stack_handle_t handle;
int pages = 1 << order;

if (!page)
return;

- shadow = shadow_page_for(page);
- origin = origin_page_for(page);
+ meta = get_page_metadata(page);

if (initialized) {
- __memset(page_address(shadow), 0, PAGE_SIZE * pages);
- __memset(page_address(origin), 0, PAGE_SIZE * pages);
+ __memset(shadow_ptr_for(meta), 0, PAGE_SIZE * pages);
+ __memset(origin_ptr_for(meta), 0, PAGE_SIZE * pages);
return;
}

@@ -191,7 +241,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
if (kmsan_in_runtime())
return;

- __memset(page_address(shadow), -1, PAGE_SIZE * pages);
+ __memset(shadow_ptr_for(meta), -1, PAGE_SIZE * pages);
kmsan_enter_runtime();
handle = kmsan_save_stack_with_flags(flags, /*extra_bits*/ 0);
kmsan_leave_runtime();
@@ -200,7 +250,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
* to just fill the origin pages with @handle.
*/
for (int i = 0; i < PAGE_SIZE * pages / sizeof(handle); i++)
- ((depot_stack_handle_t *)page_address(origin))[i] = handle;
+ ((depot_stack_handle_t *)origin_ptr_for(meta))[i] = handle;
}

void kmsan_free_page(struct page *page, unsigned int order)
@@ -238,8 +288,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
goto ret;
}
for (int i = 0; i < nr; i++) {
- s_pages[i] = shadow_page_for(pages[i]);
- o_pages[i] = origin_page_for(pages[i]);
+ struct page_metadata meta = get_page_metadata(pages[i]);
+
+ s_pages[i] = meta.shadow;
+ o_pages[i] = meta.origin;
}
prot = PAGE_KERNEL;

@@ -289,10 +341,12 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end)
page = virt_to_page_or_null((char *)start + addr);
shadow_p = virt_to_page((char *)shadow + addr);
set_no_shadow_origin_page(shadow_p);
- shadow_page_for(page) = shadow_p;
origin_p = virt_to_page((char *)origin + addr);
set_no_shadow_origin_page(origin_p);
- origin_page_for(page) = origin_p;
+ set_page_metadata(page, (struct page_metadata) {
+ .shadow = shadow_p,
+ .origin = origin_p,
+ });
}
}

@@ -302,7 +356,9 @@ void kmsan_setup_meta(struct page *page, struct page *shadow,

Muchun Song

unread,
Nov 17, 2026, 7:07:19 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
page_ext lays out client payloads consecutively without aligning their
offsets or the final entry size. This happened to work while the active
payload sizes kept the stride naturally aligned, but IOMMU debug page
allocation added a four-byte payload after data requiring eight-byte
alignment.

When page owner and IOMMU debug page allocation are both enabled on
x86-64, struct page_ext, struct page_owner, and the IOMMU payload use
8, 64, and 4 bytes respectively. This produces a 76-byte entry stride,
leaving struct page_ext and page_owner data in every other entry only
four-byte aligned instead of eight-byte aligned.

Derive a suitable alignment for each payload from the lowest set bit of
its size, capped at unsigned long alignment. Align each client offset
and round the final entry size to the largest active alignment so every
entry preserves the alignment required by all active clients.

Fixes: 466ae6978a5b ("iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC")
Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- New patch to fix the existing page_ext client alignment bug
---
mm/page_ext.c | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)

diff --git a/mm/page_ext.c b/mm/page_ext.c
index b679a8c1f7d7..dbc1bfa258f1 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -116,8 +116,22 @@ static int __init setup_early_page_ext(char *str)
}
early_param("early_page_ext", setup_early_page_ext);

+static inline size_t page_ext_data_alignment(size_t size)
+{
+ if (!size)
+ return 1;
+
+ /*
+ * A type's size is a multiple of its natural alignment. The lowest
+ * set bit of the size is therefore at least that alignment. Cap it
+ * at unsigned long alignment to avoid excessive per-page padding.
+ */
+ return min_t(size_t, 1UL << __ffs(size), __alignof__(unsigned long));
+}
+
static bool __init invoke_need_callbacks(void)
{
+ size_t max_align = 1;
int i;
int entries = ARRAY_SIZE(page_ext_ops);
bool need = false;
@@ -126,6 +140,7 @@ static bool __init invoke_need_callbacks(void)
if (page_ext_ops[i]->need()) {
if (page_ext_ops[i]->need_shared_flags) {
page_ext_size = sizeof(struct page_ext);
+ max_align = __alignof__(struct page_ext);
break;
}
}
@@ -133,11 +148,17 @@ static bool __init invoke_need_callbacks(void)

for (i = 0; i < entries; i++) {
if (page_ext_ops[i]->need()) {
+ size_t align;
+
+ align = page_ext_data_alignment(page_ext_ops[i]->size);
+ page_ext_size = ALIGN(page_ext_size, align);
page_ext_ops[i]->offset = page_ext_size;
page_ext_size += page_ext_ops[i]->size;
+ max_align = max(max_align, align);
need = true;
}
}
+ page_ext_size = ALIGN(page_ext_size, max_align);

return need;
}
--
2.54.0

Muchun Song

unread,
Nov 17, 2026, 7:07:25 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
SPARSEMEM currently tries to allocate each section's page extensions
from the buddy allocator before falling back to vmalloc. Boot-time
section allocation happens after vmalloc is available, and memory
hotplug allocations do not require physically contiguous backing.

Use vzalloc_node() directly and remove the buddy-specific allocation,
free, and kmemleak bookkeeping. This leaves one runtime backing type
and lets a later change add memblock as the distinct boot-time path
to simplify the code.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- Add a preparatory cleanup that always uses vmalloc for SPARSEMEM
section page extensions (suggested by Mike Rapoport)
---
mm/page_ext.c | 21 +++------------------
1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/mm/page_ext.c b/mm/page_ext.c
index dbc1bfa258f1..3b581242fd52 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -285,15 +285,9 @@ static struct page_ext *lookup_page_ext(const struct page *page)

static void *__meminit alloc_page_ext(size_t size, int nid)
{
- gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
- void *addr = NULL;
-
- addr = alloc_pages_exact_nid(nid, size, flags);
- if (addr)
- kmemleak_alloc(addr, size, 1, flags);
- else
- addr = vzalloc_node(size, nid);
+ void *addr;

+ addr = vzalloc_node(size, nid);
if (addr)
memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));

@@ -339,19 +333,10 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
static void free_page_ext(void *addr)
{
size_t table_size;
- struct page *page;

table_size = page_ext_size * PAGES_PER_SECTION;
memmap_pages_add(-1L * (DIV_ROUND_UP(table_size, PAGE_SIZE)));
-
- if (is_vmalloc_addr(addr)) {
- vfree(addr);
- } else {
- page = virt_to_page(addr);
- BUG_ON(PageReserved(page));
- kmemleak_free(addr);
- free_pages_exact(addr, table_size);
- }
+ vfree(addr);
}

static void __free_page_ext(unsigned long pfn)
--
2.54.0

Muchun Song

unread,
Nov 17, 2026, 7:07:32 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
FLATMEM uses separate page_ext_init_flatmem() and
page_ext_init_flatmem_late() entry points, while SPARSEMEM performs its
work through page_ext_init(). This exposes the memory model in the
generic boot sequence and obscures the allocation and client
initialization phases.

Rename the early entry point to page_ext_alloc() and use
page_ext_init() for client callbacks in both memory models. SPARSEMEM
keeps its existing late allocation behavior for now; its
page_ext_alloc() remains an empty configuration stub.

This also moves FLATMEM client callbacks from immediately after
kmem_cache_init() to after vmalloc_init(). The callbacks still run
after the buddy and slab allocators are available, as required by
page_owner.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- Split the common allocation and initialization entry points into a
preparatory patch
- Document the later FLATMEM client callback timing (noted by Chris
Down)
---
include/linux/page_ext.h | 20 +++++---------------
mm/mm_init.c | 7 +------
mm/page_ext.c | 5 ++---
3 files changed, 8 insertions(+), 24 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index 79c53ec45dfa..911a84bbb3db 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -62,13 +62,10 @@ static inline bool early_page_ext_enabled(void)
}

#ifdef CONFIG_SPARSEMEM
-static inline void page_ext_init_flatmem(void)
-{
-}
-extern void page_ext_init(void);
-static inline void page_ext_init_flatmem_late(void)
+static inline void page_ext_alloc(void)
{
}
+void page_ext_init(void);

static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
@@ -79,11 +76,8 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
return next_pfn % PAGES_PER_SECTION;
}
#else
-extern void page_ext_init_flatmem(void);
-extern void page_ext_init_flatmem_late(void);
-static inline void page_ext_init(void)
-{
-}
+void page_ext_alloc(void);
+void page_ext_init(void);

static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
@@ -205,11 +199,7 @@ static inline void page_ext_init(void)
{
}

-static inline void page_ext_init_flatmem_late(void)
-{
-}
-
-static inline void page_ext_init_flatmem(void)
+static inline void page_ext_alloc(void)
{
}

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 2367af05570b..7344e7cb8196 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2711,7 +2711,7 @@ void __init mm_core_init(void)
* page_ext requires contiguous pages,
* bigger than MAX_PAGE_ORDER unless SPARSEMEM.
*/
- page_ext_init_flatmem();
+ page_ext_alloc();
mem_debugging_and_hardening_init();
kfence_alloc_pool_and_metadata();
report_meminit();
@@ -2727,11 +2727,6 @@ void __init mm_core_init(void)
memblock_free_all();
mem_init();
kmem_cache_init();
- /*
- * page_owner must be initialized after buddy is ready, and also after
- * slab is ready so that stack_depot_init() works properly
- */
- page_ext_init_flatmem_late();
kmemleak_init();
ptlock_cache_init();
pgtable_cache_init();
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 3b581242fd52..b41d4a957bc6 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -180,7 +180,7 @@ static inline struct page_ext *get_entry(void *base, unsigned long index)
}

#ifndef CONFIG_SPARSEMEM
-void __init page_ext_init_flatmem_late(void)
+void __init page_ext_init(void)
{
invoke_init_callbacks();
}
@@ -238,9 +238,8 @@ static int __init alloc_node_page_ext(int nid)
return 0;
}

-void __init page_ext_init_flatmem(void)
+void __init page_ext_alloc(void)
{
-
int nid, fail;

if (!invoke_need_callbacks())
--
2.54.0

Muchun Song

unread,
Nov 17, 2026, 7:07:40 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN needs page extensions when kmsan_init_shadow() installs per-page
metadata, before memblock_free_all() releases memory to the buddy
allocator. FLATMEM makes page extensions available by this point,
whereas SPARSEMEM initializes them later in boot.

Allocate boot-time SPARSEMEM page extensions from memblock to match
FLATMEM's initialization timing. This gives early clients the same
page_ext availability guarantee across both memory models, while runtime
memory hotplug continues to use vmalloc.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- Allocate all boot-time SPARSEMEM page extensions from memblock
(suggested by Mike Rapoport)
- Keep memblock allocations above DMA and suppress per-section free
messages (suggested by Chris Down)
---
include/linux/page_ext.h | 9 +--
mm/mm_init.c | 4 --
mm/page_ext.c | 119 +++++++++++++++++++--------------------
3 files changed, 60 insertions(+), 72 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index 911a84bbb3db..d00320106a58 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -61,12 +61,10 @@ static inline bool early_page_ext_enabled(void)
return early_page_ext;
}

-#ifdef CONFIG_SPARSEMEM
-static inline void page_ext_alloc(void)
-{
-}
+void page_ext_alloc(void);
void page_ext_init(void);

+#ifdef CONFIG_SPARSEMEM
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
/*
@@ -76,9 +74,6 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
return next_pfn % PAGES_PER_SECTION;
}
#else
-void page_ext_alloc(void);
-void page_ext_init(void);
-
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
return true;
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 7344e7cb8196..baa2681857eb 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2707,10 +2707,6 @@ void __init mm_core_init(void)
build_all_zonelists(NULL);
page_alloc_init_cpuhp();
alloc_tag_sec_init();
- /*
- * page_ext requires contiguous pages,
- * bigger than MAX_PAGE_ORDER unless SPARSEMEM.
- */
page_ext_alloc();
mem_debugging_and_hardening_init();
kfence_alloc_pool_and_metadata();
diff --git a/mm/page_ext.c b/mm/page_ext.c
index b41d4a957bc6..2910727e99e2 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/slab.h>

/*
* struct page extension
@@ -180,11 +181,6 @@ static inline struct page_ext *get_entry(void *base, unsigned long index)
}

#ifndef CONFIG_SPARSEMEM
-void __init page_ext_init(void)
-{
- invoke_init_callbacks();
-}
-
static struct page_ext *lookup_page_ext(const struct page *page)
{
unsigned long pfn = page_to_pfn(page);
@@ -238,24 +234,16 @@ static int __init alloc_node_page_ext(int nid)
return 0;
}

-void __init page_ext_alloc(void)
+static int __init __page_ext_alloc(void)
{
- int nid, fail;
-
- if (!invoke_need_callbacks())
- return;
+ int nid;

- for_each_online_node(nid) {
- fail = alloc_node_page_ext(nid);
- if (fail)
- goto fail;
+ for_each_online_node(nid) {
+ if (alloc_node_page_ext(nid))
+ return -ENOMEM;
}
- pr_info("allocated %ld bytes of page_ext\n", total_usage);
- return;

-fail:
- pr_crit("allocation of page_ext failed.\n");
- panic("Out of memory");
+ return 0;
}

#else /* CONFIG_SPARSEMEM */
@@ -282,13 +270,21 @@ static struct page_ext *lookup_page_ext(const struct page *page)
return get_entry(page_ext, pfn);
}

-static void *__meminit alloc_page_ext(size_t size, int nid)
+static __ref void *alloc_page_ext(size_t size, int nid)
{
void *addr;

+ if (!slab_is_available()) {
+ addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
+ MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+ if (addr)
+ memmap_boot_pages_add(size / PAGE_SIZE);
+ return addr;
+ }
+
addr = vzalloc_node(size, nid);
if (addr)
- memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));
+ memmap_pages_add(size / PAGE_SIZE);

return addr;
}
@@ -304,7 +300,7 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
if (section->page_ext)
return 0;

- table_size = page_ext_size * PAGES_PER_SECTION;
+ table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
base = alloc_page_ext(table_size, nid);

/*
@@ -331,11 +327,19 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)

static void free_page_ext(void *addr)
{
- size_t table_size;
+ size_t table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
+ long nr_pages = table_size / PAGE_SIZE;
+
+ if (is_vmalloc_addr(addr)) {
+ memmap_pages_add(-nr_pages);
+ vfree(addr);
+ return;
+ }

- table_size = page_ext_size * PAGES_PER_SECTION;
- memmap_pages_add(-1L * (DIV_ROUND_UP(table_size, PAGE_SIZE)));
- vfree(addr);
+ VM_WARN_ON_ONCE(!PageReserved(virt_to_page(addr)));
+ memmap_boot_pages_add(-nr_pages);
+ kmemleak_free_part_phys(__pa(addr), table_size);
+ free_reserved_area(addr, addr + table_size, -1, NULL);
}

static void __free_page_ext(unsigned long pfn)
@@ -449,52 +453,45 @@ static int __meminit page_ext_callback(struct notifier_block *self,
return notifier_from_errno(ret);
}

-void __init page_ext_init(void)
+static int __init __page_ext_alloc(void)
{
- unsigned long pfn;
- int nid;
-
- if (!invoke_need_callbacks())
- return;
-
- for_each_node_state(nid, N_MEMORY) {
- unsigned long start_pfn, end_pfn;
-
- start_pfn = node_start_pfn(nid);
- end_pfn = node_end_pfn(nid);
- /*
- * start_pfn and end_pfn may not be aligned to SECTION and the
- * page->flags of out of node pages are not initialized. So we
- * scan [start_pfn, the biggest section's pfn < end_pfn) here.
- */
- for (pfn = start_pfn; pfn < end_pfn;
- pfn = ALIGN(pfn + 1, PAGES_PER_SECTION)) {
+ unsigned long start, end, pfn;
+ int i, nid;

+ for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) {
+ for (pfn = start; pfn < end; pfn = SECTION_ALIGN_UP(pfn + 1)) {
if (!pfn_valid(pfn))
continue;
- /*
- * Nodes's pfns can be overlapping.
- * We know some arch can have a nodes layout such as
- * -------------pfn-------------->
- * N0 | N1 | N2 | N0 | N1 | N2|....
- */
- if (pfn_to_nid(pfn) != nid)
- continue;
if (init_section_page_ext(pfn, nid))
- goto oom;
- cond_resched();
+ return -ENOMEM;
}
}
- hotplug_memory_notifier(page_ext_callback, DEFAULT_CALLBACK_PRI);
- pr_info("allocated %ld bytes of page_ext\n", total_usage);
- invoke_init_callbacks();
- return;

-oom:
- panic("Out of memory");
+ return 0;
+}
+
+#endif
+
+void __init page_ext_alloc(void)
+{
+ if (!invoke_need_callbacks())
+ return;
+ if (__page_ext_alloc())
+ panic("Out of memory");
+
+ pr_info("allocated %ld bytes of page_ext\n", total_usage);
}

+void __init page_ext_init(void)
+{
+ if (!page_ext_size)
+ return;
+
+#ifdef CONFIG_SPARSEMEM
+ hotplug_memory_notifier(page_ext_callback, DEFAULT_CALLBACK_PRI);
#endif
+ invoke_init_callbacks();
+}

/**
* page_ext_lookup() - Lookup a page extension for a PFN.
--
2.54.0

Muchun Song

unread,
Nov 17, 2026, 7:07:46 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
The KMSAN runtime needs an unchecked page extension lookup that can be
inlined into its uninstrumented compilation unit while page_ext.c remains
instrumented.

Provide an inline lookup for this use case while keeping the existing RCU
checks for other callers.

Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- Add a preparatory patch that inlines the unchecked lookup (suggested
by Chris Down)
- Use a PFN interface to avoid architecture header-order dependencies
---
include/linux/page_ext.h | 46 ++++++++++++++++++++++++++++++
mm/page_ext.c | 60 +++-------------------------------------
2 files changed, 50 insertions(+), 56 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index d00320106a58..166f8be11564 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -64,7 +64,14 @@ static inline bool early_page_ext_enabled(void)
void page_ext_alloc(void);
void page_ext_init(void);

+static __always_inline struct page_ext *page_ext_entry(void *base, unsigned long index)
+{
+ return base + page_ext_size * index;
+}
+
#ifdef CONFIG_SPARSEMEM
+#define PAGE_EXT_INVALID 0x1
+
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
/*
@@ -73,11 +80,50 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
*/
return next_pfn % PAGES_PER_SECTION;
}
+
+static __always_inline bool page_ext_invalid(struct page_ext *page_ext)
+{
+ return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID);
+}
+
+static __always_inline struct page_ext *__page_ext_lookup(unsigned long pfn)
+{
+ struct mem_section *section = __pfn_to_section(pfn);
+ struct page_ext *page_ext = READ_ONCE(section->page_ext);
+
+ /*
+ * The sanity checks the page allocator does upon freeing a page can
+ * reach here before the page_ext arrays are allocated when feeding a
+ * range of pages to the allocator for the first time during bootup or
+ * memory hotplug.
+ */
+ if (page_ext_invalid(page_ext))
+ return NULL;
+ return page_ext_entry(page_ext, pfn);
+}
#else
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
return true;
}
+
+static __always_inline struct page_ext *__page_ext_lookup(unsigned long pfn)
+{
+ int nid = pfn_to_nid(pfn);
+ struct page_ext *base = NODE_DATA(nid)->node_page_ext;
+ unsigned long index;
+
+ /*
+ * The sanity checks the page allocator does upon freeing a page can
+ * reach here before the page_ext arrays are allocated when feeding a
+ * range of pages to the allocator for the first time during bootup or
+ * memory hotplug.
+ */
+ if (unlikely(!base))
+ return NULL;
+ index = pfn - round_down(node_start_pfn(nid), MAX_ORDER_NR_PAGES);
+ return page_ext_entry(base, index);
+}
#endif

extern struct page_ext *page_ext_get(const struct page *page);
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 2910727e99e2..0f31e2da2611 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -63,10 +63,6 @@
* can utilize this callback to initialize the state of it correctly.
*/

-#ifdef CONFIG_SPARSEMEM
-#define PAGE_EXT_INVALID (0x1)
-#endif
-
#if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT)
static bool need_page_idle(void)
{
@@ -175,33 +171,7 @@ static void __init invoke_init_callbacks(void)
}
}

-static inline struct page_ext *get_entry(void *base, unsigned long index)
-{
- return base + page_ext_size * index;
-}
-
#ifndef CONFIG_SPARSEMEM
-static struct page_ext *lookup_page_ext(const struct page *page)
-{
- unsigned long pfn = page_to_pfn(page);
- unsigned long index;
- struct page_ext *base;
-
- WARN_ON_ONCE(!rcu_read_lock_held());
- base = NODE_DATA(page_to_nid(page))->node_page_ext;
- /*
- * The sanity checks the page allocator does upon freeing a
- * page can reach here before the page_ext arrays are
- * allocated when feeding a range of pages to the allocator
- * for the first time during bootup or memory hotplug.
- */
- if (unlikely(!base))
- return NULL;
- index = pfn - round_down(node_start_pfn(page_to_nid(page)),
- MAX_ORDER_NR_PAGES);
- return get_entry(base, index);
-}
-
static int __init alloc_node_page_ext(int nid)
{
struct page_ext *base;
@@ -247,29 +217,6 @@ static int __init __page_ext_alloc(void)
}

#else /* CONFIG_SPARSEMEM */
-static bool page_ext_invalid(struct page_ext *page_ext)
-{
- return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID);
-}
-
-static struct page_ext *lookup_page_ext(const struct page *page)
-{
- unsigned long pfn = page_to_pfn(page);
- struct mem_section *section = __pfn_to_section(pfn);
- struct page_ext *page_ext = READ_ONCE(section->page_ext);
-
- WARN_ON_ONCE(!rcu_read_lock_held());
- /*
- * The sanity checks the page allocator does upon freeing a
- * page can reach here before the page_ext arrays are
- * allocated when feeding a range of pages to the allocator
- * for the first time during bootup or memory hotplug.
- */
- if (page_ext_invalid(page_ext))
- return NULL;
- return get_entry(page_ext, pfn);
-}
-
static __ref void *alloc_page_ext(size_t size, int nid)
{
void *addr;
@@ -360,7 +307,7 @@ static void __free_page_ext(unsigned long pfn)
base = (void *)base - PAGE_EXT_INVALID;
WRITE_ONCE(ms->page_ext, NULL);

- base = get_entry(base, pfn);
+ base = page_ext_entry(base, pfn);
free_page_ext(base);
}

@@ -503,7 +450,8 @@ void __init page_ext_init(void)
*/
struct page_ext *page_ext_lookup(unsigned long pfn)
{
- return lookup_page_ext(pfn_to_page(pfn));
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ return __page_ext_lookup(pfn);
}

/**
@@ -522,7 +470,7 @@ struct page_ext *page_ext_get(const struct page *page)
struct page_ext *page_ext;

rcu_read_lock();
- page_ext = lookup_page_ext(page);
+ page_ext = page_ext_lookup(page_to_pfn(page));
if (!page_ext) {
rcu_read_unlock();
return NULL;
--
2.54.0

Muchun Song

unread,
Nov 17, 2026, 7:07:52 AM (3 days ago) Nov 17
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN stores each page's shadow and origin pointers in struct page.
Boot-time gigantic HugeTLB pages are reinitialized after
kmsan_init_shadow() installs these pointers, dropping the mappings and
leaving the allocated metadata pages unreachable.

Move the pointers to page_ext, whose boot-time storage is available
before kmsan_init_shadow() and survives struct page reinitialization.
Use the inline unchecked lookup under notrace preemption so metadata
lookup cannot recurse through instrumented hooks.

This shrinks struct page from 80 to 64 bytes on x86-64 KMSAN builds,
making its size a power of two and allowing HugeTLB Vmemmap Optimization
with KMSAN.

Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song <songm...@bytedance.com>
---
v2:
- Update __mm_zero_struct_page() for the smaller struct page.
- Keep the unchecked lookup inline in KMSAN and leave page_ext.c
instrumented (suggested by Chris Down)
- Document page_ext metadata storage and KMSAN HVO support (suggested
by Chris Down)
- Use the common boot-time page_ext allocation instead of need_early
---
Documentation/dev-tools/kmsan.rst | 8 +++---
include/linux/kmsan.h | 2 ++
include/linux/mm.h | 12 +++------
include/linux/mm_types.h | 12 ---------
lib/Kconfig.kmsan | 1 +
mm/kmsan/shadow.c | 45 ++++++++++++++++++++++++++-----
mm/page_ext.c | 4 +++
7 files changed, 52 insertions(+), 32 deletions(-)

diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/kmsan.rst
index 0dc668b183f6..5330d21d141d 100644
--- a/Documentation/dev-tools/kmsan.rst
+++ b/Documentation/dev-tools/kmsan.rst
@@ -376,13 +376,11 @@ Metadata allocation

There are several places in the kernel for which the metadata is stored.

-1. Each ``struct page`` instance contains two pointers to its shadow and
-origin pages::
+1. Each page frame has a corresponding ``struct page_metadata`` entry containing
+two pointers to its shadow and origin pages::

- struct page {
- ...
+ struct page_metadata {
struct page *shadow, *origin;
- ...
};

At boot-time, the kernel allocates shadow and origin pages for every available
diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h
index 7da9fd506b39..08fb79566235 100644
--- a/include/linux/kmsan.h
+++ b/include/linux/kmsan.h
@@ -15,12 +15,14 @@
#include <linux/types.h>

struct page;
+struct page_ext_operations;
struct kmem_cache;
struct task_struct;
struct scatterlist;
struct urb;

#ifdef CONFIG_KMSAN
+extern struct page_ext_operations kmsan_ops;

/**
* kmsan_task_create() - Initialize KMSAN state for the task.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 77b5d450fe45..9349209cd2c0 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -133,7 +133,7 @@ extern int mmap_rnd_compat_bits __read_mostly;
* define their own version of this macro in <asm/pgtable.h>
*/
#if BITS_PER_LONG == 64
-/* This function must be updated when the size of struct page grows above 96
+/* This function must be updated when the size of struct page grows above 80
* or reduces below 56. The idea that compiler optimizes out switch()
* statement, and only leaves move/store instructions. Also the compiler can
* combine write statements if they are both assignments and can be reordered,
@@ -144,18 +144,12 @@ static inline void __mm_zero_struct_page(struct page *page)
{
unsigned long *_pp = (void *)page;

- /* Check that struct page is either 56, 64, 72, 80, 88 or 96 bytes */
+ /* Check that struct page is either 56, 64, 72 or 80 bytes */
BUILD_BUG_ON(sizeof(struct page) & 7);
BUILD_BUG_ON(sizeof(struct page) < 56);
- BUILD_BUG_ON(sizeof(struct page) > 96);
+ BUILD_BUG_ON(sizeof(struct page) > 80);

switch (sizeof(struct page)) {
- case 96:
- _pp[11] = 0;
- fallthrough;
- case 88:
- _pp[10] = 0;
- fallthrough;
case 80:
_pp[9] = 0;
fallthrough;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..dc73f9f5a51b 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -208,18 +208,6 @@ struct page {
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
int _last_cpupid;
#endif
-
-#ifdef CONFIG_KMSAN
- /*
- * KMSAN metadata for this page:
- * - shadow page: every bit indicates whether the corresponding
- * bit of the original page is initialized (0) or not (1);
- * - origin page: every 4 bytes contain an id of the stack trace
- * where the uninitialized value was created.
- */
- struct page *kmsan_shadow;
- struct page *kmsan_origin;
-#endif
} _struct_page_alignment;

/*
diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan
index cae1ddcc18e1..015e85c70793 100644
--- a/lib/Kconfig.kmsan
+++ b/lib/Kconfig.kmsan
@@ -10,6 +10,7 @@ config KMSAN
depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER
depends on DEBUG_KERNEL && !KASAN && !KCSAN
depends on !PREEMPT_RT
+ select PAGE_EXTENSION
select STACKDEPOT
select STACKDEPOT_ALWAYS_INIT
help
diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 300b48818af1..fa2a403d87c2 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -13,6 +13,7 @@
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
+#include <linux/page_ext.h>
#include <linux/slab.h>
#include <linux/smp.h>
#include <linux/stddef.h>
@@ -24,23 +25,55 @@
#define KMSAN_META_MASK_SHADOW BIT(0)
#define KMSAN_META_MASK_ORIGIN BIT(1)

+/*
+ * KMSAN metadata for this page:
+ * - shadow page: every bit indicates whether the corresponding
+ * bit of the original page is initialized (0) or not (1);
+ * - origin page: every 4 bytes contain an id of the stack trace
+ * where the uninitialized value was created.
+ */
struct page_metadata {
struct page *shadow;
struct page *origin;
};

+static __init bool need_page_metadata(void)
+{
+ return true;
+}
+
+struct page_ext_operations kmsan_ops = {
+ .size = sizeof(struct page_metadata),
+ .need = need_page_metadata,
+};
+
static inline struct page_metadata get_page_metadata(struct page *page)
{
- return (struct page_metadata) {
- .shadow = page->kmsan_shadow,
- .origin = page->kmsan_origin,
- };
+ struct page_metadata meta = {};
+ struct page_ext *page_ext;
+
+ /*
+ * Avoid instrumented preemption debug and tracing hooks while resolving
+ * KMSAN metadata, as they would recurse back into metadata lookup.
+ */
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
+ preempt_enable_no_resched_notrace();
+
+ return meta;
}

static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- page->kmsan_shadow = meta.shadow;
- page->kmsan_origin = meta.origin;
+ struct page_ext *page_ext;
+
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops) = meta;
+ preempt_enable_no_resched_notrace();
}

static inline void *shadow_ptr_for(struct page_metadata meta)
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 0f31e2da2611..274c3e778be6 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/kmsan.h>
#include <linux/slab.h>

/*
@@ -87,6 +88,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {
#ifdef CONFIG_PAGE_TABLE_CHECK
&page_table_check_ops,
#endif
+#ifdef CONFIG_KMSAN
+ &kmsan_ops,
+#endif
#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
&page_iommu_debug_ops,
#endif
--
2.54.0

sashi...@kernel.org

unread,
Nov 17, 2026, 2:36:29 PM (2 days ago) Nov 17
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] mm/page_ext: 4-byte alignment cap in
page_ext_data_alignment() on 32-bit
--

commit 4ecca9da35d7e4448f4270965043775293174eda
Author: Muchun Song <muchu...@linux.dev>

mm/page_ext: fix client data alignment

This commit aligns page_ext client offsets and rounds the final entry size to
the largest active client alignment.

> diff --git a/mm/page_ext.c b/mm/page_ext.c
> index b679a8c1f7d7e..dbc1bfa258f15 100644
> --- a/mm/page_ext.c
> +++ b/mm/page_ext.c
> @@ -116,8 +116,22 @@ static int __init setup_early_page_ext(char *str)
> }
> early_param("early_page_ext", setup_early_page_ext);
>
> +static inline size_t page_ext_data_alignment(size_t size)
> +{
> + if (!size)
> + return 1;
> +
> + /*
> + * A type's size is a multiple of its natural alignment. The lowest
> + * set bit of the size is therefore at least that alignment. Cap it
> + * at unsigned long alignment to avoid excessive per-page padding.
> + */
> + return min_t(size_t, 1UL << __ffs(size), __alignof__(unsigned long));
> +}

[Severity: Medium]
Can capping the alignment at __alignof__(unsigned long) leave struct
page_owner misaligned on 32-bit architectures?

struct page_owner contains u64 ts_nsec and u64 free_ts_nsec, which require
8-byte alignment on 32-bit architectures such as ARM EABI, MIPS32, and
PowerPC32 where __alignof__(u64) is 8 but __alignof__(unsigned long) is 4.
With page_owner=on, sizeof(struct page_ext) is 4, so invoke_need_callbacks()
places page_owner_ops.offset at 4 and sets page_ext_size to 68, causing
misaligned 64-bit accesses on ts_nsec and free_ts_nsec.

Capping at __alignof__(u64) instead of __alignof__(unsigned long) would keep
the 8-byte cap while preserving u64 alignment on 32-bit systems.

--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117120627.5...@bytedance.com?part=2

Andrew Morton

unread,
Nov 17, 2026, 2:46:39 PM (2 days ago) Nov 17
to sashiko...@lists.linux.dev, sashi...@kernel.org, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
> [blah blah]
>

Yeah, but the whole expression is quite reader-hostile, as min/max
things often are.

How about doing

/* comment */
if (this1)
return that1;
/* comment */
if (this2)
return that2;
/* comment */
if (this3)
return that3;

?

The code generation will probably be just as good and it's __init-time
stuff anyway.

Chris Down

unread,
Nov 17, 2026, 7:06:49 PM (2 days ago) Nov 17
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
Hi Muchun,

Thanks! The changes look great, thanks for working on this.

I think patch 5 still has some issues and have replied there, but for patches
1-4, 6, and 7, feel free to add:

Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>

Thanks!

Chris

Chris Down

unread,
Nov 17, 2026, 7:06:54 PM (2 days ago) Nov 17
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
Hi Muchun,

Muchun Song writes:
>+ if (!slab_is_available()) {
>+ addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
>+ MEMBLOCK_ALLOC_ACCESSIBLE, nid);

Hmm, unless I'm mistaken, memblock allocates top-down, and with
movablecore= or kernelcore= the top of each node is ZONE_MOVABLE, so this
puts the page_ext tables there. It looks like that then offlining those
memory blocks would fail with EBUSY because of the reserved pages.

I can see why Mike suggested always using memblock here, but only KMSAN
needs the tables before the buddy allocator is up. So I think this needs
to either go back to using memblock only for clients that need it, as v1
did, or keep these allocations below ZONE_MOVABLE.

Thanks!

Chris

Muchun Song

unread,
Nov 17, 2026, 10:12:06 PM (2 days ago) Nov 17
to Chris Down, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org


> On Nov 18, 2026, at 08:06, Chris Down <ch...@chrisdown.name> wrote:
>
> Hi Muchun,

Hi Chris,

Thanks for taking a look.

>
> Muchun Song writes:
>> + if (!slab_is_available()) {
>> + addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
>> + MEMBLOCK_ALLOC_ACCESSIBLE, nid);
>
> Hmm, unless I'm mistaken, memblock allocates top-down, and with
> movablecore= or kernelcore= the top of each node is ZONE_MOVABLE, so this
> puts the page_ext tables there. It looks like that then offlining those
> memory blocks would fail with EBUSY because of the reserved pages.

Yes, this is a valid concern. However, early vmemmap backing is also
allocated from memblock before page_ext and can have the same problem, so
handling page_ext alone cannot guarantee successful offlining.

Mike's suggestion considerably simplifies the code, and early page_ext is
only needed by optional debugging and instrumentation users. I would prefer
to keep this approach.

What do you think?

Thanks,
Muchun

Muchun Song

unread,
Nov 17, 2026, 10:13:14 PM (2 days ago) Nov 17
to Chris Down, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org


> On Nov 18, 2026, at 08:06, Chris Down <ch...@chrisdown.name> wrote:
>
Really thanks for your review and test.

Thanks,
Muchun

>
> Thanks!
>
> Chris

Muchun Song

unread,
Nov 17, 2026, 10:50:28 PM (2 days ago) Nov 17
to Andrew Morton, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
Something like?

static inline size_t page_ext_data_alignment(size_t size)
{
/* No client data means no alignment requirement. */
if (!size)
return 1;

/* u64 has the strictest alignment required by page_ext clients. */
if (IS_ALIGNED(size, __alignof__(u64)))
return __alignof__(u64);

/*
* A type's size is a multiple of its natural alignment, so its
* lowest set bit is at least that alignment.
*/
return 1UL << __ffs(size);
}

Thanks,
Muchun


Andrew Morton

unread,
Nov 17, 2026, 11:21:39 PM (2 days ago) Nov 17
to Muchun Song, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
Well I prefer it!

The final bit-twiddle could be (size & ~size) but I'm not sure that's
any clearer.

We should have a handy helper for this operation but I can't find one.
max_pow_of_two_factor() is close, but only works on uint.

Muchun Song

unread,
Nov 18, 2026, 12:39:25 AM (yesterday) Nov 18
to Andrew Morton, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
How about changing the helper to

#define max_pow_of_two_factor(n) ({ \
typeof(n) __n = (n); \
static_assert(__same_type(__n, 0U) || __same_type(__n, 0UL)); \
__n & -__n; \
})

?

Andrew Morton

unread,
Nov 18, 2026, 12:51:32 AM (yesterday) Nov 18
to Muchun Song, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
On Wed, 18 Nov 2026 13:39:04 +0800 Muchun Song <muchu...@linux.dev> wrote:

>
> >>
> >> Something like?
> >>
> >> static inline size_t page_ext_data_alignment(size_t size)
> >> {
> >> /* No client data means no alignment requirement. */
> >> if (!size)
> >> return 1;
> >>
> >> /* u64 has the strictest alignment required by page_ext clients. */
> >> if (IS_ALIGNED(size, __alignof__(u64)))
> >> return __alignof__(u64);
> >>
> >> /*
> >> * A type's size is a multiple of its natural alignment, so its
> >> * lowest set bit is at least that alignment.
> >> */
> >> return 1UL << __ffs(size);
> >> }
> >
> > Well I prefer it!
> >
> > The final bit-twiddle could be (size & ~size) but I'm not sure that's
> > any clearer.

err, (size & -size).

> > We should have a handy helper for this operation but I can't find one.
> > max_pow_of_two_factor() is close, but only works on uint.
>
> How about changing the helper to
>
> #define max_pow_of_two_factor(n) ({ \
> typeof(n) __n = (n); \
> static_assert(__same_type(__n, 0U) || __same_type(__n, 0UL)); \
> __n & -__n; \
> })
>

Switching max_pow_of_two_factor() to take a ulong should be OK - the
thing only has two callers. But oh, that name!

But do we really want to go there? The drama with min/max/min_t/max_t
and especially clamp has been endless.

Your open-coded bit-twiddle looks great ;)

Muchun Song

unread,
Nov 18, 2026, 12:57:44 AM (yesterday) Nov 18
to Andrew Morton, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
OK. I'll use size & -size directly. I think the adding comment could make it clear.




Chris Down

unread,
Nov 18, 2026, 2:00:15 AM (yesterday) Nov 18
to Muchun Song, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
Hi Muchun,

Muchun Song writes:
>Mike's suggestion considerably simplifies the code, and early page_ext is
>only needed by optional debugging and instrumentation users. I would prefer
>to keep this approach.

You're probably right. But can I cheekily suggest something in the middle
perhaps? :-)

Given the zones are already set up before page_ext_alloc() runs, perhaps
you can pass the start of the node's ZONE_MOVABLE as max_addr to
memblock_alloc_try_nid() when the node has one. The general idea is:

struct zone *movable = &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
phys_addr_t max_addr = MEMBLOCK_ALLOC_ACCESSIBLE;

if (!zone_is_empty(movable))
max_addr = PFN_PHYS(movable->zone_start_pfn);
addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
max_addr, nid);
if (!addr && max_addr != MEMBLOCK_ALLOC_ACCESSIBLE)
addr = memblock_alloc_try_nid(size, PAGE_SIZE,
__pa(MAX_DMA_ADDRESS),
MEMBLOCK_ALLOC_ACCESSIBLE, nid);

I briefly tried that and it seems to work quite nicely.

But I don't object to it even without that, so feel free to add either way:

Reviewed-by: Chris Down <ch...@chrisdown.name>

Thanks!

Chris

Mike Rapoport

unread,
Nov 18, 2026, 8:41:00 AM (yesterday) Nov 18
to Muchun Song, Andrew Morton, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
Maybe add a const local for __alignof__(u64)?
No strong feelings about it.

> >>
> >> /*
> >> * A type's size is a multiple of its natural alignment, so its
> >> * lowest set bit is at least that alignment.
> >> */
> >> return 1UL << __ffs(size);
> >> }
> >
> > Well I prefer it!
> >
> > The final bit-twiddle could be (size & ~size) but I'm not sure that's
> > any clearer.
> >
> > We should have a handy helper for this operation but I can't find one.
> > max_pow_of_two_factor() is close, but only works on uint.
>
> How about changing the helper to
>
> #define max_pow_of_two_factor(n) ({ \
> typeof(n) __n = (n); \
> static_assert(__same_type(__n, 0U) || __same_type(__n, 0UL)); \
> __n & -__n; \
> })
>
> ?
>

--
Sincerely yours,
Mike.

Mike Rapoport

unread,
Nov 18, 2026, 8:44:09 AM (yesterday) Nov 18
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
On Tue, Nov 17, 2026 at 08:06:23PM +0800, Muchun Song wrote:
> SPARSEMEM currently tries to allocate each section's page extensions
> from the buddy allocator before falling back to vmalloc. Boot-time
> section allocation happens after vmalloc is available, and memory
> hotplug allocations do not require physically contiguous backing.
>
> Use vzalloc_node() directly and remove the buddy-specific allocation,
> free, and kmemleak bookkeeping. This leaves one runtime backing type
> and lets a later change add memblock as the distinct boot-time path
> to simplify the code.
>
> Signed-off-by: Muchun Song <songm...@bytedance.com>

Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>

> ---
> v2:
> - Add a preparatory cleanup that always uses vmalloc for SPARSEMEM
> section page extensions (suggested by Mike Rapoport)
> ---
> mm/page_ext.c | 21 +++------------------
> 1 file changed, 3 insertions(+), 18 deletions(-)

--
Sincerely yours,
Mike.

Mike Rapoport

unread,
Nov 18, 2026, 9:18:07 AM (yesterday) Nov 18
to Chris Down, Muchun Song, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
On Tue, Nov 17, 2026 at 11:00:06PM -0800, Chris Down wrote:
> Hi Muchun,
>
> Muchun Song writes:
> > Mike's suggestion considerably simplifies the code, and early page_ext is
> > only needed by optional debugging and instrumentation users. I would prefer
> > to keep this approach.
>
> You're probably right.

I think so :)
The memap_alloc() that essentially allocates the memory map does

memblock_alloc_try_nid_raw(size, align, min_addr,
MEMBLOCK_ALLOC_NOLEAKTRACE, nid);

so we anyway may end up with reserved memory in ZONE_MOVABLE.

Until recently we had memory map allocation before the zones were setup, so
there wasn't much we could do about it, but ...

> But can I cheekily suggest something in the middle perhaps? :-)
>
> Given the zones are already set up before page_ext_alloc() runs, perhaps
> you can pass the start of the node's ZONE_MOVABLE as max_addr to
> memblock_alloc_try_nid() when the node has one. The general idea is:
>
> struct zone *movable = &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
> phys_addr_t max_addr = MEMBLOCK_ALLOC_ACCESSIBLE;
>
> if (!zone_is_empty(movable))
> max_addr = PFN_PHYS(movable->zone_start_pfn);
> addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
> max_addr, nid);
> if (!addr && max_addr != MEMBLOCK_ALLOC_ACCESSIBLE)
> addr = memblock_alloc_try_nid(size, PAGE_SIZE,
> __pa(MAX_DMA_ADDRESS),
> MEMBLOCK_ALLOC_ACCESSIBLE, nid);

... with zone extents known we could take it even further :)

page_ext is treated as the memory map in many ways, so we can allocate the
early page_ext sections with memmap_alloc().
And since the latter always runs after zones are set up it could include
this change.

But I think this should come as a separate series that also could move
memmap_boot_pages_add to memmap_alloc().

> I briefly tried that and it seems to work quite nicely.
>
> But I don't object to it even without that, so feel free to add either way:
>
> Reviewed-by: Chris Down <ch...@chrisdown.name>
>
> Thanks!
>
> Chris

--
Sincerely yours,
Mike.

Mike Rapoport

unread,
Nov 18, 2026, 9:18:59 AM (yesterday) Nov 18
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
On Tue, Nov 17, 2026 at 08:06:24PM +0800, Muchun Song wrote:
> FLATMEM uses separate page_ext_init_flatmem() and
> page_ext_init_flatmem_late() entry points, while SPARSEMEM performs its
> work through page_ext_init(). This exposes the memory model in the
> generic boot sequence and obscures the allocation and client
> initialization phases.
>
> Rename the early entry point to page_ext_alloc() and use
> page_ext_init() for client callbacks in both memory models. SPARSEMEM
> keeps its existing late allocation behavior for now; its
> page_ext_alloc() remains an empty configuration stub.
>
> This also moves FLATMEM client callbacks from immediately after
> kmem_cache_init() to after vmalloc_init(). The callbacks still run
> after the buddy and slab allocators are available, as required by
> page_owner.
>
> Signed-off-by: Muchun Song <songm...@bytedance.com>

Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>

> ---
> v2:
> - Split the common allocation and initialization entry points into a
> preparatory patch
> - Document the later FLATMEM client callback timing (noted by Chris
> Down)
> ---
> include/linux/page_ext.h | 20 +++++---------------
> mm/mm_init.c | 7 +------
> mm/page_ext.c | 5 ++---
> 3 files changed, 8 insertions(+), 24 deletions(-)

--
Sincerely yours,
Mike.

Mike Rapoport

unread,
Nov 18, 2026, 9:22:21 AM (yesterday) Nov 18
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
On Tue, Nov 17, 2026 at 08:06:25PM +0800, Muchun Song wrote:
> KMSAN needs page extensions when kmsan_init_shadow() installs per-page
> metadata, before memblock_free_all() releases memory to the buddy
> allocator. FLATMEM makes page extensions available by this point,
> whereas SPARSEMEM initializes them later in boot.
>
> Allocate boot-time SPARSEMEM page extensions from memblock to match
> FLATMEM's initialization timing. This gives early clients the same
> page_ext availability guarantee across both memory models, while runtime
> memory hotplug continues to use vmalloc.
>
> Signed-off-by: Muchun Song <songm...@bytedance.com>

Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>

with a nit below

> ---
> v2:
> - Allocate all boot-time SPARSEMEM page extensions from memblock
> (suggested by Mike Rapoport)
> - Keep memblock allocations above DMA and suppress per-section free
> messages (suggested by Chris Down)
> ---
> include/linux/page_ext.h | 9 +--
> mm/mm_init.c | 4 --
> mm/page_ext.c | 119 +++++++++++++++++++--------------------
> 3 files changed, 60 insertions(+), 72 deletions(-)
>
> +static __ref void *alloc_page_ext(size_t size, int nid)
> {
> void *addr;
>
> + if (!slab_is_available()) {
> + addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
> + MEMBLOCK_ALLOC_ACCESSIBLE, nid);
> + if (addr)
> + memmap_boot_pages_add(size / PAGE_SIZE);

I think it should be DIV_ROUND_UP().

> + return addr;
> + }
> +
> addr = vzalloc_node(size, nid);
> if (addr)
> - memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));
> + memmap_pages_add(size / PAGE_SIZE);

And here too.

>
> return addr;
> }
> @@ -304,7 +300,7 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
> if (section->page_ext)
> return 0;
>
> - table_size = page_ext_size * PAGES_PER_SECTION;
> + table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
> base = alloc_page_ext(table_size, nid);
>
> /*

--
Sincerely yours,
Mike.

Muchun Song

unread,
Nov 18, 2026, 10:04:08 AM (yesterday) Nov 18
to Mike Rapoport, Andrew Morton, sashiko...@lists.linux.dev, Sashiko, Muchun Song, Alexander Potapenko, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, kasa...@googlegroups.com, linu...@kvack.org
Maybe better. I can give it a try.

Muchun Song

unread,
Nov 18, 2026, 10:13:44 AM (yesterday) Nov 18
to Mike Rapoport, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Muchun Song, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org


On 2026/11/18 22:22, Mike Rapoport wrote:
> On Tue, Nov 17, 2026 at 08:06:25PM +0800, Muchun Song wrote:
>> KMSAN needs page extensions when kmsan_init_shadow() installs per-page
>> metadata, before memblock_free_all() releases memory to the buddy
>> allocator. FLATMEM makes page extensions available by this point,
>> whereas SPARSEMEM initializes them later in boot.
>>
>> Allocate boot-time SPARSEMEM page extensions from memblock to match
>> FLATMEM's initialization timing. This gives early clients the same
>> page_ext availability guarantee across both memory models, while runtime
>> memory hotplug continues to use vmalloc.
>>
>> Signed-off-by: Muchun Song <songm...@bytedance.com>
> Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>

Thanks for reviewing.

>
> with a nit below
>
>> ---
>> v2:
>> - Allocate all boot-time SPARSEMEM page extensions from memblock
>> (suggested by Mike Rapoport)
>> - Keep memblock allocations above DMA and suppress per-section free
>> messages (suggested by Chris Down)
>> ---
>> include/linux/page_ext.h | 9 +--
>> mm/mm_init.c | 4 --
>> mm/page_ext.c | 119 +++++++++++++++++++--------------------
>> 3 files changed, 60 insertions(+), 72 deletions(-)
>>
>> +static __ref void *alloc_page_ext(size_t size, int nid)
>> {
>> void *addr;
>>
>> + if (!slab_is_available()) {
>> + addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
>> + MEMBLOCK_ALLOC_ACCESSIBLE, nid);
>> + if (addr)
>> + memmap_boot_pages_add(size / PAGE_SIZE);
> I think it should be DIV_ROUND_UP().

size is currently page-aligned by init_section_page_ext(), so both forms are
equivalent here. Is the suggestion to avoid relying on this caller-side
invariant and make it more robust? If so, I'll use DIV_ROUND_UP() for both
accounting calls.

Thanks,
Muchun

Muchun Song

unread,
Nov 18, 2026, 10:38:13 AM (yesterday) Nov 18
to Mike Rapoport, Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Chris Down
Agreed. Using memmap_alloc() for early page_ext allocations and
teaching it to avoid ZONE_MOVABLE sounds cleaner. I'll keep this
series as is and handle that in a separate series.

Thanks!

Mike Rapoport

unread,
Nov 18, 2026, 2:04:04 PM (yesterday) Nov 18
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Muchun Song, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
Yeah, keeping it here does not cost much and whatever
init_section_page_ext() does we still count the pages right.

Especially if we move the whole thingy to memmap_alloc().

> Thanks,
> Muchun
>
> >
> > > + return addr;
> > > + }
> > > +
> > > addr = vzalloc_node(size, nid);
> > > if (addr)
> > > - memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));
> > > + memmap_pages_add(size / PAGE_SIZE);
> > And here too.
> >
> > > return addr;
> > > }
> > > @@ -304,7 +300,7 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
> > > if (section->page_ext)
> > > return 0;
> > > - table_size = page_ext_size * PAGES_PER_SECTION;
> > > + table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
> > > base = alloc_page_ext(table_size, nid);
> > > /*
>

--
Sincerely yours,
Mike.

Muchun Song

unread,
1:59 AM (18 hours ago) 1:59 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
This series fixes a KMSAN metadata-loss bug triggered when boot-time
gigantic HugeTLB pages reinitialize their page descriptors after
kmsan_init_shadow(). KMSAN currently stores each page's shadow and
origin pointers in struct page, so the later initialization drops the
metadata mappings and leaves their backing pages unreachable.

Preserving the pointers around the HugeTLB initialization would add
KMSAN-specific state handling to the memmap path. Moving
kmsan_init_shadow() later is not viable either, because it allocates
metadata from memblock while gigantic page descriptors are initialized
after memblock teardown.

Move the metadata pointers to page_ext instead. page_ext storage is
independent of struct page contents and is made available before
kmsan_init_shadow(), so later page-descriptor initialization cannot
erase the mappings.

The series first decouples KMSAN metadata access from its storage, then
makes page_ext safe and available during early boot, and finally moves
the metadata pointers. The move also shrinks struct page from 80 to 64
bytes on x86-64 KMSAN builds, allowing HugeTLB Vmemmap Optimization to
be used with KMSAN.

v3:
- Fix page_ext client alignment on 32-bit architectures by using u64
alignment as the cap (reported by Sashiko).
- Open-code the lowest power-of-two factor and use a local for the u64
alignment (suggested by Andrew Morton and Mike Rapoport).
- Use DIV_ROUND_UP() for page_ext accounting instead of relying on the
caller's size alignment (suggested by Mike Rapoport).
- Simplify the page_ext alignment commit message.
- Collect Reviewed-by and Tested-by tags from Chris Down for patches
1-4, 6, and 7, and Reviewed-by for patch 5.
- Collect Reviewed-by tags from Mike Rapoport for patches 3-5.
- Rebase onto mm/mm-new.

v2: https://lore.kernel.org/20261117120627.5...@bytedance.com/
- Fetch shadow and origin through one masked metadata resolver
(suggested by Chris Down).
- Add a fix for pre-existing page_ext client misalignment.
- Use vmalloc for runtime SPARSEMEM page_ext backing
(suggested by Mike Rapoport).
- Unify page_ext allocation and initialization entry points and document
the later FLATMEM callback timing (noted by Chris Down).
- Allocate all boot-time SPARSEMEM page_ext from memblock
(suggested by Mike Rapoport).
- Keep memblock allocations above DMA and suppress per-section free
messages (suggested by Chris Down).
- Inline the unchecked page_ext lookup while keeping page_ext.c
instrumented (suggested by Chris Down).
- Update the KMSAN metadata documentation and describe HVO enablement
(suggested by Chris Down).
- Expand the series from three to seven patches.
- Drop all v1 Reviewed-by tags because the affected patches were
substantially reworked and split.

v1: https://lore.kernel.org/20261114104150.5...@bytedance.com/

Muchun Song (7):
kmsan: encapsulate per-page metadata access
mm/page_ext: fix client data alignment
mm/page_ext: always use vmalloc for section page extensions
mm/page_ext: use common allocation and initialization entry points
mm/page_ext: allocate page extensions early for SPARSEMEM
mm/page_ext: inline unchecked page extension lookup
kmsan: fix metadata loss during HugeTLB struct page reinitialization

Documentation/dev-tools/kmsan.rst | 8 +-
include/linux/kmsan.h | 2 +
include/linux/mm.h | 12 +-
include/linux/mm_types.h | 12 --
include/linux/page_ext.h | 61 +++++++--
lib/Kconfig.kmsan | 1 +
mm/kmsan/shadow.c | 199 +++++++++++++++++++--------
mm/mm_init.c | 11 +-
mm/page_ext.c | 221 ++++++++++++------------------
9 files changed, 290 insertions(+), 237 deletions(-)


base-commit: 3dfc09a9493ddaa5e7722a29165d8450b55b4d3f
--
2.54.0

Muchun Song

unread,
1:59 AM (18 hours ago) 1:59 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN directly accesses page->kmsan_shadow and page->kmsan_origin
throughout the shadow implementation. This couples these users to
storing the metadata pointers in struct page, requiring every access
site to change when the storage moves elsewhere.

Introduce helpers that get and set both metadata page pointers as a pair
and convert the existing users to them.

kmsan_get_shadow_origin_ptr() is the main metadata lookup path for
compiler-instrumented memory accesses and requires both pointers. For
page-backed addresses, resolve the backing page once and obtain the
metadata pair through one lookup instead of calling the single-metadata
path twice. This also avoids duplicate lookups after moving the metadata
storage out of struct page.

No functional change is intended.

Signed-off-by: Muchun Song <songm...@bytedance.com>
Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>
---
v3:
- Collect Reviewed-by and Tested-by from Chris Down

v2:
- Use one page metadata lookup for shadow and origin pointers (suggested
by Chris Down)
---
mm/kmsan/shadow.c | 166 +++++++++++++++++++++++++++++++---------------
1 file changed, 111 insertions(+), 55 deletions(-)

diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 0c88d89bf0d6..300b48818af1 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -9,6 +9,7 @@

#include <asm/kmsan.h>
#include <asm/tlbflush.h>
+#include <linux/bits.h>
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
@@ -20,29 +21,46 @@
#include "../vmalloc.h"
#include "kmsan.h"

-#define shadow_page_for(page) ((page)->kmsan_shadow)
+#define KMSAN_META_MASK_SHADOW BIT(0)
+#define KMSAN_META_MASK_ORIGIN BIT(1)

-#define origin_page_for(page) ((page)->kmsan_origin)
+struct page_metadata {
+ struct page *shadow;
+ struct page *origin;
+};

-static void *shadow_ptr_for(struct page *page)
+static inline struct page_metadata get_page_metadata(struct page *page)
{
- return page_address(shadow_page_for(page));
+ return (struct page_metadata) {
+ .shadow = page->kmsan_shadow,
+ .origin = page->kmsan_origin,
+ };
}

-static void *origin_ptr_for(struct page *page)
+static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- return page_address(origin_page_for(page));
+ page->kmsan_shadow = meta.shadow;
+ page->kmsan_origin = meta.origin;
}

-static bool page_has_metadata(struct page *page)
+static inline void *shadow_ptr_for(struct page_metadata meta)
{
- return shadow_page_for(page) && origin_page_for(page);
+ return page_address(meta.shadow);
+}
+
+static inline void *origin_ptr_for(struct page_metadata meta)
+{
+ return page_address(meta.origin);
+}
+
+static inline bool page_has_metadata(struct page_metadata meta)
+{
+ return meta.shadow && meta.origin;
}

static void set_no_shadow_origin_page(struct page *page)
{
- shadow_page_for(page) = NULL;
- origin_page_for(page) = NULL;
+ set_page_metadata(page, (struct page_metadata) {});
}

/*
@@ -79,11 +97,56 @@ static struct page *virt_to_page_or_null(void *vaddr)
return NULL;
}

+static __always_inline
+struct shadow_origin_ptr __kmsan_get_metadata(void *address, unsigned int mask)
+{
+ struct shadow_origin_ptr ret = {};
+ struct page_metadata meta;
+ struct page *page;
+ u64 addr = (u64)address;
+ u64 origin_addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE);
+
+ if (kmsan_internal_is_vmalloc_addr(address) ||
+ kmsan_internal_is_module_addr(address)) {
+ if (mask & KMSAN_META_MASK_SHADOW)
+ ret.shadow = (void *)vmalloc_meta(address, KMSAN_META_SHADOW);
+ if (mask & KMSAN_META_MASK_ORIGIN)
+ ret.origin = (void *)vmalloc_meta((void *)origin_addr,
+ KMSAN_META_ORIGIN);
+ return ret;
+ }
+
+ if (mask & KMSAN_META_MASK_SHADOW) {
+ ret.shadow = arch_kmsan_get_meta_or_null(address, KMSAN_META_SHADOW);
+ if (!ret.shadow)
+ goto lookup_page;
+ }
+ if (mask & KMSAN_META_MASK_ORIGIN)
+ ret.origin = arch_kmsan_get_meta_or_null((void *)origin_addr,
+ KMSAN_META_ORIGIN);
+ if (ret.shadow || ret.origin)
+ return ret;
+
+lookup_page:
+ page = virt_to_page_or_null(address);
+ if (!page)
+ return ret;
+ meta = get_page_metadata(page);
+ if (!page_has_metadata(meta))
+ return ret;
+
+ if (mask & KMSAN_META_MASK_SHADOW)
+ ret.shadow = shadow_ptr_for(meta) + offset_in_page(addr);
+ if (mask & KMSAN_META_MASK_ORIGIN)
+ ret.origin = origin_ptr_for(meta) + offset_in_page(origin_addr);
+
+ return ret;
+}
+
struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
bool store)
{
struct shadow_origin_ptr ret;
- void *shadow;

/*
* Even if we redirect this memory access to the dummy page, it will
@@ -95,12 +158,10 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
goto return_dummy;

KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(address, size));
- shadow = kmsan_get_metadata(address, KMSAN_META_SHADOW);
- if (!shadow)
+ ret = __kmsan_get_metadata(address, KMSAN_META_MASK_SHADOW |
+ KMSAN_META_MASK_ORIGIN);
+ if (!ret.shadow)
goto return_dummy;
-
- ret.shadow = shadow;
- ret.origin = kmsan_get_metadata(address, KMSAN_META_ORIGIN);
return ret;

return_dummy:
@@ -124,46 +185,36 @@ struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 size,
*/
void *kmsan_get_metadata(void *address, bool is_origin)
{
- u64 addr = (u64)address, off;
- struct page *page;
- void *ret;
-
- if (is_origin)
- addr = ALIGN_DOWN(addr, KMSAN_ORIGIN_SIZE);
- address = (void *)addr;
- if (kmsan_internal_is_vmalloc_addr(address) ||
- kmsan_internal_is_module_addr(address))
- return (void *)vmalloc_meta(address, is_origin);
-
- ret = arch_kmsan_get_meta_or_null(address, is_origin);
- if (ret)
- return ret;
-
- page = virt_to_page_or_null(address);
- if (!page)
- return NULL;
- if (!page_has_metadata(page))
- return NULL;
- off = offset_in_page(addr);
+ struct shadow_origin_ptr ret;
+ unsigned int mask;

- return (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) + off;
+ mask = is_origin ? KMSAN_META_MASK_ORIGIN : KMSAN_META_MASK_SHADOW;
+ ret = __kmsan_get_metadata(address, mask);
+ return is_origin ? ret.origin : ret.shadow;
}

void kmsan_copy_page_meta(struct page *dst, struct page *src)
{
+ struct page_metadata dst_meta, src_meta;
+
if (!kmsan_enabled || kmsan_in_runtime())
return;
- if (!dst || !page_has_metadata(dst))
+ if (!dst)
return;
- if (!src || !page_has_metadata(src)) {
+ dst_meta = get_page_metadata(dst);
+ if (!page_has_metadata(dst_meta))
+ return;
+ if (src)
+ src_meta = get_page_metadata(src);
+ if (!src || !page_has_metadata(src_meta)) {
kmsan_internal_unpoison_memory(page_address(dst), PAGE_SIZE,
/*checked*/ false);
return;
}

kmsan_enter_runtime();
- __memcpy(shadow_ptr_for(dst), shadow_ptr_for(src), PAGE_SIZE);
- __memcpy(origin_ptr_for(dst), origin_ptr_for(src), PAGE_SIZE);
+ __memcpy(shadow_ptr_for(dst_meta), shadow_ptr_for(src_meta), PAGE_SIZE);
+ __memcpy(origin_ptr_for(dst_meta), origin_ptr_for(src_meta), PAGE_SIZE);
kmsan_leave_runtime();
}
EXPORT_SYMBOL(kmsan_copy_page_meta);
@@ -171,19 +222,18 @@ EXPORT_SYMBOL(kmsan_copy_page_meta);
void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
{
bool initialized = (flags & __GFP_ZERO) || !kmsan_enabled;
- struct page *shadow, *origin;
+ struct page_metadata meta;
depot_stack_handle_t handle;
int pages = 1 << order;

if (!page)
return;

- shadow = shadow_page_for(page);
- origin = origin_page_for(page);
+ meta = get_page_metadata(page);

if (initialized) {
- __memset(page_address(shadow), 0, PAGE_SIZE * pages);
- __memset(page_address(origin), 0, PAGE_SIZE * pages);
+ __memset(shadow_ptr_for(meta), 0, PAGE_SIZE * pages);
+ __memset(origin_ptr_for(meta), 0, PAGE_SIZE * pages);
return;
}

@@ -191,7 +241,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
if (kmsan_in_runtime())
return;

- __memset(page_address(shadow), -1, PAGE_SIZE * pages);
+ __memset(shadow_ptr_for(meta), -1, PAGE_SIZE * pages);
kmsan_enter_runtime();
handle = kmsan_save_stack_with_flags(flags, /*extra_bits*/ 0);
kmsan_leave_runtime();
@@ -200,7 +250,7 @@ void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)
* to just fill the origin pages with @handle.
*/
for (int i = 0; i < PAGE_SIZE * pages / sizeof(handle); i++)
- ((depot_stack_handle_t *)page_address(origin))[i] = handle;
+ ((depot_stack_handle_t *)origin_ptr_for(meta))[i] = handle;
}

void kmsan_free_page(struct page *page, unsigned int order)
@@ -238,8 +288,10 @@ int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,
goto ret;
}
for (int i = 0; i < nr; i++) {
- s_pages[i] = shadow_page_for(pages[i]);
- o_pages[i] = origin_page_for(pages[i]);
+ struct page_metadata meta = get_page_metadata(pages[i]);
+
+ s_pages[i] = meta.shadow;
+ o_pages[i] = meta.origin;
}
prot = PAGE_KERNEL;

@@ -289,10 +341,12 @@ void __init kmsan_init_alloc_meta_for_range(void *start, void *end)
page = virt_to_page_or_null((char *)start + addr);
shadow_p = virt_to_page((char *)shadow + addr);
set_no_shadow_origin_page(shadow_p);
- shadow_page_for(page) = shadow_p;
origin_p = virt_to_page((char *)origin + addr);
set_no_shadow_origin_page(origin_p);
- origin_page_for(page) = origin_p;
+ set_page_metadata(page, (struct page_metadata) {
+ .shadow = shadow_p,
+ .origin = origin_p,
+ });
}
}

@@ -302,7 +356,9 @@ void kmsan_setup_meta(struct page *page, struct page *shadow,
for (int i = 0; i < (1 << order); i++) {
set_no_shadow_origin_page(&shadow[i]);
set_no_shadow_origin_page(&origin[i]);
- shadow_page_for(&page[i]) = &shadow[i];
- origin_page_for(&page[i]) = &origin[i];
+ set_page_metadata(&page[i], (struct page_metadata) {
+ .shadow = &shadow[i],
+ .origin = &origin[i],
+ });
}
}
--
2.54.0

Muchun Song

unread,
2:00 AM (18 hours ago) 2:00 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
page_ext lays out client payloads consecutively without aligning their
offsets or the final entry size. This happened to work while the active
payload sizes kept the stride naturally aligned, but IOMMU debug page
allocation added a four-byte payload after data requiring eight-byte
alignment.

When page owner and IOMMU debug page allocation are both enabled on
x86-64, struct page_ext, struct page_owner, and the IOMMU payload use
8, 64, and 4 bytes respectively. This produces a 76-byte entry stride,
leaving struct page_ext and page_owner data in every other entry only
four-byte aligned instead of eight-byte aligned.

Align each client offset and round the final entry size to the largest
active alignment so every page_ext entry preserves the alignment
required by all active clients.

Fixes: 466ae6978a5b ("iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC")
Signed-off-by: Muchun Song <songm...@bytedance.com>
Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>
---
v3:
- Cap client alignment at u64 alignment to preserve 64-bit alignment on
32-bit architectures (reported by Sashiko)
- Open-code the lowest power-of-two factor for readability (suggested by
Andrew Morton)
- Use a local for the u64 alignment (suggested by Mike Rapoport)
- Simplify the commit message to focus on the alignment guarantee
- Collect Reviewed-by and Tested-by from Chris Down

v2:
- New patch to fix the existing page_ext client alignment bug
---
mm/page_ext.c | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)

diff --git a/mm/page_ext.c b/mm/page_ext.c
index b679a8c1f7d7..39a4464fc979 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -116,8 +116,27 @@ static int __init setup_early_page_ext(char *str)
}
early_param("early_page_ext", setup_early_page_ext);

+static inline size_t page_ext_data_alignment(size_t size)
+{
+ const size_t align = __alignof__(u64);
+
+ if (!size)
+ return 1;
+
+ /* u64 has the strictest alignment required by page_ext clients. */
+ if (IS_ALIGNED(size, align))
+ return align;
+
+ /*
+ * A type's size is a multiple of its natural alignment, so its
+ * lowest set bit is at least that alignment.
+ */
+ return size & -size;
+}
+
static bool __init invoke_need_callbacks(void)
{
+ size_t max_align = 1;
int i;
int entries = ARRAY_SIZE(page_ext_ops);
bool need = false;
@@ -126,6 +145,7 @@ static bool __init invoke_need_callbacks(void)
if (page_ext_ops[i]->need()) {
if (page_ext_ops[i]->need_shared_flags) {
page_ext_size = sizeof(struct page_ext);
+ max_align = __alignof__(struct page_ext);
break;
}
}
@@ -133,11 +153,17 @@ static bool __init invoke_need_callbacks(void)

for (i = 0; i < entries; i++) {
if (page_ext_ops[i]->need()) {
+ size_t align;
+
+ align = page_ext_data_alignment(page_ext_ops[i]->size);
+ page_ext_size = ALIGN(page_ext_size, align);
page_ext_ops[i]->offset = page_ext_size;
page_ext_size += page_ext_ops[i]->size;
+ max_align = max(max_align, align);
need = true;
}
}
+ page_ext_size = ALIGN(page_ext_size, max_align);

return need;
}
--
2.54.0

Muchun Song

unread,
2:00 AM (18 hours ago) 2:00 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
SPARSEMEM currently tries to allocate each section's page extensions
from the buddy allocator before falling back to vmalloc. Boot-time
section allocation happens after vmalloc is available, and memory
hotplug allocations do not require physically contiguous backing.

Use vzalloc_node() directly and remove the buddy-specific allocation,
free, and kmemleak bookkeeping. This leaves one runtime backing type
and lets a later change add memblock as the distinct boot-time path
to simplify the code.

Signed-off-by: Muchun Song <songm...@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>
Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>
---
v3:
- Collect Reviewed-by from Mike Rapoport and Chris Down
- Collect Tested-by from Chris Down

v2:
- Add a preparatory cleanup that always uses vmalloc for SPARSEMEM
section page extensions (suggested by Mike Rapoport)
---
mm/page_ext.c | 21 +++------------------
1 file changed, 3 insertions(+), 18 deletions(-)

diff --git a/mm/page_ext.c b/mm/page_ext.c
index 39a4464fc979..6196bab6868c 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -290,15 +290,9 @@ static struct page_ext *lookup_page_ext(const struct page *page)

static void *__meminit alloc_page_ext(size_t size, int nid)
{
- gfp_t flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN;
- void *addr = NULL;
-
- addr = alloc_pages_exact_nid(nid, size, flags);
- if (addr)
- kmemleak_alloc(addr, size, 1, flags);
- else
- addr = vzalloc_node(size, nid);
+ void *addr;

+ addr = vzalloc_node(size, nid);
if (addr)
memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));

@@ -344,19 +338,10 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
static void free_page_ext(void *addr)
{
size_t table_size;
- struct page *page;

table_size = page_ext_size * PAGES_PER_SECTION;
memmap_pages_add(-1L * (DIV_ROUND_UP(table_size, PAGE_SIZE)));
-
- if (is_vmalloc_addr(addr)) {
- vfree(addr);
- } else {
- page = virt_to_page(addr);
- BUG_ON(PageReserved(page));
- kmemleak_free(addr);
- free_pages_exact(addr, table_size);
- }
+ vfree(addr);
}

static void __free_page_ext(unsigned long pfn)
--
2.54.0

Muchun Song

unread,
2:00 AM (18 hours ago) 2:00 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
FLATMEM uses separate page_ext_init_flatmem() and
page_ext_init_flatmem_late() entry points, while SPARSEMEM performs its
work through page_ext_init(). This exposes the memory model in the
generic boot sequence and obscures the allocation and client
initialization phases.

Rename the early entry point to page_ext_alloc() and use
page_ext_init() for client callbacks in both memory models. SPARSEMEM
keeps its existing late allocation behavior for now; its
page_ext_alloc() remains an empty configuration stub.

This also moves FLATMEM client callbacks from immediately after
kmem_cache_init() to after vmalloc_init(). The callbacks still run
after the buddy and slab allocators are available, as required by
page_owner.

Signed-off-by: Muchun Song <songm...@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>
Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>
---
v3:
- Collect Reviewed-by from Mike Rapoport and Chris Down
- Collect Tested-by from Chris Down

v2:
- Split the common allocation and initialization entry points into a
preparatory patch
- Document the later FLATMEM client callback timing (noted by Chris
Down)
---
include/linux/page_ext.h | 20 +++++---------------
mm/mm_init.c | 7 +------
mm/page_ext.c | 5 ++---
3 files changed, 8 insertions(+), 24 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index 79c53ec45dfa..911a84bbb3db 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -62,13 +62,10 @@ static inline bool early_page_ext_enabled(void)
}

#ifdef CONFIG_SPARSEMEM
-static inline void page_ext_init_flatmem(void)
-{
-}
-extern void page_ext_init(void);
-static inline void page_ext_init_flatmem_late(void)
+static inline void page_ext_alloc(void)
{
}
+void page_ext_init(void);

static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
@@ -79,11 +76,8 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
return next_pfn % PAGES_PER_SECTION;
}
#else
-extern void page_ext_init_flatmem(void);
-extern void page_ext_init_flatmem_late(void);
-static inline void page_ext_init(void)
-{
-}
+void page_ext_alloc(void);
+void page_ext_init(void);

static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
@@ -205,11 +199,7 @@ static inline void page_ext_init(void)
{
}

-static inline void page_ext_init_flatmem_late(void)
-{
-}
-
-static inline void page_ext_init_flatmem(void)
+static inline void page_ext_alloc(void)
{
}

diff --git a/mm/mm_init.c b/mm/mm_init.c
index a9d848c92678..bcc4e7e8cd1f 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2709,7 +2709,7 @@ void __init mm_core_init(void)
* page_ext requires contiguous pages,
* bigger than MAX_PAGE_ORDER unless SPARSEMEM.
*/
- page_ext_init_flatmem();
+ page_ext_alloc();
mem_debugging_and_hardening_init();
kfence_alloc_pool_and_metadata();
report_meminit();
@@ -2725,11 +2725,6 @@ void __init mm_core_init(void)
memblock_free_all();
mem_init();
kmem_cache_init();
- /*
- * page_owner must be initialized after buddy is ready, and also after
- * slab is ready so that stack_depot_init() works properly
- */
- page_ext_init_flatmem_late();
kmemleak_init();
ptlock_cache_init();
pgtable_cache_init();
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 6196bab6868c..aff00f4e1bdd 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -185,7 +185,7 @@ static inline struct page_ext *get_entry(void *base, unsigned long index)
}

#ifndef CONFIG_SPARSEMEM
-void __init page_ext_init_flatmem_late(void)
+void __init page_ext_init(void)
{
invoke_init_callbacks();
}
@@ -243,9 +243,8 @@ static int __init alloc_node_page_ext(int nid)
return 0;
}

-void __init page_ext_init_flatmem(void)
+void __init page_ext_alloc(void)
{
-
int nid, fail;

if (!invoke_need_callbacks())
--
2.54.0

Muchun Song

unread,
2:01 AM (18 hours ago) 2:01 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN needs page extensions when kmsan_init_shadow() installs per-page
metadata, before memblock_free_all() releases memory to the buddy
allocator. FLATMEM makes page extensions available by this point,
whereas SPARSEMEM initializes them later in boot.

Allocate boot-time SPARSEMEM page extensions from memblock to match
FLATMEM's initialization timing. This gives early clients the same
page_ext availability guarantee across both memory models, while runtime
memory hotplug continues to use vmalloc.

Signed-off-by: Muchun Song <songm...@bytedance.com>
Reviewed-by: Chris Down <ch...@chrisdown.name>
Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>
---
v3:
- Use DIV_ROUND_UP() for page accounting instead of relying on the
caller's size alignment (suggested by Mike Rapoport)
- Collect Reviewed-by from Chris Down and Mike Rapoport

v2:
- Allocate all boot-time SPARSEMEM page extensions from memblock
(suggested by Mike Rapoport)
- Keep memblock allocations above DMA and suppress per-section free
messages (suggested by Chris Down)
---
include/linux/page_ext.h | 9 +--
mm/mm_init.c | 4 --
mm/page_ext.c | 117 +++++++++++++++++++--------------------
3 files changed, 59 insertions(+), 71 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index 911a84bbb3db..d00320106a58 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -61,12 +61,10 @@ static inline bool early_page_ext_enabled(void)
return early_page_ext;
}

-#ifdef CONFIG_SPARSEMEM
-static inline void page_ext_alloc(void)
-{
-}
+void page_ext_alloc(void);
void page_ext_init(void);

+#ifdef CONFIG_SPARSEMEM
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
/*
@@ -76,9 +74,6 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
return next_pfn % PAGES_PER_SECTION;
}
#else
-void page_ext_alloc(void);
-void page_ext_init(void);
-
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
return true;
diff --git a/mm/mm_init.c b/mm/mm_init.c
index bcc4e7e8cd1f..ea763e299769 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2705,10 +2705,6 @@ void __init mm_core_init(void)
build_all_zonelists(NULL);
page_alloc_init_cpuhp();
alloc_tag_sec_init();
- /*
- * page_ext requires contiguous pages,
- * bigger than MAX_PAGE_ORDER unless SPARSEMEM.
- */
page_ext_alloc();
mem_debugging_and_hardening_init();
kfence_alloc_pool_and_metadata();
diff --git a/mm/page_ext.c b/mm/page_ext.c
index aff00f4e1bdd..cfd20ed72884 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/slab.h>

/*
* struct page extension
@@ -185,11 +186,6 @@ static inline struct page_ext *get_entry(void *base, unsigned long index)
}

#ifndef CONFIG_SPARSEMEM
-void __init page_ext_init(void)
-{
- invoke_init_callbacks();
-}
-
static struct page_ext *lookup_page_ext(const struct page *page)
{
unsigned long pfn = page_to_pfn(page);
@@ -243,24 +239,16 @@ static int __init alloc_node_page_ext(int nid)
return 0;
}

-void __init page_ext_alloc(void)
+static int __init __page_ext_alloc(void)
{
- int nid, fail;
-
- if (!invoke_need_callbacks())
- return;
+ int nid;

- for_each_online_node(nid) {
- fail = alloc_node_page_ext(nid);
- if (fail)
- goto fail;
+ for_each_online_node(nid) {
+ if (alloc_node_page_ext(nid))
+ return -ENOMEM;
}
- pr_info("allocated %ld bytes of page_ext\n", total_usage);
- return;

-fail:
- pr_crit("allocation of page_ext failed.\n");
- panic("Out of memory");
+ return 0;
}

#else /* CONFIG_SPARSEMEM */
@@ -287,10 +275,18 @@ static struct page_ext *lookup_page_ext(const struct page *page)
return get_entry(page_ext, pfn);
}

-static void *__meminit alloc_page_ext(size_t size, int nid)
+static __ref void *alloc_page_ext(size_t size, int nid)
{
void *addr;

+ if (!slab_is_available()) {
+ addr = memblock_alloc_try_nid(size, PAGE_SIZE, __pa(MAX_DMA_ADDRESS),
+ MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+ if (addr)
+ memmap_boot_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));
+ return addr;
+ }
+
addr = vzalloc_node(size, nid);
if (addr)
memmap_pages_add(DIV_ROUND_UP(size, PAGE_SIZE));
@@ -309,7 +305,7 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)
if (section->page_ext)
return 0;

- table_size = page_ext_size * PAGES_PER_SECTION;
+ table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
base = alloc_page_ext(table_size, nid);

/*
@@ -336,11 +332,19 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid)

static void free_page_ext(void *addr)
{
- size_t table_size;
+ size_t table_size = PAGE_ALIGN(page_ext_size * PAGES_PER_SECTION);
+ long nr_pages = table_size / PAGE_SIZE;
+
+ if (is_vmalloc_addr(addr)) {
+ memmap_pages_add(-nr_pages);
+ vfree(addr);
+ return;
+ }

- table_size = page_ext_size * PAGES_PER_SECTION;
- memmap_pages_add(-1L * (DIV_ROUND_UP(table_size, PAGE_SIZE)));
- vfree(addr);
+ VM_WARN_ON_ONCE(!PageReserved(virt_to_page(addr)));
+ memmap_boot_pages_add(-nr_pages);
+ kmemleak_free_part_phys(__pa(addr), table_size);
+ free_reserved_area(addr, addr + table_size, -1, NULL);
}

static void __free_page_ext(unsigned long pfn)
@@ -454,52 +458,45 @@ static int __meminit page_ext_callback(struct notifier_block *self,
return notifier_from_errno(ret);
}

-void __init page_ext_init(void)
+static int __init __page_ext_alloc(void)
{
- unsigned long pfn;
- int nid;
-
- if (!invoke_need_callbacks())
- return;
-
- for_each_node_state(nid, N_MEMORY) {
- unsigned long start_pfn, end_pfn;
-
- start_pfn = node_start_pfn(nid);
- end_pfn = node_end_pfn(nid);
- /*
- * start_pfn and end_pfn may not be aligned to SECTION and the
- * page->flags of out of node pages are not initialized. So we
- * scan [start_pfn, the biggest section's pfn < end_pfn) here.
- */
- for (pfn = start_pfn; pfn < end_pfn;
- pfn = ALIGN(pfn + 1, PAGES_PER_SECTION)) {
+ unsigned long start, end, pfn;
+ int i, nid;

+ for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) {
+ for (pfn = start; pfn < end; pfn = SECTION_ALIGN_UP(pfn + 1)) {
if (!pfn_valid(pfn))
continue;
- /*
- * Nodes's pfns can be overlapping.
- * We know some arch can have a nodes layout such as
- * -------------pfn-------------->
- * N0 | N1 | N2 | N0 | N1 | N2|....
- */
- if (pfn_to_nid(pfn) != nid)
- continue;
if (init_section_page_ext(pfn, nid))
- goto oom;
- cond_resched();
+ return -ENOMEM;
}
}
- hotplug_memory_notifier(page_ext_callback, DEFAULT_CALLBACK_PRI);
- pr_info("allocated %ld bytes of page_ext\n", total_usage);
- invoke_init_callbacks();
- return;

-oom:
- panic("Out of memory");
+ return 0;
+}
+
+#endif
+
+void __init page_ext_alloc(void)
+{
+ if (!invoke_need_callbacks())
+ return;
+ if (__page_ext_alloc())
+ panic("Out of memory");
+
+ pr_info("allocated %ld bytes of page_ext\n", total_usage);
}

+void __init page_ext_init(void)
+{
+ if (!page_ext_size)
+ return;
+
+#ifdef CONFIG_SPARSEMEM
+ hotplug_memory_notifier(page_ext_callback, DEFAULT_CALLBACK_PRI);
#endif
+ invoke_init_callbacks();
+}

/**
* page_ext_lookup() - Lookup a page extension for a PFN.
--
2.54.0

Muchun Song

unread,
2:01 AM (18 hours ago) 2:01 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
The KMSAN runtime needs an unchecked page extension lookup that can be
inlined into its uninstrumented compilation unit while page_ext.c remains
instrumented.

Provide an inline lookup for this use case while keeping the existing RCU
checks for other callers.

Signed-off-by: Muchun Song <songm...@bytedance.com>
Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>
---
v3:
- Collect Reviewed-by and Tested-by from Chris Down

v2:
- Add a preparatory patch that inlines the unchecked lookup (suggested
by Chris Down)
- Use a PFN interface to avoid architecture header-order dependencies
---
include/linux/page_ext.h | 46 ++++++++++++++++++++++++++++++
mm/page_ext.c | 60 +++-------------------------------------
2 files changed, 50 insertions(+), 56 deletions(-)

diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h
index d00320106a58..166f8be11564 100644
--- a/include/linux/page_ext.h
+++ b/include/linux/page_ext.h
@@ -64,7 +64,14 @@ static inline bool early_page_ext_enabled(void)
void page_ext_alloc(void);
void page_ext_init(void);

+static __always_inline struct page_ext *page_ext_entry(void *base, unsigned long index)
+{
+ return base + page_ext_size * index;
+}
+
#ifdef CONFIG_SPARSEMEM
+#define PAGE_EXT_INVALID 0x1
+
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
/*
@@ -73,11 +80,50 @@ static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
*/
return next_pfn % PAGES_PER_SECTION;
}
+
+static __always_inline bool page_ext_invalid(struct page_ext *page_ext)
+{
+ return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID);
+}
+
+static __always_inline struct page_ext *__page_ext_lookup(unsigned long pfn)
+{
+ struct mem_section *section = __pfn_to_section(pfn);
+ struct page_ext *page_ext = READ_ONCE(section->page_ext);
+
+ /*
+ * The sanity checks the page allocator does upon freeing a page can
+ * reach here before the page_ext arrays are allocated when feeding a
+ * range of pages to the allocator for the first time during bootup or
+ * memory hotplug.
+ */
+ if (page_ext_invalid(page_ext))
+ return NULL;
+ return page_ext_entry(page_ext, pfn);
+}
#else
static inline bool page_ext_iter_next_fast_possible(unsigned long next_pfn)
{
return true;
}
+
+static __always_inline struct page_ext *__page_ext_lookup(unsigned long pfn)
+{
+ int nid = pfn_to_nid(pfn);
+ struct page_ext *base = NODE_DATA(nid)->node_page_ext;
+ unsigned long index;
+
+ /*
+ * The sanity checks the page allocator does upon freeing a page can
+ * reach here before the page_ext arrays are allocated when feeding a
+ * range of pages to the allocator for the first time during bootup or
+ * memory hotplug.
+ */
+ if (unlikely(!base))
+ return NULL;
+ index = pfn - round_down(node_start_pfn(nid), MAX_ORDER_NR_PAGES);
+ return page_ext_entry(base, index);
+}
#endif

extern struct page_ext *page_ext_get(const struct page *page);
diff --git a/mm/page_ext.c b/mm/page_ext.c
index cfd20ed72884..63af3d62ed74 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -63,10 +63,6 @@
* can utilize this callback to initialize the state of it correctly.
*/

-#ifdef CONFIG_SPARSEMEM
-#define PAGE_EXT_INVALID (0x1)
-#endif
-
#if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT)
static bool need_page_idle(void)
{
@@ -180,33 +176,7 @@ static void __init invoke_init_callbacks(void)
}
}

-static inline struct page_ext *get_entry(void *base, unsigned long index)
-{
- return base + page_ext_size * index;
-}
-
#ifndef CONFIG_SPARSEMEM
-static struct page_ext *lookup_page_ext(const struct page *page)
-{
- unsigned long pfn = page_to_pfn(page);
- unsigned long index;
- struct page_ext *base;
-
- WARN_ON_ONCE(!rcu_read_lock_held());
- base = NODE_DATA(page_to_nid(page))->node_page_ext;
- /*
- * The sanity checks the page allocator does upon freeing a
- * page can reach here before the page_ext arrays are
- * allocated when feeding a range of pages to the allocator
- * for the first time during bootup or memory hotplug.
- */
- if (unlikely(!base))
- return NULL;
- index = pfn - round_down(node_start_pfn(page_to_nid(page)),
- MAX_ORDER_NR_PAGES);
- return get_entry(base, index);
-}
-
static int __init alloc_node_page_ext(int nid)
{
struct page_ext *base;
@@ -252,29 +222,6 @@ static int __init __page_ext_alloc(void)
}

#else /* CONFIG_SPARSEMEM */
-static bool page_ext_invalid(struct page_ext *page_ext)
-{
- return !page_ext || (((unsigned long)page_ext & PAGE_EXT_INVALID) == PAGE_EXT_INVALID);
-}
-
-static struct page_ext *lookup_page_ext(const struct page *page)
-{
- unsigned long pfn = page_to_pfn(page);
- struct mem_section *section = __pfn_to_section(pfn);
- struct page_ext *page_ext = READ_ONCE(section->page_ext);
-
- WARN_ON_ONCE(!rcu_read_lock_held());
- /*
- * The sanity checks the page allocator does upon freeing a
- * page can reach here before the page_ext arrays are
- * allocated when feeding a range of pages to the allocator
- * for the first time during bootup or memory hotplug.
- */
- if (page_ext_invalid(page_ext))
- return NULL;
- return get_entry(page_ext, pfn);
-}
-
static __ref void *alloc_page_ext(size_t size, int nid)
{
void *addr;
@@ -365,7 +312,7 @@ static void __free_page_ext(unsigned long pfn)
base = (void *)base - PAGE_EXT_INVALID;
WRITE_ONCE(ms->page_ext, NULL);

- base = get_entry(base, pfn);
+ base = page_ext_entry(base, pfn);
free_page_ext(base);
}

@@ -508,7 +455,8 @@ void __init page_ext_init(void)
*/
struct page_ext *page_ext_lookup(unsigned long pfn)
{
- return lookup_page_ext(pfn_to_page(pfn));
+ WARN_ON_ONCE(!rcu_read_lock_held());
+ return __page_ext_lookup(pfn);
}

/**
@@ -527,7 +475,7 @@ struct page_ext *page_ext_get(const struct page *page)
struct page_ext *page_ext;

rcu_read_lock();
- page_ext = lookup_page_ext(page);
+ page_ext = page_ext_lookup(page_to_pfn(page));
if (!page_ext) {
rcu_read_unlock();
return NULL;
--
2.54.0

Muchun Song

unread,
2:01 AM (18 hours ago) 2:01 AM
to Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org, Muchun Song
KMSAN stores each page's shadow and origin pointers in struct page.
Boot-time gigantic HugeTLB pages are reinitialized after
kmsan_init_shadow() installs these pointers, dropping the mappings and
leaving the allocated metadata pages unreachable.

Move the pointers to page_ext, whose boot-time storage is available
before kmsan_init_shadow() and survives struct page reinitialization.
Use the inline unchecked lookup under notrace preemption so metadata
lookup cannot recurse through instrumented hooks.

This shrinks struct page from 80 to 64 bytes on x86-64 KMSAN builds,
making its size a power of two and allowing HugeTLB Vmemmap Optimization
with KMSAN.

Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO")
Signed-off-by: Muchun Song <songm...@bytedance.com>
Reported-by: Sashiko <sashi...@kernel.org>
Closes: https://sashiko.dev/#/bug/linux-ac161ba4-30cc-4bd3-b93c-ab7af8a6f64f
Reviewed-by: Chris Down <ch...@chrisdown.name>
Tested-by: Chris Down <ch...@chrisdown.name>
---
v3:
- Collect Reviewed-by and Tested-by from Chris Down

v2:
- Update __mm_zero_struct_page() for the smaller struct page.
- Keep the unchecked lookup inline in KMSAN and leave page_ext.c
instrumented (suggested by Chris Down)
- Document page_ext metadata storage and KMSAN HVO support (suggested
by Chris Down)
- Use the common boot-time page_ext allocation instead of need_early
---
Documentation/dev-tools/kmsan.rst | 8 +++---
include/linux/kmsan.h | 2 ++
include/linux/mm.h | 12 +++------
include/linux/mm_types.h | 12 ---------
lib/Kconfig.kmsan | 1 +
mm/kmsan/shadow.c | 45 ++++++++++++++++++++++++++-----
mm/page_ext.c | 4 +++
7 files changed, 52 insertions(+), 32 deletions(-)

diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/kmsan.rst
index 0dc668b183f6..5330d21d141d 100644
--- a/Documentation/dev-tools/kmsan.rst
+++ b/Documentation/dev-tools/kmsan.rst
@@ -376,13 +376,11 @@ Metadata allocation

There are several places in the kernel for which the metadata is stored.

-1. Each ``struct page`` instance contains two pointers to its shadow and
-origin pages::
+1. Each page frame has a corresponding ``struct page_metadata`` entry containing
+two pointers to its shadow and origin pages::

- struct page {
- ...
+ struct page_metadata {
struct page *shadow, *origin;
- ...
};

At boot-time, the kernel allocates shadow and origin pages for every available
diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h
index 7da9fd506b39..08fb79566235 100644
--- a/include/linux/kmsan.h
+++ b/include/linux/kmsan.h
@@ -15,12 +15,14 @@
#include <linux/types.h>

struct page;
+struct page_ext_operations;
struct kmem_cache;
struct task_struct;
struct scatterlist;
struct urb;

#ifdef CONFIG_KMSAN
+extern struct page_ext_operations kmsan_ops;

/**
* kmsan_task_create() - Initialize KMSAN state for the task.
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 3578f53ea9cc..e04cd6289546 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -134,7 +134,7 @@ extern int mmap_rnd_compat_bits __read_mostly;
* define their own version of this macro in <asm/pgtable.h>
*/
#if BITS_PER_LONG == 64
-/* This function must be updated when the size of struct page grows above 96
+/* This function must be updated when the size of struct page grows above 80
* or reduces below 56. The idea that compiler optimizes out switch()
* statement, and only leaves move/store instructions. Also the compiler can
* combine write statements if they are both assignments and can be reordered,
@@ -145,18 +145,12 @@ static inline void __mm_zero_struct_page(struct page *page)
{
unsigned long *_pp = (void *)page;

- /* Check that struct page is either 56, 64, 72, 80, 88 or 96 bytes */
+ /* Check that struct page is either 56, 64, 72 or 80 bytes */
BUILD_BUG_ON(sizeof(struct page) & 7);
BUILD_BUG_ON(sizeof(struct page) < 56);
- BUILD_BUG_ON(sizeof(struct page) > 96);
+ BUILD_BUG_ON(sizeof(struct page) > 80);

switch (sizeof(struct page)) {
- case 96:
- _pp[11] = 0;
- fallthrough;
- case 88:
- _pp[10] = 0;
- fallthrough;
case 80:
_pp[9] = 0;
fallthrough;
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e..dc73f9f5a51b 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -208,18 +208,6 @@ struct page {
#ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS
int _last_cpupid;
#endif
-
-#ifdef CONFIG_KMSAN
- /*
- * KMSAN metadata for this page:
- * - shadow page: every bit indicates whether the corresponding
- * bit of the original page is initialized (0) or not (1);
- * - origin page: every 4 bytes contain an id of the stack trace
- * where the uninitialized value was created.
- */
- struct page *kmsan_shadow;
- struct page *kmsan_origin;
-#endif
} _struct_page_alignment;

/*
diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan
index cae1ddcc18e1..015e85c70793 100644
--- a/lib/Kconfig.kmsan
+++ b/lib/Kconfig.kmsan
@@ -10,6 +10,7 @@ config KMSAN
depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER
depends on DEBUG_KERNEL && !KASAN && !KCSAN
depends on !PREEMPT_RT
+ select PAGE_EXTENSION
select STACKDEPOT
select STACKDEPOT_ALWAYS_INIT
help
diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c
index 300b48818af1..fa2a403d87c2 100644
--- a/mm/kmsan/shadow.c
+++ b/mm/kmsan/shadow.c
@@ -13,6 +13,7 @@
#include <linux/cacheflush.h>
#include <linux/memblock.h>
#include <linux/mm_types.h>
+#include <linux/page_ext.h>
#include <linux/slab.h>
#include <linux/smp.h>
#include <linux/stddef.h>
@@ -24,23 +25,55 @@
#define KMSAN_META_MASK_SHADOW BIT(0)
#define KMSAN_META_MASK_ORIGIN BIT(1)

+/*
+ * KMSAN metadata for this page:
+ * - shadow page: every bit indicates whether the corresponding
+ * bit of the original page is initialized (0) or not (1);
+ * - origin page: every 4 bytes contain an id of the stack trace
+ * where the uninitialized value was created.
+ */
struct page_metadata {
struct page *shadow;
struct page *origin;
};

+static __init bool need_page_metadata(void)
+{
+ return true;
+}
+
+struct page_ext_operations kmsan_ops = {
+ .size = sizeof(struct page_metadata),
+ .need = need_page_metadata,
+};
+
static inline struct page_metadata get_page_metadata(struct page *page)
{
- return (struct page_metadata) {
- .shadow = page->kmsan_shadow,
- .origin = page->kmsan_origin,
- };
+ struct page_metadata meta = {};
+ struct page_ext *page_ext;
+
+ /*
+ * Avoid instrumented preemption debug and tracing hooks while resolving
+ * KMSAN metadata, as they would recurse back into metadata lookup.
+ */
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ meta = *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops);
+ preempt_enable_no_resched_notrace();
+
+ return meta;
}

static inline void set_page_metadata(struct page *page, struct page_metadata meta)
{
- page->kmsan_shadow = meta.shadow;
- page->kmsan_origin = meta.origin;
+ struct page_ext *page_ext;
+
+ preempt_disable_notrace();
+ page_ext = __page_ext_lookup(page_to_pfn(page));
+ if (page_ext)
+ *(struct page_metadata *)page_ext_data(page_ext, &kmsan_ops) = meta;
+ preempt_enable_no_resched_notrace();
}

static inline void *shadow_ptr_for(struct page_metadata meta)
diff --git a/mm/page_ext.c b/mm/page_ext.c
index 63af3d62ed74..985595bdf420 100644
--- a/mm/page_ext.c
+++ b/mm/page_ext.c
@@ -12,6 +12,7 @@
#include <linux/rcupdate.h>
#include <linux/pgalloc_tag.h>
#include <linux/iommu-debug-pagealloc.h>
+#include <linux/kmsan.h>
#include <linux/slab.h>

/*
@@ -87,6 +88,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {
#ifdef CONFIG_PAGE_TABLE_CHECK
&page_table_check_ops,
#endif
+#ifdef CONFIG_KMSAN
+ &kmsan_ops,
+#endif
#ifdef CONFIG_IOMMU_DEBUG_PAGEALLOC
&page_iommu_debug_ops,
#endif
--
2.54.0

Mike Rapoport

unread,
2:57 AM (17 hours ago) 2:57 AM
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
On Thu, Nov 19, 2026 at 02:58:11PM +0800, Muchun Song wrote:
> The KMSAN runtime needs an unchecked page extension lookup that can be
> inlined into its uninstrumented compilation unit while page_ext.c remains
> instrumented.
>
> Provide an inline lookup for this use case while keeping the existing RCU
> checks for other callers.
>
> Signed-off-by: Muchun Song <songm...@bytedance.com>
> Reviewed-by: Chris Down <ch...@chrisdown.name>
> Tested-by: Chris Down <ch...@chrisdown.name>

Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>

> ---
> v3:
> - Collect Reviewed-by and Tested-by from Chris Down
>
> v2:
> - Add a preparatory patch that inlines the unchecked lookup (suggested
> by Chris Down)
> - Use a PFN interface to avoid architecture header-order dependencies
> ---
> include/linux/page_ext.h | 46 ++++++++++++++++++++++++++++++
> mm/page_ext.c | 60 +++-------------------------------------
> 2 files changed, 50 insertions(+), 56 deletions(-)

--
Sincerely yours,
Mike.

Mike Rapoport

unread,
3:00 AM (17 hours ago) 3:00 AM
to Muchun Song, Alexander Potapenko, Andrew Morton, Marco Elver, Dmitry Vyukov, Chris Down, David Hildenbrand, Liam R . Howlett, Vlastimil Babka, Suren Baghdasaryan, Michal Hocko, Johannes Weiner, Brendan Jackman, Zi Yan, Muchun Song, Joerg Roedel, Lu Baolu, Pranjal Shrivastava, Ketan, Mostafa Saleh, Sang-Heon Jeon, Jonathan Corbet, Shuah Khan, Randy Dunlap, Sashiko, kasa...@googlegroups.com, work...@vger.kernel.org, linu...@vger.kernel.org, linux-...@vger.kernel.org, linu...@kvack.org
On Thu, Nov 19, 2026 at 02:58:07PM +0800, Muchun Song wrote:
> page_ext lays out client payloads consecutively without aligning their
> offsets or the final entry size. This happened to work while the active
> payload sizes kept the stride naturally aligned, but IOMMU debug page
> allocation added a four-byte payload after data requiring eight-byte
> alignment.
>
> When page owner and IOMMU debug page allocation are both enabled on
> x86-64, struct page_ext, struct page_owner, and the IOMMU payload use
> 8, 64, and 4 bytes respectively. This produces a 76-byte entry stride,
> leaving struct page_ext and page_owner data in every other entry only
> four-byte aligned instead of eight-byte aligned.
>
> Align each client offset and round the final entry size to the largest
> active alignment so every page_ext entry preserves the alignment
> required by all active clients.
>
> Fixes: 466ae6978a5b ("iommu: Add page_ext for IOMMU_DEBUG_PAGEALLOC")
> Signed-off-by: Muchun Song <songm...@bytedance.com>
> Reviewed-by: Chris Down <ch...@chrisdown.name>
> Tested-by: Chris Down <ch...@chrisdown.name>

Reviewed-by: Mike Rapoport (Microsoft) <rp...@kernel.org>

> ---
> v3:
> - Cap client alignment at u64 alignment to preserve 64-bit alignment on
> 32-bit architectures (reported by Sashiko)
> - Open-code the lowest power-of-two factor for readability (suggested by
> Andrew Morton)
> - Use a local for the u64 alignment (suggested by Mike Rapoport)
> - Simplify the commit message to focus on the alignment guarantee
> - Collect Reviewed-by and Tested-by from Chris Down
>
> v2:
> - New patch to fix the existing page_ext client alignment bug
> ---
> mm/page_ext.c | 26 ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)

--
Sincerely yours,
Mike.
Reply all
Reply to author
Forward
0 new messages