[PATCH 00/20] kasan: boot parameters for hardware tag-based mode

[Andrey Konovalov]

=== Overview

Hardware tag-based KASAN mode [1] is intended to eventually be used in
production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.

This patchset adds a few boot parameters for hardware tag-based KASAN that
allow to disable or otherwise control particular KASAN features.

There's another planned patchset what will further optimize hardware
tag-based KASAN, provide proper benchmarking and tests, and will fully
enable tag-based KASAN for production use.

Hardware tag-based KASAN relies on arm64 Memory Tagging Extension (MTE)
[2] to perform memory and pointer tagging. Please see [3] and [4] for
detailed analysis of how MTE helps to fight memory safety problems.

The features that can be controlled are:

1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.

The patch titled "kasan: add and integrate kasan boot parameters" of this
series adds a few new boot parameters.

kasan.mode allows to choose one of three main modes:

- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled

The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:

- kasan.stack=off/on - enable stacks collection
(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)

If kasan.mode parameter is not provided, it defaults to full when
CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.

It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by
the Android GKI (Generic Kernel Image) initiative.

=== Benchmarks

For now I've only performed a few simple benchmarks such as measuring
kernel boot time and slab memory usage after boot. There's an upcoming
patchset which will optimize KASAN further and include more detailed
benchmarking results.

The benchmarks were performed in QEMU and the results below exclude the
slowdown caused by QEMU memory tagging emulation (as it's different from
the slowdown that will be introduced by hardware and is therefore
irrelevant).

KASAN_HW_TAGS=y + kasan.mode=off introduces no performance or memory
impact compared to KASAN_HW_TAGS=n.

kasan.mode=prod (manually excluding tagging) introduces 3% of performance
and no memory impact (except memory used by hardware to store tags)
compared to kasan.mode=off.

kasan.mode=full has about 40% performance and 30% memory impact over
kasan.mode=prod. Both come from alloc/free stack collection.

=== Notes

This patchset is available here:

https://github.com/xairy/linux/tree/up-boot-mte-v1

and on Gerrit here:

https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/3707

This patchset is based on v8 of "kasan: add hardware tag-based mode for
arm64" patchset [1].

For testing in QEMU hardware tag-based KASAN requires:

1. QEMU built from master [6] (use "-machine virt,mte=on -cpu max" arguments
to run).
2. GCC version 10.

[1] https://lkml.org/lkml/2020/11/4/1208
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] https://arxiv.org/pdf/1802.09517.pdf
[4] https://github.com/microsoft/MSRC-Security-Research/blob/master/papers/2020/Security%20analysis%20of%20memory%20tagging.pdf
[5] https://source.android.com/devices/architecture/kernel/generic-kernel-image
[6] https://github.com/qemu/qemu

=== History

Changes RFC v2 -> v1:
- Rebrand the patchset from fully enabling production use to partially
addressing that; another optimization and testing patchset will be
required.
- Rebase onto v8 of KASAN_HW_TAGS series.
- Fix "ASYNC" -> "async" typo.
- Rework depends condition for VMAP_STACK and update config text.
- Remove unneeded reset_tag() macro, use kasan_reset_tag() instead.
- Rename kasan.stack to kasan.stacks to avoid confusion with stack
instrumentation.
- Introduce kasan_stack_collection_enabled() and kasan_is_enabled()
helpers.
- Simplify kasan_stack_collection_enabled() usage.
- Rework SLAB_KASAN flag and metadata allocation (see the corresponding
patch for details).
- Allow cache merging with KASAN_HW_TAGS when kasan.stacks is off.
- Use sync mode dy default for both prod and full KASAN modes.
- Drop kasan.trap=sync/async boot parameter, as async mode isn't supported
yet.
- Choose prod or full mode depending on CONFIG_DEBUG_KERNEL when no
kasan.mode boot parameter is provided.
- Drop krealloc optimization changes, those will be included in a separate
patchset.
- Update KASAN documentation to mention boot parameters.

Changes RFC v1 -> RFC v2:
- Rework boot parameters.
- Drop __init from empty kasan_init_tags() definition.
- Add cpu_supports_mte() helper that can be used during early boot and use
it in kasan_init_tags()
- Lots of new KASAN optimization commits.

Andrey Konovalov (20):
kasan: simplify quarantine_put call site
kasan: rename get_alloc/free_info
kasan: introduce set_alloc_info
kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK
kasan: allow VMAP_STACK for HW_TAGS mode
kasan: remove __kasan_unpoison_stack
kasan: inline kasan_reset_tag for tag-based modes
kasan: inline random_tag for HW_TAGS
kasan: inline kasan_poison_memory and check_invalid_free
kasan: inline and rename kasan_unpoison_memory
kasan: add and integrate kasan boot parameters
kasan, mm: check kasan_enabled in annotations
kasan: simplify kasan_poison_kfree
kasan, mm: rename kasan_poison_kfree
kasan: don't round_up too much
kasan: simplify assign_tag and set_tag calls
kasan: clarify comment in __kasan_kfree_large
kasan: clean up metadata allocation and usage
kasan, mm: allow cache merging with no metadata
kasan: update documentation

Documentation/dev-tools/kasan.rst | 180 ++++++++++++--------
arch/Kconfig | 8 +-
arch/arm64/kernel/sleep.S | 2 +-
arch/x86/kernel/acpi/wakeup_64.S | 2 +-
include/linux/kasan.h | 253 +++++++++++++++++++++------
include/linux/mm.h | 22 ++-
kernel/fork.c | 2 +-
mm/kasan/common.c | 274 ++++++++++++++++++------------
mm/kasan/generic.c | 27 +--
mm/kasan/hw_tags.c | 171 ++++++++++++++++---
mm/kasan/kasan.h | 113 ++++++++----
mm/kasan/quarantine.c | 13 +-
mm/kasan/report.c | 61 ++++---
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 13 +-
mm/kasan/shadow.c | 5 +-
mm/kasan/sw_tags.c | 17 +-
mm/mempool.c | 2 +-
mm/slab_common.c | 13 +-
19 files changed, 816 insertions(+), 364 deletions(-)

--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Move get_free_info() call into quarantine_put() to simplify the call site.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>
Link: https://linux-review.googlesource.com/id/Iab0f04e7ebf8d83247024b7190c67c3c34c7940f
---
mm/kasan/common.c | 2 +-
mm/kasan/kasan.h | 5 ++---
mm/kasan/quarantine.c | 3 ++-
3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 2bb0ef6da6bd..5712c66c11c1 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -308,7 +308,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

kasan_set_free_info(cache, object, tag);

- quarantine_put(get_free_info(cache, object), cache);
+ quarantine_put(cache, object);

return IS_ENABLED(CONFIG_KASAN_GENERIC);
}
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index b0a57d8f9803..994be9979ffd 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -214,12 +214,11 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

#if defined(CONFIG_KASAN_GENERIC) && \
(defined(CONFIG_SLAB) || defined(CONFIG_SLUB))
-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache);
+void quarantine_put(struct kmem_cache *cache, void *object);
void quarantine_reduce(void);
void quarantine_remove_cache(struct kmem_cache *cache);
#else
-static inline void quarantine_put(struct kasan_free_meta *info,
- struct kmem_cache *cache) { }
+static inline void quarantine_put(struct kmem_cache *cache, void *object) { }
static inline void quarantine_reduce(void) { }
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif
diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index 580ff5610fc1..a0792f0d6d0f 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -161,11 +161,12 @@ static void qlist_free_all(struct qlist_head *q, struct kmem_cache *cache)
qlist_init(q);
}

-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache)
+void quarantine_put(struct kmem_cache *cache, void *object)
{
unsigned long flags;
struct qlist_head *q;
struct qlist_head temp = QLIST_INIT;
+ struct kasan_free_meta *info = get_free_info(cache, object);

/*
* Note: irq must be disabled until after we move the batch to the
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Rename get_alloc_info() and get_free_info() to kasan_get_alloc_meta()
and kasan_get_free_meta() to better reflect what those do and avoid
confusion with kasan_set_free_info().

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ib6e4ba61c8b12112b403d3479a9799ac8fff8de1
---
mm/kasan/common.c | 16 ++++++++--------
mm/kasan/generic.c | 12 ++++++------
mm/kasan/hw_tags.c | 4 ++--
mm/kasan/kasan.h | 8 ++++----
mm/kasan/quarantine.c | 4 ++--
mm/kasan/report.c | 12 ++++++------
mm/kasan/report_sw_tags.c | 2 +-
mm/kasan/sw_tags.c | 4 ++--
8 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 5712c66c11c1..8fd04415d8f4 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -175,14 +175,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
sizeof(struct kasan_free_meta) : 0);
}

-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object)
{
return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
@@ -259,13 +259,13 @@ static u8 assign_tag(struct kmem_cache *cache, const void *object,
void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
const void *object)
{
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;

if (!(cache->flags & SLAB_KASAN))
return (void *)object;

- alloc_info = get_alloc_info(cache, object);
- __memset(alloc_info, 0, sizeof(*alloc_info));
+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));
@@ -345,7 +345,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
KASAN_KMALLOC_REDZONE);

if (cache->flags & SLAB_KASAN)
- kasan_set_track(&get_alloc_info(cache, object)->alloc_track, flags);
+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

return set_tag(object, tag);
}
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index adb254df1b1d..d259e4c3aefd 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -329,7 +329,7 @@ void kasan_record_aux_stack(void *addr)
{
struct page *page = kasan_addr_to_page(addr);
struct kmem_cache *cache;
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;
void *object;

if (!(page && PageSlab(page)))
@@ -337,13 +337,13 @@ void kasan_record_aux_stack(void *addr)

cache = page->slab_cache;
object = nearest_obj(cache, page, addr);
- alloc_info = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);

/*
* record the last two call_rcu() call stacks.
*/
- alloc_info->aux_stack[1] = alloc_info->aux_stack[0];
- alloc_info->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
+ alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];
+ alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
}

void kasan_set_free_info(struct kmem_cache *cache,
@@ -351,7 +351,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
{
struct kasan_free_meta *free_meta;

- free_meta = get_free_info(cache, object);
+ free_meta = kasan_get_free_meta(cache, object);
kasan_set_track(&free_meta->free_track, GFP_NOWAIT);

/*
@@ -365,5 +365,5 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;
- return &get_free_info(cache, object)->free_track;
+ return &kasan_get_free_meta(cache, object)->free_track;
}
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 25ae7b43db87..d858aeb7387f 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -62,7 +62,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
}

@@ -71,6 +71,6 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);
return &alloc_meta->free_track[0];
}
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 994be9979ffd..5513b4685007 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -148,10 +148,10 @@ struct kasan_free_meta {
#endif
};

-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object);
-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object);
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object);
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object);

void kasan_poison_memory(const void *address, size_t size, u8 value);

diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index a0792f0d6d0f..0da3d37e1589 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -166,7 +166,7 @@ void quarantine_put(struct kmem_cache *cache, void *object)

unsigned long flags;
struct qlist_head *q;
struct qlist_head temp = QLIST_INIT;

- struct kasan_free_meta *info = get_free_info(cache, object);
+ struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);

/*
* Note: irq must be disabled until after we move the batch to the

@@ -179,7 +179,7 @@ void quarantine_put(struct kmem_cache *cache, void *object)
local_irq_save(flags);

q = this_cpu_ptr(&cpu_quarantine);
- qlist_put(q, &info->quarantine_link, cache->size);
+ qlist_put(q, &meta->quarantine_link, cache->size);
if (unlikely(q->bytes > QUARANTINE_PERCPU_SIZE)) {
qlist_move_all(q, &temp);

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index ce06005d4052..0cac53a57c14 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -164,12 +164,12 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,
static void describe_object(struct kmem_cache *cache, void *object,
const void *addr, u8 tag)
{
- struct kasan_alloc_meta *alloc_info = get_alloc_info(cache, object);
+ struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);

if (cache->flags & SLAB_KASAN) {
struct kasan_track *free_track;

- print_track(&alloc_info->alloc_track, "Allocated");
+ print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");
free_track = kasan_get_free_track(cache, object, tag);
if (free_track) {
@@ -178,14 +178,14 @@ static void describe_object(struct kmem_cache *cache, void *object,
}

#ifdef CONFIG_KASAN_GENERIC
- if (alloc_info->aux_stack[0]) {
+ if (alloc_meta->aux_stack[0]) {
pr_err("Last call_rcu():\n");
- print_stack(alloc_info->aux_stack[0]);
+ print_stack(alloc_meta->aux_stack[0]);
pr_err("\n");
}
- if (alloc_info->aux_stack[1]) {
+ if (alloc_meta->aux_stack[1]) {
pr_err("Second to last call_rcu():\n");
- print_stack(alloc_info->aux_stack[1]);
+ print_stack(alloc_meta->aux_stack[1]);
pr_err("\n");
}
#endif
diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index aebc44a29e83..317100fd95b9 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -46,7 +46,7 @@ const char *get_bug_type(struct kasan_access_info *info)
if (page && PageSlab(page)) {
cache = page->slab_cache;
object = nearest_obj(cache, page, (void *)addr);
- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);

for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
if (alloc_meta->free_pointer_tag[i] == tag)
diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index dfe707dd8d0d..3bffb489b144 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -174,7 +174,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;
u8 idx = 0;

- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;
@@ -191,7 +191,7 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;
int i = 0;

- alloc_meta = get_alloc_info(cache, object);
+ alloc_meta = kasan_get_alloc_meta(cache, object);

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Add set_alloc_info() helper and move kasan_set_track() into it. This will
simplify the code for one of the upcoming changes.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I0316193cbb4ecc9b87b7c2eee0dd79f8ec908c1a
---
mm/kasan/common.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 8fd04415d8f4..a880e5a547ed 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -318,6 +318,11 @@ bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
return __kasan_slab_free(cache, object, ip, true);
}

+static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
+{

+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

+}
+

static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

size_t size, gfp_t flags, bool keep_tag)
{
@@ -345,7 +350,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

KASAN_KMALLOC_REDZONE);

if (cache->flags & SLAB_KASAN)

- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
+ set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);
}
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

There's a config option CONFIG_KASAN_STACK that has to be enabled for
KASAN to use stack instrumentation and perform validity checks for
stack variables.

There's no need to unpoison stack when CONFIG_KASAN_STACK is not enabled.
Only call kasan_unpoison_task_stack[_below]() when CONFIG_KASAN_STACK is
enabled.

Note, that CONFIG_KASAN_STACK is an option that is currently always
defined when CONFIG_KASAN is enabled, and therefore has to be tested
with #if instead of #ifdef.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/If8a891e9fe01ea543e00b576852685afec0887e3
---

arch/arm64/kernel/sleep.S | 2 +-
arch/x86/kernel/acpi/wakeup_64.S | 2 +-

include/linux/kasan.h | 10 ++++++----
mm/kasan/common.c | 2 ++
4 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index ba40d57757d6..bdadfa56b40e 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -133,7 +133,7 @@ SYM_FUNC_START(_cpu_resume)
*/
bl cpu_do_resume

-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
mov x0, sp
bl kasan_unpoison_task_stack_below
#endif
diff --git a/arch/x86/kernel/acpi/wakeup_64.S b/arch/x86/kernel/acpi/wakeup_64.S
index c8daa92f38dc..5d3a0b8fd379 100644
--- a/arch/x86/kernel/acpi/wakeup_64.S
+++ b/arch/x86/kernel/acpi/wakeup_64.S
@@ -112,7 +112,7 @@ SYM_FUNC_START(do_suspend_lowlevel)
movq pt_regs_r14(%rax), %r14
movq pt_regs_r15(%rax), %r15

-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
/*
* The suspend path may have poisoned some areas deeper in the stack,
* which we now need to unpoison.
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 42a556c5d67c..2c37a39b76ed 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -74,8 +74,6 @@ static inline void kasan_disable_current(void) {}

void kasan_unpoison_memory(const void *address, size_t size);

-void kasan_unpoison_task_stack(struct task_struct *task);
-
void kasan_alloc_pages(struct page *page, unsigned int order);
void kasan_free_pages(struct page *page, unsigned int order);

@@ -120,8 +118,6 @@ void kasan_restore_multi_shot(bool enabled);

static inline void kasan_unpoison_memory(const void *address, size_t size) {}

-static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

@@ -173,6 +169,12 @@ static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

#endif /* CONFIG_KASAN */

+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
+void kasan_unpoison_task_stack(struct task_struct *task);
+#else
+static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
+#endif
+
#ifdef CONFIG_KASAN_GENERIC

void kasan_cache_shrink(struct kmem_cache *cache);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a880e5a547ed..a3e67d49b893 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -58,6 +58,7 @@ void kasan_disable_current(void)
}
#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

+#if CONFIG_KASAN_STACK
static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
{
void *base = task_stack_page(task);
@@ -84,6 +85,7 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)

kasan_unpoison_memory(base, watermark - base);
}
+#endif /* CONFIG_KASAN_STACK */

void kasan_alloc_pages(struct page *page, unsigned int order)
{
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Even though hardware tag-based mode currently doesn't support checking
vmalloc allocations, it doesn't use shadow memory and works with
VMAP_STACK as is. Change VMAP_STACK definition accordingly.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/I3552cbc12321dec82cd7372676e9372a2eb452ac
---
arch/Kconfig | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig
index 56b6ccc0e32d..7e7d14fae568 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -914,16 +914,16 @@ config VMAP_STACK
default y
bool "Use a virtually-mapped stack"
depends on HAVE_ARCH_VMAP_STACK
- depends on !KASAN || KASAN_VMALLOC
+ depends on !KASAN || KASAN_HW_TAGS || KASAN_VMALLOC
help
Enable this if you want the use virtually-mapped kernel stacks
with guard pages. This causes kernel stack overflows to be
caught immediately rather than causing difficult-to-diagnose
corruption.

- To use this with KASAN, the architecture must support backing
- virtual mappings with real shadow memory, and KASAN_VMALLOC must
- be enabled.
+ To use this with software KASAN modes, the architecture must support
+ backing virtual mappings with real shadow memory, and KASAN_VMALLOC
+ must be enabled.

config ARCH_OPTIONAL_KERNEL_RWX
def_bool n
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

There's no need for __kasan_unpoison_stack() helper, as it's only
currently used in a single place. Removing it also removes unneeded
arithmetic.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ie5ba549d445292fe629b4a96735e4034957bcc50
---
mm/kasan/common.c | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a3e67d49b893..9008fc6b0810 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -59,18 +59,12 @@ void kasan_disable_current(void)

#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

#if CONFIG_KASAN_STACK
-static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
-{
- void *base = task_stack_page(task);
- size_t size = sp - base;
-
- kasan_unpoison_memory(base, size);
-}
-
/* Unpoison the entire stack for a task. */
void kasan_unpoison_task_stack(struct task_struct *task)
{
- __kasan_unpoison_stack(task, task_stack_page(task) + THREAD_SIZE);
+ void *base = task_stack_page(task);
+
+ kasan_unpoison_memory(base, THREAD_SIZE);
}

/* Unpoison the stack for the current task beyond a watermark sp value. */
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Using kasan_reset_tag() currently results in a function call. As it's
called quite often from the allocator code, this leads to a noticeable
slowdown. Move it to include/linux/kasan.h and turn it into a static
inline function. Also remove the now unneeded reset_tag() internal KASAN
macro and use kasan_reset_tag() instead.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/I4d2061acfe91d480a75df00b07c22d8494ef14b5
---
include/linux/kasan.h | 5 ++++-
mm/kasan/common.c | 6 +++---
mm/kasan/hw_tags.c | 9 ++-------
mm/kasan/kasan.h | 4 ----
mm/kasan/report.c | 4 ++--
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 4 ++--
mm/kasan/shadow.c | 4 ++--
mm/kasan/sw_tags.c | 9 ++-------
9 files changed, 18 insertions(+), 29 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 2c37a39b76ed..0211a4ec5d87 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -191,7 +191,10 @@ static inline void kasan_record_aux_stack(void *ptr) {}

#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)

-void *kasan_reset_tag(const void *addr);
+static inline void *kasan_reset_tag(const void *addr)
+{
+ return (void *)arch_kasan_reset_tag(addr);
+}

bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 9008fc6b0810..a266b90636a1 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -174,14 +174,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
- return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
- return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

void kasan_poison_slab(struct page *page)
@@ -278,7 +278,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

tag = get_tag(object);
tagged_object = object;
- object = reset_tag(object);
+ object = kasan_reset_tag(object);

if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) !=
object)) {
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index d858aeb7387f..fe8e6c8e6319 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -26,20 +26,15 @@ void kasan_init_hw_tags(void)
pr_info("KernelAddressSanitizer initialized\n");
}

-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-

void kasan_poison_memory(const void *address, size_t size, u8 value)

{
- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), value);

}

void kasan_unpoison_memory(const void *address, size_t size)

{
- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 5513b4685007..e9c7d061fbe5 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -246,15 +246,11 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)
return addr;
}
#endif
-#ifndef arch_kasan_reset_tag
-#define arch_kasan_reset_tag(addr) ((void *)(addr))
-#endif
#ifndef arch_kasan_get_tag
#define arch_kasan_get_tag(addr) 0
#endif

#define set_tag(addr, tag) ((void *)arch_kasan_set_tag((addr), (tag)))
-#define reset_tag(addr) ((void *)arch_kasan_reset_tag(addr))
#define get_tag(addr) arch_kasan_get_tag(addr)

#ifdef CONFIG_KASAN_HW_TAGS
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 0cac53a57c14..25ca66c99e48 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -328,7 +328,7 @@ void kasan_report_invalid_free(void *object, unsigned long ip)
unsigned long flags;
u8 tag = get_tag(object);

- object = reset_tag(object);
+ object = kasan_reset_tag(object);

#if IS_ENABLED(CONFIG_KUNIT)
if (current->kunit_test)
@@ -361,7 +361,7 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write,
disable_trace_on_warning();

tagged_addr = (void *)addr;
- untagged_addr = reset_tag(tagged_addr);
+ untagged_addr = kasan_reset_tag(tagged_addr);

info.access_addr = tagged_addr;
if (addr_has_metadata(untagged_addr))
diff --git a/mm/kasan/report_hw_tags.c b/mm/kasan/report_hw_tags.c
index da543eb832cd..57114f0e14d1 100644
--- a/mm/kasan/report_hw_tags.c
+++ b/mm/kasan/report_hw_tags.c
@@ -22,7 +22,7 @@ const char *get_bug_type(struct kasan_access_info *info)

void *find_first_bad_addr(void *addr, size_t size)
{
- return reset_tag(addr);
+ return kasan_reset_tag(addr);
}

void metadata_fetch_row(char *buffer, void *row)
diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index 317100fd95b9..7604b46239d4 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -41,7 +41,7 @@ const char *get_bug_type(struct kasan_access_info *info)
int i;

tag = get_tag(info->access_addr);
- addr = reset_tag(info->access_addr);
+ addr = kasan_reset_tag(info->access_addr);
page = kasan_addr_to_page(addr);

if (page && PageSlab(page)) {
cache = page->slab_cache;

@@ -72,7 +72,7 @@ const char *get_bug_type(struct kasan_access_info *info)
void *find_first_bad_addr(void *addr, size_t size)
{
u8 tag = get_tag(addr);
- void *p = reset_tag(addr);
+ void *p = kasan_reset_tag(addr);
void *end = p + size;

while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p))
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 616ac64c4a21..8e4fa9157a0b 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -81,7 +81,7 @@ void kasan_poison_memory(const void *address, size_t size, u8 value)
* some of the callers (e.g. kasan_poison_object_data) pass tagged
* addresses to this function.
*/
- address = reset_tag(address);
+ address = kasan_reset_tag(address);

shadow_start = kasan_mem_to_shadow(address);
shadow_end = kasan_mem_to_shadow(address + size);
@@ -98,7 +98,7 @@ void kasan_unpoison_memory(const void *address, size_t size)
* some of the callers (e.g. kasan_unpoison_object_data) pass tagged
* addresses to this function.
*/
- address = reset_tag(address);
+ address = kasan_reset_tag(address);

kasan_poison_memory(address, size, tag);

diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index 3bffb489b144..d1af6f6c6d12 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -67,11 +67,6 @@ u8 random_tag(void)
return (u8)(state % (KASAN_TAG_MAX + 1));
}

-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip)
{
@@ -107,7 +102,7 @@ bool check_memory_region(unsigned long addr, size_t size, bool write,
if (tag == KASAN_TAG_KERNEL)
return true;

- untagged_addr = reset_tag((const void *)addr);
+ untagged_addr = kasan_reset_tag((const void *)addr);
if (unlikely(untagged_addr <
kasan_shadow_to_mem((void *)KASAN_SHADOW_START))) {
return !kasan_report(addr, size, write, ret_ip);
@@ -126,7 +121,7 @@ bool check_memory_region(unsigned long addr, size_t size, bool write,
bool check_invalid_free(void *addr)
{
u8 tag = get_tag(addr);
- u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(reset_tag(addr)));
+ u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(kasan_reset_tag(addr)));

return (shadow_byte == KASAN_TAG_INVALID) ||
(tag != KASAN_TAG_KERNEL && tag != shadow_byte);
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Using random_tag() currently results in a function call. Move its
definition to mm/kasan/kasan.h and turn it into a static inline function
for hardware tag-based mode to avoid uneeded function calls.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/Iac5b2faf9a912900e16cca6834d621f5d4abf427
---
mm/kasan/hw_tags.c | 5 -----
mm/kasan/kasan.h | 34 +++++++++++++++++-----------------
2 files changed, 17 insertions(+), 22 deletions(-)

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index fe8e6c8e6319..d5824530fd15 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -38,11 +38,6 @@ void kasan_unpoison_memory(const void *address, size_t size)
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

-u8 random_tag(void)
-{
- return hw_get_random_tag();
-}
-
bool check_invalid_free(void *addr)
{
u8 ptr_tag = get_tag(addr);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index e9c7d061fbe5..d7a03eab5814 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -188,6 +188,12 @@ static inline bool addr_has_metadata(const void *addr)

#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+void print_tags(u8 addr_tag, const void *addr);
+#else
+static inline void print_tags(u8 addr_tag, const void *addr) { }
+#endif
+
bool check_invalid_free(void *addr);

void *find_first_bad_addr(void *addr, size_t size);
@@ -223,23 +229,6 @@ static inline void quarantine_reduce(void) { }

static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif

-#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
-
-void print_tags(u8 addr_tag, const void *addr);
-
-u8 random_tag(void);
-
-#else
-
-static inline void print_tags(u8 addr_tag, const void *addr) { }
-
-static inline u8 random_tag(void)
-{
- return 0;
-}
-
-#endif
-
#ifndef arch_kasan_set_tag

static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)

{
@@ -275,6 +264,17 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)

#endif /* CONFIG_KASAN_HW_TAGS */

+#ifdef CONFIG_KASAN_SW_TAGS
+u8 random_tag(void);
+#elif defined(CONFIG_KASAN_HW_TAGS)
+#define random_tag() hw_get_random_tag()
+#else
+static inline u8 random_tag(void)
+{
+ return 0;
+}
+#endif
+
/*
* Exported functions for interfaces called from assembly or from generated
* code. Declarations here to avoid warning about missing declarations.
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Using kasan_poison_memory() or check_invalid_free() currently results in
function calls. Move their definitions to mm/kasan/kasan.h and turn them
into static inline functions for hardware tag-based mode to avoid
unneeded function calls.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>
Link: https://linux-review.googlesource.com/id/Ia9d8191024a12d1374675b3d27197f10193f50bb
---
mm/kasan/hw_tags.c | 15 ---------------
mm/kasan/kasan.h | 28 ++++++++++++++++++++++++----
2 files changed, 24 insertions(+), 19 deletions(-)

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index d5824530fd15..9d7b1f1a2553 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -26,27 +26,12 @@ void kasan_init_hw_tags(void)

pr_info("KernelAddressSanitizer initialized\n");
}

-void kasan_poison_memory(const void *address, size_t size, u8 value)
-{
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), value);
-}
-

void kasan_unpoison_memory(const void *address, size_t size)

{
hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

-bool check_invalid_free(void *addr)
-{
- u8 ptr_tag = get_tag(addr);
- u8 mem_tag = hw_get_mem_tag(addr);
-
- return (mem_tag == KASAN_TAG_INVALID) ||
- (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
-}
-
void kasan_set_free_info(struct kmem_cache *cache,
void *object, u8 tag)
{
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index d7a03eab5814..73364acf6ec8 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -153,8 +153,6 @@ struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object);

-void kasan_poison_memory(const void *address, size_t size, u8 value);
-
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

static inline const void *kasan_shadow_to_mem(const void *shadow_addr)
@@ -194,8 +192,6 @@ void print_tags(u8 addr_tag, const void *addr);

static inline void print_tags(u8 addr_tag, const void *addr) { }

#endif

-bool check_invalid_free(void *addr);
-

void *find_first_bad_addr(void *addr, size_t size);

const char *get_bug_type(struct kasan_access_info *info);

void metadata_fetch_row(char *buffer, void *row);
@@ -275,6 +271,30 @@ static inline u8 random_tag(void)
}
#endif

+#ifdef CONFIG_KASAN_HW_TAGS
+
+static inline void kasan_poison_memory(const void *address, size_t size, u8 value)
+{
+ hw_set_mem_tag_range(kasan_reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), value);
+}
+
+static inline bool check_invalid_free(void *addr)
+{
+ u8 ptr_tag = get_tag(addr);
+ u8 mem_tag = hw_get_mem_tag(addr);
+
+ return (mem_tag == KASAN_TAG_INVALID) ||
+ (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
+}
+
+#else /* CONFIG_KASAN_HW_TAGS */
+
+void kasan_poison_memory(const void *address, size_t size, u8 value);
+bool check_invalid_free(void *addr);
+
+#endif /* CONFIG_KASAN_HW_TAGS */

[Andrey Konovalov]

Currently kasan_unpoison_memory() is used as both an external annotation
and as an internal memory poisoning helper. Rename external annotation to
kasan_unpoison_data() and inline the internal helper for hardware
tag-based mode to avoid undeeded function calls.

There's the external annotation kasan_unpoison_slab() that is currently
defined as static inline and uses kasan_unpoison_memory(). With this
change it's turned into a function call. Overall, this results in the
same number of calls for hardware tag-based mode as
kasan_unpoison_memory() is now inlined.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/Ia7c8b659f79209935cbaab3913bf7f082cc43a0e
---
include/linux/kasan.h | 16 ++++++----------
kernel/fork.c | 2 +-
mm/kasan/common.c | 10 ++++++++++
mm/kasan/hw_tags.c | 6 ------
mm/kasan/kasan.h | 7 +++++++
mm/slab_common.c | 2 +-
6 files changed, 25 insertions(+), 18 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 0211a4ec5d87..34236f134472 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -72,14 +72,15 @@ static inline void kasan_disable_current(void) {}

#ifdef CONFIG_KASAN

-void kasan_unpoison_memory(const void *address, size_t size);

-
void kasan_alloc_pages(struct page *page, unsigned int order);
void kasan_free_pages(struct page *page, unsigned int order);

void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
slab_flags_t *flags);

+void kasan_unpoison_data(const void *address, size_t size);
+void kasan_unpoison_slab(const void *ptr);
+
void kasan_poison_slab(struct page *page);
void kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
void kasan_poison_object_data(struct kmem_cache *cache, void *object);
@@ -104,11 +105,6 @@ struct kasan_cache {
int free_meta_offset;
};

-size_t __ksize(const void *);
-static inline void kasan_unpoison_slab(const void *ptr)
-{
- kasan_unpoison_memory(ptr, __ksize(ptr));
-}
size_t kasan_metadata_size(struct kmem_cache *cache);

bool kasan_save_enable_multi_shot(void);
@@ -116,8 +112,6 @@ void kasan_restore_multi_shot(bool enabled);

#else /* CONFIG_KASAN */

-static inline void kasan_unpoison_memory(const void *address, size_t size) {}

-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

@@ -125,6 +119,9 @@ static inline void kasan_cache_create(struct kmem_cache *cache,
unsigned int *size,
slab_flags_t *flags) {}

+static inline void kasan_unpoison_data(const void *address, size_t size) { }
+static inline void kasan_unpoison_slab(const void *ptr) { }
+
static inline void kasan_poison_slab(struct page *page) {}
static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
void *object) {}
@@ -164,7 +161,6 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
return false;
}

-static inline void kasan_unpoison_slab(const void *ptr) { }

static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

#endif /* CONFIG_KASAN */

diff --git a/kernel/fork.c b/kernel/fork.c
index 463ef51f2b05..d6ff6b5650aa 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -226,7 +226,7 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
continue;

/* Mark stack accessible for KASAN. */
- kasan_unpoison_memory(s->addr, THREAD_SIZE);
+ kasan_unpoison_data(s->addr, THREAD_SIZE);

/* Clear stale pointers from reused stack. */
memset(s->addr, 0, THREAD_SIZE);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a266b90636a1..4598c1364f19 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -184,6 +184,16 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

+void kasan_unpoison_data(const void *address, size_t size)
+{
+ kasan_unpoison_memory(address, size);
+}
+
+void kasan_unpoison_slab(const void *ptr)
+{
+ kasan_unpoison_memory(ptr, __ksize(ptr));
+}
+
void kasan_poison_slab(struct page *page)
{
unsigned long i;
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 9d7b1f1a2553..bd8bf05c8034 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -26,12 +26,6 @@ void kasan_init_hw_tags(void)

pr_info("KernelAddressSanitizer initialized\n");
}

-void kasan_unpoison_memory(const void *address, size_t size)
-{
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), get_tag(address));

-}
-
void kasan_set_free_info(struct kmem_cache *cache,
void *object, u8 tag)
{
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h

index 73364acf6ec8..ba850285a360 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -279,6 +279,12 @@ static inline void kasan_poison_memory(const void *address, size_t size, u8 valu
round_up(size, KASAN_GRANULE_SIZE), value);
}

+static inline void kasan_unpoison_memory(const void *address, size_t size)
+{
+ hw_set_mem_tag_range(kasan_reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
+}
+
static inline bool check_invalid_free(void *addr)
{
u8 ptr_tag = get_tag(addr);
@@ -291,6 +297,7 @@ static inline bool check_invalid_free(void *addr)
#else /* CONFIG_KASAN_HW_TAGS */

void kasan_poison_memory(const void *address, size_t size, u8 value);
+void kasan_unpoison_memory(const void *address, size_t size);
bool check_invalid_free(void *addr);

#endif /* CONFIG_KASAN_HW_TAGS */
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 53d0f8bb57ea..f1b0c4a22f08 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1176,7 +1176,7 @@ size_t ksize(const void *objp)
* We assume that ksize callers could use whole allocated area,
* so we need to unpoison this area.
*/
- kasan_unpoison_memory(objp, size);
+ kasan_unpoison_data(objp, size);
return size;
}
EXPORT_SYMBOL(ksize);
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Hardware tag-based KASAN mode is intended to eventually be used in

production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.

This change adds a few boot parameters for hardware tag-based KASAN that

allow to disable or otherwise control particular KASAN features.

The features that can be controlled are:

1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.

With this change a new boot parameter kasan.mode allows to choose one of

three main modes:

- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled

The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:

- kasan.stack=off/on - enable stacks collection
(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)

If kasan.mode parameter is not provided, it defaults to full when
CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.

It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by

the Android GKI (Generic Kernel Image) initiative [1].

[1] https://source.android.com/devices/architecture/kernel/generic-kernel-image

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/If7d37003875b2ed3e0935702c8015c223d6416a4
---
mm/kasan/common.c | 22 +++++--
mm/kasan/hw_tags.c | 144 +++++++++++++++++++++++++++++++++++++++++++++
mm/kasan/kasan.h | 16 +++++
mm/kasan/report.c | 14 ++++-
4 files changed, 189 insertions(+), 7 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 4598c1364f19..efad5ed6a3bd 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -129,6 +129,11 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
unsigned int redzone_size;
int redzone_adjust;

+ if (!kasan_stack_collection_enabled()) {
+ *flags |= SLAB_KASAN;
+ return;
+ }
+
/* Add alloc meta. */
cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);
@@ -165,6 +170,8 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,

size_t kasan_metadata_size(struct kmem_cache *cache)
{
+ if (!kasan_stack_collection_enabled())
+ return 0;
return (cache->kasan_info.alloc_meta_offset ?
sizeof(struct kasan_alloc_meta) : 0) +
(cache->kasan_info.free_meta_offset ?
@@ -267,11 +274,13 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;
+ if (kasan_stack_collection_enabled()) {
+ if (!(cache->flags & SLAB_KASAN))
+ return (void *)object;

- alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ }

if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));

@@ -308,6 +317,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
kasan_poison_memory(object, rounded_up_size, KASAN_KMALLOC_FREE);

+ if (!kasan_stack_collection_enabled())
+ return false;
+
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
unlikely(!(cache->flags & SLAB_KASAN)))
return false;
@@ -355,7 +367,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);

- if (cache->flags & SLAB_KASAN)
+ if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))

set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index bd8bf05c8034..52984825c75f 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -8,6 +8,8 @@

#define pr_fmt(fmt) "kasan: " fmt

+#include <linux/init.h>
+#include <linux/jump_label.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
#include <linux/memory.h>
@@ -17,11 +19,153 @@

#include "kasan.h"

+enum kasan_arg_mode {
+ KASAN_ARG_MODE_DEFAULT,
+ KASAN_ARG_MODE_OFF,
+ KASAN_ARG_MODE_PROD,
+ KASAN_ARG_MODE_FULL,
+};
+
+enum kasan_arg_stacks {
+ KASAN_ARG_STACKS_DEFAULT,
+ KASAN_ARG_STACKS_OFF,
+ KASAN_ARG_STACKS_ON,
+};
+
+enum kasan_arg_fault {
+ KASAN_ARG_FAULT_DEFAULT,
+ KASAN_ARG_FAULT_REPORT,
+ KASAN_ARG_FAULT_PANIC,
+};
+
+static enum kasan_arg_mode kasan_arg_mode __ro_after_init;
+static enum kasan_arg_stacks kasan_arg_stacks __ro_after_init;
+static enum kasan_arg_fault kasan_arg_fault __ro_after_init;
+
+/* Whether KASAN is enabled at all. */
+DEFINE_STATIC_KEY_FALSE_RO(kasan_flag_enabled);
+EXPORT_SYMBOL(kasan_flag_enabled);
+
+/* Whether to collect alloc/free stack traces. */
+DEFINE_STATIC_KEY_FALSE_RO(kasan_flag_stacks);
+
+/* Whether panic or disable tag checking on fault. */
+bool kasan_flag_panic __ro_after_init;
+
+/* kasan.mode=off/prod/full */
+static int __init early_kasan_mode(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))
+ kasan_arg_mode = KASAN_ARG_MODE_OFF;
+ else if (!strcmp(arg, "prod"))
+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ else if (!strcmp(arg, "full"))
+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else
+ return -EINVAL;

+
+ return 0;
+}

+early_param("kasan.mode", early_kasan_mode);
+
+/* kasan.stack=off/on */
+static int __init early_kasan_flag_stacks(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))
+ kasan_arg_stacks = KASAN_ARG_STACKS_OFF;
+ else if (!strcmp(arg, "on"))
+ kasan_arg_stacks = KASAN_ARG_STACKS_ON;
+ else
+ return -EINVAL;

+
+ return 0;
+}

+early_param("kasan.stacks", early_kasan_flag_stacks);
+
+/* kasan.fault=report/panic */
+static int __init early_kasan_fault(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "report"))
+ kasan_arg_fault = KASAN_ARG_FAULT_REPORT;
+ else if (!strcmp(arg, "panic"))
+ kasan_arg_fault = KASAN_ARG_FAULT_PANIC;
+ else
+ return -EINVAL;

+
+ return 0;
+}

+early_param("kasan.fault", early_kasan_fault);
+
/* kasan_init_hw_tags() is called for each CPU. */
void kasan_init_hw_tags(void)
{
+ /* Choose KASAN mode if kasan boot parameter is not provided. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_DEFAULT) {
+ if (IS_ENABLED(CONFIG_DEBUG_KERNEL))
+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else
+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ }
+
+ /* If KASAN isn't enabled, do nothing. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_OFF)
+ return;
+
+ /* Only process the boot parameters on boot CPU. */
+ if (smp_processor_id() == 0) {
+ /* Preset parameter values based on the mode. */
+ switch (kasan_arg_mode) {
+ case KASAN_ARG_MODE_OFF:
+ return;
+ case KASAN_ARG_MODE_PROD:
+ static_branch_enable(&kasan_flag_enabled);
+ break;
+ case KASAN_ARG_MODE_FULL:
+ static_branch_enable(&kasan_flag_enabled);
+ static_branch_enable(&kasan_flag_stacks);
+ break;
+ default:
+ break;
+ }
+
+ /* Now, optionally override the presets. */
+
+ switch (kasan_arg_stacks) {
+ case KASAN_ARG_STACKS_OFF:
+ static_branch_disable(&kasan_flag_stacks);
+ break;
+ case KASAN_ARG_STACKS_ON:
+ static_branch_enable(&kasan_flag_stacks);
+ break;
+ default:
+ break;
+ }
+
+ switch (kasan_arg_fault) {
+ case KASAN_ARG_FAULT_REPORT:
+ kasan_flag_panic = false;
+ break;
+ case KASAN_ARG_FAULT_PANIC:
+ kasan_flag_panic = true;
+ break;
+ default:
+ break;
+ }
+ }
+
+ /* Init tags for each CPU. */
hw_init_tags(KASAN_TAG_MAX);

+ /* Only print the message on boot CPU. */
if (smp_processor_id() == 0)

pr_info("KernelAddressSanitizer initialized\n");
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index ba850285a360..8a4cd9618142 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -5,6 +5,22 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>

+#ifdef CONFIG_KASAN_HW_TAGS
+#include <linux/jump_label.h>
+DECLARE_STATIC_KEY_FALSE(kasan_flag_stacks);
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return static_branch_unlikely(&kasan_flag_stacks);
+}
+#else
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return true;
+}
+#endif
+
+extern bool kasan_flag_panic __ro_after_init;
+
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
#else
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 25ca66c99e48..7d86af340148 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -99,6 +99,10 @@ static void end_report(unsigned long *flags)
panic_on_warn = 0;
panic("panic_on_warn set ...\n");
}
+#ifdef CONFIG_KASAN_HW_TAGS
+ if (kasan_flag_panic)
+ panic("kasan.fault=panic set ...\n");
+#endif
kasan_enable_current();
}

@@ -161,8 +165,8 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,
(void *)(object_addr + cache->object_size));
}

-static void describe_object(struct kmem_cache *cache, void *object,
- const void *addr, u8 tag)
+static void describe_object_stacks(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
{

struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);

@@ -190,7 +194,13 @@ static void describe_object(struct kmem_cache *cache, void *object,
}
#endif
}
+}

+static void describe_object(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
+{
+ if (kasan_stack_collection_enabled())
+ describe_object_stacks(cache, object, addr, tag);
describe_object_addr(cache, object, addr);
}

--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Declare the kasan_enabled static key in include/linux/kasan.h and in
include/linux/mm.h and check it in all kasan annotations. This allows to
avoid any slowdown caused by function calls when kasan_enabled is
disabled.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/I2589451d3c96c97abbcbf714baabe6161c6f153e
---
include/linux/kasan.h | 220 ++++++++++++++++++++++++++++++++----------
include/linux/mm.h | 22 +++--
mm/kasan/common.c | 60 ++++++------
3 files changed, 216 insertions(+), 86 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 34236f134472..ae1046fc74e5 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -2,6 +2,7 @@
#ifndef _LINUX_KASAN_H
#define _LINUX_KASAN_H

+#include <linux/jump_label.h>
#include <linux/types.h>

struct kmem_cache;
@@ -72,56 +73,179 @@ static inline void kasan_disable_current(void) {}

#ifdef CONFIG_KASAN

-void kasan_alloc_pages(struct page *page, unsigned int order);
-void kasan_free_pages(struct page *page, unsigned int order);
+struct kasan_cache {
+ int alloc_meta_offset;
+ int free_meta_offset;
+};

-void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags);
+#ifdef CONFIG_KASAN_HW_TAGS
+DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
+static inline kasan_enabled(void)
+{
+ return static_branch_likely(&kasan_flag_enabled);
+}
+#else
+static inline kasan_enabled(void)

+{
+ return true;
+}
+#endif

-void kasan_unpoison_data(const void *address, size_t size);
-void kasan_unpoison_slab(const void *ptr);
+void __kasan_alloc_pages(struct page *page, unsigned int order);
+static inline void kasan_alloc_pages(struct page *page, unsigned int order)
+{
+ if (kasan_enabled())
+ __kasan_alloc_pages(page, order);
+}

-void kasan_poison_slab(struct page *page);
-void kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
-void kasan_poison_object_data(struct kmem_cache *cache, void *object);
-void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
- const void *object);
+void __kasan_free_pages(struct page *page, unsigned int order);
+static inline void kasan_free_pages(struct page *page, unsigned int order)
+{
+ if (kasan_enabled())
+ __kasan_free_pages(page, order);
+}

-void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
- gfp_t flags);
-void kasan_kfree_large(void *ptr, unsigned long ip);
-void kasan_poison_kfree(void *ptr, unsigned long ip);
-void * __must_check kasan_kmalloc(struct kmem_cache *s, const void *object,
- size_t size, gfp_t flags);
-void * __must_check kasan_krealloc(const void *object, size_t new_size,
- gfp_t flags);
+void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags);
+static inline void kasan_cache_create(struct kmem_cache *cache,
+ unsigned int *size, slab_flags_t *flags)
+{
+ if (kasan_enabled())
+ __kasan_cache_create(cache, size, flags);
+}

-void * __must_check kasan_slab_alloc(struct kmem_cache *s, void *object,
- gfp_t flags);
-bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip);
+size_t __kasan_metadata_size(struct kmem_cache *cache);
+static inline size_t kasan_metadata_size(struct kmem_cache *cache)
+{
+ if (kasan_enabled())
+ return __kasan_metadata_size(cache);
+ return 0;
+}

-struct kasan_cache {
- int alloc_meta_offset;
- int free_meta_offset;
-};
+void __kasan_unpoison_data(const void *addr, size_t size);
+static inline void kasan_unpoison_data(const void *addr, size_t size)
+{
+ if (kasan_enabled())
+ __kasan_unpoison_data(addr, size);
+}
+
+void __kasan_unpoison_slab(const void *ptr);

+static inline void kasan_unpoison_slab(const void *ptr)

+{
+ if (kasan_enabled())
+ __kasan_unpoison_slab(ptr);
+}
+
+void __kasan_poison_slab(struct page *page);
+static inline void kasan_poison_slab(struct page *page)
+{
+ if (kasan_enabled())
+ return __kasan_poison_slab(page);
+}
+
+void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
+static inline void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
+{
+ if (kasan_enabled())
+ return __kasan_unpoison_object_data(cache, object);
+}
+
+void __kasan_poison_object_data(struct kmem_cache *cache, void *object);
+static inline void kasan_poison_object_data(struct kmem_cache *cache, void *object)
+{
+ if (kasan_enabled())
+ __kasan_poison_object_data(cache, object);
+}
+
+void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
+ const void *object);
+static inline void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
+ const void *object)
+{
+ if (kasan_enabled())
+ return __kasan_init_slab_obj(cache, object);
+ return (void *)object;
+}
+
+bool __kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip);
+static inline bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip)
+{
+ if (kasan_enabled())
+ return __kasan_slab_free(s, object, ip);
+ return false;
+}
+
+void * __must_check __kasan_slab_alloc(struct kmem_cache *s,
+ void *object, gfp_t flags);
+static inline void * __must_check kasan_slab_alloc(struct kmem_cache *s,
+ void *object, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_slab_alloc(s, object, flags);
+ return object;
+}

-size_t kasan_metadata_size(struct kmem_cache *cache);
+void * __must_check __kasan_kmalloc(struct kmem_cache *s, const void *object,
+ size_t size, gfp_t flags);
+static inline void * __must_check kasan_kmalloc(struct kmem_cache *s, const void *object,
+ size_t size, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_kmalloc(s, object, size, flags);
+ return (void *)object;
+}
+
+void * __must_check __kasan_kmalloc_large(const void *ptr,
+ size_t size, gfp_t flags);
+static inline void * __must_check kasan_kmalloc_large(const void *ptr,
+ size_t size, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_kmalloc_large(ptr, size, flags);
+ return (void *)ptr;
+}
+
+void * __must_check __kasan_krealloc(const void *object,
+ size_t new_size, gfp_t flags);
+static inline void * __must_check kasan_krealloc(const void *object,
+ size_t new_size, gfp_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_krealloc(object, new_size, flags);
+ return (void *)object;
+}
+
+void __kasan_poison_kfree(void *ptr, unsigned long ip);
+static inline void kasan_poison_kfree(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())
+ __kasan_poison_kfree(ptr, ip);
+}
+
+void __kasan_kfree_large(void *ptr, unsigned long ip);
+static inline void kasan_kfree_large(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())
+ __kasan_kfree_large(ptr, ip);
+}

bool kasan_save_enable_multi_shot(void);

void kasan_restore_multi_shot(bool enabled);

#else /* CONFIG_KASAN */

+static inline kasan_enabled(void)
+{
+ return false;
+}

static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

-
static inline void kasan_cache_create(struct kmem_cache *cache,
unsigned int *size,
slab_flags_t *flags) {}
-
-static inline void kasan_unpoison_data(const void *address, size_t size) { }

-static inline void kasan_unpoison_slab(const void *ptr) { }
-

+static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

+static inline void kasan_unpoison_data(const void *address, size_t size) {}
+static inline void kasan_unpoison_slab(const void *ptr) {}

static inline void kasan_poison_slab(struct page *page) {}
static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
void *object) {}

@@ -132,36 +256,32 @@ static inline void *kasan_init_slab_obj(struct kmem_cache *cache,
{
return (void *)object;
}
-
-static inline void *kasan_kmalloc_large(void *ptr, size_t size, gfp_t flags)
+static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
+ unsigned long ip)
{
- return ptr;
+ return false;
+}
+static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
+ gfp_t flags)
+{
+ return object;
}
-static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}
-static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
static inline void *kasan_kmalloc(struct kmem_cache *s, const void *object,
size_t size, gfp_t flags)
{
return (void *)object;
}
+static inline void *kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags)
+{
+ return (void *)ptr;
+}
static inline void *kasan_krealloc(const void *object, size_t new_size,
gfp_t flags)
{
return (void *)object;
}
-
-static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
- gfp_t flags)
-{
- return object;
-}
-static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
- unsigned long ip)
-{
- return false;
-}
-
-static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
+static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
+static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}

#endif /* CONFIG_KASAN */

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0793d03a4183..8d84a6b2fa3c 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -31,6 +31,7 @@
#include <linux/sizes.h>
#include <linux/sched.h>
#include <linux/pgtable.h>
+#include <linux/kasan.h>

struct mempolicy;
struct anon_vma;
@@ -1414,22 +1415,30 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
#endif /* CONFIG_NUMA_BALANCING */

#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+
static inline u8 page_kasan_tag(const struct page *page)
{
- return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
+ if (kasan_enabled())
+ return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
+ return 0xff;
}

static inline void page_kasan_tag_set(struct page *page, u8 tag)
{
- page->flags &= ~(KASAN_TAG_MASK << KASAN_TAG_PGSHIFT);
- page->flags |= (tag & KASAN_TAG_MASK) << KASAN_TAG_PGSHIFT;
+ if (kasan_enabled()) {
+ page->flags &= ~(KASAN_TAG_MASK << KASAN_TAG_PGSHIFT);
+ page->flags |= (tag & KASAN_TAG_MASK) << KASAN_TAG_PGSHIFT;
+ }
}

static inline void page_kasan_tag_reset(struct page *page)
{
- page_kasan_tag_set(page, 0xff);
+ if (kasan_enabled())
+ page_kasan_tag_set(page, 0xff);
}
-#else
+
+#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+
static inline u8 page_kasan_tag(const struct page *page)
{
return 0xff;
@@ -1437,7 +1446,8 @@ static inline u8 page_kasan_tag(const struct page *page)

static inline void page_kasan_tag_set(struct page *page, u8 tag) { }
static inline void page_kasan_tag_reset(struct page *page) { }
-#endif
+
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */

static inline struct zone *page_zone(const struct page *page)
{
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index efad5ed6a3bd..385863eaec2c 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -81,7 +81,7 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)
}
#endif /* CONFIG_KASAN_STACK */

-void kasan_alloc_pages(struct page *page, unsigned int order)
+void __kasan_alloc_pages(struct page *page, unsigned int order)
{
u8 tag;
unsigned long i;
@@ -95,7 +95,7 @@ void kasan_alloc_pages(struct page *page, unsigned int order)
kasan_unpoison_memory(page_address(page), PAGE_SIZE << order);
}

-void kasan_free_pages(struct page *page, unsigned int order)
+void __kasan_free_pages(struct page *page, unsigned int order)
{
if (likely(!PageHighMem(page)))
kasan_poison_memory(page_address(page),
@@ -122,8 +122,8 @@ static inline unsigned int optimal_redzone(unsigned int object_size)
object_size <= (1 << 16) - 1024 ? 1024 : 2048;
}

-void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags)
+void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags)
{
unsigned int orig_size = *size;
unsigned int redzone_size;
@@ -168,7 +168,7 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
*flags |= SLAB_KASAN;
}

-size_t kasan_metadata_size(struct kmem_cache *cache)
+size_t __kasan_metadata_size(struct kmem_cache *cache)
{
if (!kasan_stack_collection_enabled())
return 0;
@@ -191,17 +191,17 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

-void kasan_unpoison_data(const void *address, size_t size)
+void __kasan_unpoison_data(const void *addr, size_t size)
{
- kasan_unpoison_memory(address, size);
+ kasan_unpoison_memory(addr, size);
}

-void kasan_unpoison_slab(const void *ptr)
+void __kasan_unpoison_slab(const void *ptr)
{
kasan_unpoison_memory(ptr, __ksize(ptr));
}

-void kasan_poison_slab(struct page *page)
+void __kasan_poison_slab(struct page *page)
{
unsigned long i;

@@ -211,12 +211,12 @@ void kasan_poison_slab(struct page *page)
KASAN_KMALLOC_REDZONE);
}

-void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
+void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
{
kasan_unpoison_memory(object, cache->object_size);
}

-void kasan_poison_object_data(struct kmem_cache *cache, void *object)
+void __kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
kasan_poison_memory(object,
round_up(cache->object_size, KASAN_GRANULE_SIZE),
@@ -269,7 +269,7 @@ static u8 assign_tag(struct kmem_cache *cache, const void *object,
#endif
}

-void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
+void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
const void *object)
{
struct kasan_alloc_meta *alloc_meta;
@@ -288,7 +288,7 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
return (void *)object;
}

-static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
+static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
unsigned long ip, bool quarantine)
{
u8 tag;
@@ -331,9 +331,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
return IS_ENABLED(CONFIG_KASAN_GENERIC);
}

-bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
+bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
{
- return __kasan_slab_free(cache, object, ip, true);
+ return ____kasan_slab_free(cache, object, ip, true);

}

static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)

@@ -341,7 +341,7 @@ static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
}

-static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
+static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,

size_t size, gfp_t flags, bool keep_tag)
{

unsigned long redzone_start;
@@ -373,20 +373,20 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
return set_tag(object, tag);
}

-void * __must_check kasan_slab_alloc(struct kmem_cache *cache, void *object,
- gfp_t flags)
+void * __must_check __kasan_slab_alloc(struct kmem_cache *cache,
+ void *object, gfp_t flags)
{
- return __kasan_kmalloc(cache, object, cache->object_size, flags, false);
+ return ____kasan_kmalloc(cache, object, cache->object_size, flags, false);
}

-void * __must_check kasan_kmalloc(struct kmem_cache *cache, const void *object,
- size_t size, gfp_t flags)
+void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *object,
+ size_t size, gfp_t flags)
{
- return __kasan_kmalloc(cache, object, size, flags, true);
+ return ____kasan_kmalloc(cache, object, size, flags, true);
}
-EXPORT_SYMBOL(kasan_kmalloc);
+EXPORT_SYMBOL(__kasan_kmalloc);

-void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
+void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size,
gfp_t flags)
{
struct page *page;
@@ -411,7 +411,7 @@ void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
return (void *)ptr;
}

-void * __must_check kasan_krealloc(const void *object, size_t size, gfp_t flags)
+void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flags)
{
struct page *page;

@@ -421,13 +421,13 @@ void * __must_check kasan_krealloc(const void *object, size_t size, gfp_t flags)
page = virt_to_head_page(object);

if (unlikely(!PageSlab(page)))
- return kasan_kmalloc_large(object, size, flags);
+ return __kasan_kmalloc_large(object, size, flags);
else
- return __kasan_kmalloc(page->slab_cache, object, size,
+ return ____kasan_kmalloc(page->slab_cache, object, size,
flags, true);
}

-void kasan_poison_kfree(void *ptr, unsigned long ip)
+void __kasan_poison_kfree(void *ptr, unsigned long ip)
{
struct page *page;

@@ -440,11 +440,11 @@ void kasan_poison_kfree(void *ptr, unsigned long ip)
}
kasan_poison_memory(ptr, page_size(page), KASAN_FREE_PAGE);
} else {
- __kasan_slab_free(page->slab_cache, ptr, ip, false);
+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);
}
}

-void kasan_kfree_large(void *ptr, unsigned long ip)
+void __kasan_kfree_large(void *ptr, unsigned long ip)
{
if (ptr != page_address(virt_to_head_page(ptr)))
kasan_report_invalid_free(ptr, ip);
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

kasan_poison_kfree() is currently only called for mempool allocations
that are backed by either kmem_cache_alloc() or kmalloc(). Therefore, the
page passed to kasan_poison_kfree() is always PageSlab() and there's no
need to do the check. Remove it.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/If31f88726745da8744c6bea96fb32584e6c2778c
---
mm/kasan/common.c | 11 +----------
1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 385863eaec2c..819403548f2e 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -432,16 +432,7 @@ void __kasan_poison_kfree(void *ptr, unsigned long ip)
struct page *page;

page = virt_to_head_page(ptr);
-
- if (unlikely(!PageSlab(page))) {
- if (ptr != page_address(page)) {
- kasan_report_invalid_free(ptr, ip);
- return;
- }
- kasan_poison_memory(ptr, page_size(page), KASAN_FREE_PAGE);
- } else {
- ____kasan_slab_free(page->slab_cache, ptr, ip, false);

- }
+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);
}

void __kasan_kfree_large(void *ptr, unsigned long ip)

--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Rename kasan_poison_kfree() to kasan_slab_free_mempool() as it better
reflects what this annotation does.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I5026f87364e556b506ef1baee725144bb04b8810
---
include/linux/kasan.h | 16 ++++++++--------
mm/kasan/common.c | 16 ++++++++--------
mm/mempool.c | 2 +-
3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index ae1046fc74e5..d47601517dad 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -175,6 +175,13 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned
return false;
}

+void __kasan_slab_free_mempool(void *ptr, unsigned long ip);
+static inline void kasan_slab_free_mempool(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())
+ __kasan_slab_free_mempool(ptr, ip);
+}
+

void * __must_check __kasan_slab_alloc(struct kmem_cache *s,

void *object, gfp_t flags);

static inline void * __must_check kasan_slab_alloc(struct kmem_cache *s,

@@ -215,13 +222,6 @@ static inline void * __must_check kasan_krealloc(const void *object,
return (void *)object;
}

-void __kasan_poison_kfree(void *ptr, unsigned long ip);

-static inline void kasan_poison_kfree(void *ptr, unsigned long ip)

-{
- if (kasan_enabled())
- __kasan_poison_kfree(ptr, ip);
-}
-
void __kasan_kfree_large(void *ptr, unsigned long ip);

static inline void kasan_kfree_large(void *ptr, unsigned long ip)

{
@@ -261,6 +261,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
{
return false;
}
+static inline void kasan_slab_free_mempool(void *ptr, unsigned long ip) {}

static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,

gfp_t flags)
{
@@ -280,7 +281,6 @@ static inline void *kasan_krealloc(const void *object, size_t new_size,
{
return (void *)object;

}
-static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}

static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}

#endif /* CONFIG_KASAN */

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 819403548f2e..60793f8695a8 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -336,6 +336,14 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)

return ____kasan_slab_free(cache, object, ip, true);
}

+void __kasan_slab_free_mempool(void *ptr, unsigned long ip)
+{
+ struct page *page;
+
+ page = virt_to_head_page(ptr);

+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);

+}
+

static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{
kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

@@ -427,14 +435,6 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag
flags, true);
}

-void __kasan_poison_kfree(void *ptr, unsigned long ip)
-{
- struct page *page;
-
- page = virt_to_head_page(ptr);

- ____kasan_slab_free(page->slab_cache, ptr, ip, false);
-}

-

void __kasan_kfree_large(void *ptr, unsigned long ip)

{
if (ptr != page_address(virt_to_head_page(ptr)))
diff --git a/mm/mempool.c b/mm/mempool.c
index f473cdddaff0..b1f39fa75ade 100644
--- a/mm/mempool.c
+++ b/mm/mempool.c
@@ -104,7 +104,7 @@ static inline void poison_element(mempool_t *pool, void *element)
static __always_inline void kasan_poison_element(mempool_t *pool, void *element)
{
if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
- kasan_poison_kfree(element, _RET_IP_);
+ kasan_slab_free_mempool(element, _RET_IP_);
else if (pool->alloc == mempool_alloc_pages)
kasan_free_pages(element, (unsigned long)pool->pool_data);
}
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

For hardware tag-based mode kasan_poison_memory() already rounds up the
size. Do the same for software modes and remove round_up() from the common
code.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ib397128fac6eba874008662b4964d65352db4aa4
---
mm/kasan/common.c | 8 ++------
mm/kasan/shadow.c | 1 +
2 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 60793f8695a8..69ab880abacc 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -218,9 +218,7 @@ void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object)

void __kasan_poison_object_data(struct kmem_cache *cache, void *object)

{
- kasan_poison_memory(object,
- round_up(cache->object_size, KASAN_GRANULE_SIZE),
- KASAN_KMALLOC_REDZONE);
+ kasan_poison_memory(object, cache->object_size, KASAN_KMALLOC_REDZONE);
}

/*
@@ -293,7 +291,6 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
{
u8 tag;
void *tagged_object;
- unsigned long rounded_up_size;

tag = get_tag(object);
tagged_object = object;

@@ -314,8 +311,7 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
return true;
}

- rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
- kasan_poison_memory(object, rounded_up_size, KASAN_KMALLOC_FREE);
+ kasan_poison_memory(object, cache->object_size, KASAN_KMALLOC_FREE);

if (!kasan_stack_collection_enabled())
return false;
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 8e4fa9157a0b..3f64c9ecbcc0 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -82,6 +82,7 @@ void kasan_poison_memory(const void *address, size_t size, u8 value)

* addresses to this function.
*/

address = kasan_reset_tag(address);
+ size = round_up(size, KASAN_GRANULE_SIZE);

shadow_start = kasan_mem_to_shadow(address);
shadow_end = kasan_mem_to_shadow(address + size);

--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

set_tag() already ignores the tag for the generic mode, so just call it
as is. Add a check for the generic mode to assign_tag(), and simplify its
call in ____kasan_kmalloc().

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I18905ca78fb4a3d60e1a34a4ca00247272480438
---
mm/kasan/common.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 69ab880abacc..40ff3ce07a76 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -238,6 +238,9 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object)

static u8 assign_tag(struct kmem_cache *cache, const void *object,

bool init, bool keep_tag)
{
+ if (IS_ENABLED(CONFIG_KASAN_GENERIC))
+ return 0xff;
+
/*
* 1. When an object is kmalloc()'ed, two hooks are called:
* kasan_slab_alloc() and kasan_kmalloc(). We assign the
@@ -280,8 +283,8 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
__memset(alloc_meta, 0, sizeof(*alloc_meta));
}

- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
- object = set_tag(object, assign_tag(cache, object, true, false));
+ /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */
+ object = set_tag(object, assign_tag(cache, object, true, false));

return (void *)object;
}
@@ -362,9 +365,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,
KASAN_GRANULE_SIZE);
redzone_end = round_up((unsigned long)object + cache->object_size,
KASAN_GRANULE_SIZE);
-
- if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
- tag = assign_tag(cache, object, false, keep_tag);
+ tag = assign_tag(cache, object, false, keep_tag);

/* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */
kasan_unpoison_memory(set_tag(object, tag), size);
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

Currently it says that the memory gets poisoned by page_alloc code.
Clarify this by mentioning the specific callback that poisons the
memory.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I1334dffb69b87d7986fab88a1a039cc3ea764725
---
mm/kasan/common.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 40ff3ce07a76..4360292ad7f3 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -436,5 +436,5 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)
{
if (ptr != page_address(virt_to_head_page(ptr)))
kasan_report_invalid_free(ptr, ip);
- /* The object will be poisoned by page_alloc. */
+ /* The object will be poisoned by kasan_free_pages(). */
}
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

KASAN marks caches that are sanitized with the SLAB_KASAN cache flag.
Currently if the metadata that is appended after the object (stores e.g.
stack trace ids) doesn't fit into KMALLOC_MAX_SIZE (can only happen with
SLAB, see the comment in the patch), KASAN turns off sanitization
completely.

With this change sanitization of the object data is always enabled.
However the metadata is only stored when it fits. Instead of checking for
SLAB_KASAN flag accross the code to find out whether the metadata is
there, use cache->kasan_info.alloc/free_meta_offset. As 0 can be a valid
value for free_meta_offset, introduce KASAN_NO_FREE_META as an indicator
that the free metadata is missing.

Along the way rework __kasan_cache_create() and add claryfying comments.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/Icd947e2bea054cb5cfbdc6cf6652227d97032dcb
---
mm/kasan/common.c | 112 +++++++++++++++++++++++++-------------
mm/kasan/generic.c | 15 ++---
mm/kasan/hw_tags.c | 6 +-
mm/kasan/kasan.h | 13 ++++-
mm/kasan/quarantine.c | 8 +++
mm/kasan/report.c | 43 ++++++++-------
mm/kasan/report_sw_tags.c | 7 ++-
mm/kasan/sw_tags.c | 4 ++
8 files changed, 138 insertions(+), 70 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 4360292ad7f3..940b42231069 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -109,9 +109,6 @@ void __kasan_free_pages(struct page *page, unsigned int order)
*/

static inline unsigned int optimal_redzone(unsigned int object_size)

{
- if (!IS_ENABLED(CONFIG_KASAN_GENERIC))
- return 0;
-
return
object_size <= 64 - 16 ? 16 :
object_size <= 128 - 32 ? 32 :
@@ -125,47 +122,79 @@ static inline unsigned int optimal_redzone(unsigned int object_size)

void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,

slab_flags_t *flags)
{
- unsigned int orig_size = *size;
+ unsigned int ok_size;
unsigned int redzone_size;
- int redzone_adjust;
+ unsigned int optimal_size;
+
+ /*
+ * SLAB_KASAN is used to mark caches as ones that are sanitized by
+ * KASAN. Currently this is used in two places:
+ * 1. In slab_ksize() when calculating the size of the accessible
+ * memory within the object.
+ * 2. In slab_common.c to prevent merging of sanitized caches.
+ */
+ *flags |= SLAB_KASAN;

- if (!kasan_stack_collection_enabled()) {
- *flags |= SLAB_KASAN;
+ if (!kasan_stack_collection_enabled())
return;
- }

- /* Add alloc meta. */
+ ok_size = *size;
+
+ /* Add alloc meta into redzone. */

cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);

- /* Add free meta. */
- if (IS_ENABLED(CONFIG_KASAN_GENERIC) &&
- (cache->flags & SLAB_TYPESAFE_BY_RCU || cache->ctor ||
- cache->object_size < sizeof(struct kasan_free_meta))) {
- cache->kasan_info.free_meta_offset = *size;
- *size += sizeof(struct kasan_free_meta);
+ /*
+ * If alloc meta doesn't fit, don't add it.
+ * This can only happen with SLAB, as it has KMALLOC_MAX_SIZE equal
+ * to KMALLOC_MAX_CACHE_SIZE and doesn't fall back to page_alloc for
+ * larger sizes.
+ */
+ if (*size > KMALLOC_MAX_SIZE) {
+ cache->kasan_info.alloc_meta_offset = 0;
+ *size = ok_size;
+ /* Continue, since free meta might still fit. */
}

- redzone_size = optimal_redzone(cache->object_size);
- redzone_adjust = redzone_size - (*size - cache->object_size);
- if (redzone_adjust > 0)
- *size += redzone_adjust;
-
- *size = min_t(unsigned int, KMALLOC_MAX_SIZE,
- max(*size, cache->object_size + redzone_size));
+ /* Only the generic mode uses free meta or flexible redzones. */
+ if (!IS_ENABLED(CONFIG_KASAN_GENERIC)) {
+ cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
+ return;
+ }

/*
- * If the metadata doesn't fit, don't enable KASAN at all.
+ * Add free meta into redzone when it's not possible to store
+ * it in the object. This is the case when:
+ * 1. Object is SLAB_TYPESAFE_BY_RCU, which means that is can
+ * be touched after it was freed, or
+ * 2. Object has a constructor, which means it's expected to
+ * retain its content until the next allocation, or
+ * 3. Object is too small.
+ * Otherwise cache->kasan_info.free_meta_offset = 0 is implied.
*/
- if (*size <= cache->kasan_info.alloc_meta_offset ||
- *size <= cache->kasan_info.free_meta_offset) {
- cache->kasan_info.alloc_meta_offset = 0;
- cache->kasan_info.free_meta_offset = 0;
- *size = orig_size;
- return;
+ if (cache->flags & SLAB_TYPESAFE_BY_RCU || cache->ctor ||
+ cache->object_size < sizeof(struct kasan_free_meta)) {
+ ok_size = *size;
+
+ cache->kasan_info.free_meta_offset = *size;
+ *size += sizeof(struct kasan_free_meta);
+
+ /* If free meta doesn't fit, don't add it. */
+ if (*size > KMALLOC_MAX_SIZE) {
+ cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
+ *size = ok_size;
+ }
}

- *flags |= SLAB_KASAN;
+ redzone_size = optimal_redzone(cache->object_size);
+ /* Calculate size with optimal redzone. */
+ optimal_size = cache->object_size + redzone_size;
+ /* Limit it with KMALLOC_MAX_SIZE (relevant for SLAB only). */
+ if (optimal_size > KMALLOC_MAX_SIZE)
+ optimal_size = KMALLOC_MAX_SIZE;
+ /* Use optimal size if the size with added metas is not large enough. */
+ if (*size < optimal_size)
+ *size = optimal_size;
}

size_t __kasan_metadata_size(struct kmem_cache *cache)
@@ -181,15 +210,21 @@ size_t __kasan_metadata_size(struct kmem_cache *cache)
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
+ if (!cache->kasan_info.alloc_meta_offset)
+ return NULL;

return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

+#ifdef CONFIG_KASAN_GENERIC

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);

+ if (cache->kasan_info.free_meta_offset == KASAN_NO_FREE_META)
+ return NULL;

return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

+#endif

void __kasan_unpoison_data(const void *addr, size_t size)

{
@@ -276,11 +311,9 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

if (kasan_stack_collection_enabled()) {

- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;

-
alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ if (alloc_meta)
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

}

/* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */

@@ -319,8 +352,7 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
if (!kasan_stack_collection_enabled())
return false;

- if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
- unlikely(!(cache->flags & SLAB_KASAN)))
+ if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine))
return false;

kasan_set_free_info(cache, object, tag);
@@ -345,7 +377,11 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip)

static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{

- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

+ struct kasan_alloc_meta *alloc_meta;
+

+ alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (alloc_meta)
+ kasan_set_track(&alloc_meta->alloc_track, flags);

}

static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,

@@ -372,7 +408,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,

kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);

- if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))
+ if (kasan_stack_collection_enabled())

set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);

diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index d259e4c3aefd..97e39516f8fe 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -338,10 +338,10 @@ void kasan_record_aux_stack(void *addr)

cache = page->slab_cache;
object = nearest_obj(cache, page, addr);

alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return;

- /*
- * record the last two call_rcu() call stacks.
- */
+ /* Record the last two call_rcu() call stacks. */

alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];

alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
}

@@ -352,11 +352,11 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_free_meta *free_meta;

free_meta = kasan_get_free_meta(cache, object);
- kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
+ if (!free_meta)
+ return;

- /*
- * the object was freed and has free track set
- */
+ kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
+ /* The object was freed and has free track set. */
*(u8 *)kasan_mem_to_shadow(object) = KASAN_KMALLOC_FREETRACK;
}

@@ -365,5 +365,6 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;

+ /* Free meta must be present with KASAN_KMALLOC_FREETRACK. */
return &kasan_get_free_meta(cache, object)->free_track;
}
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 52984825c75f..a0bc7db4e8ff 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -176,7 +176,8 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

alloc_meta = kasan_get_alloc_meta(cache, object);
- kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
+ if (alloc_meta)
+ kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);

}

struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

@@ -185,5 +186,8 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return NULL;
+
return &alloc_meta->free_track[0];
}
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 8a4cd9618142..14ab24931287 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -154,20 +154,31 @@ struct kasan_alloc_meta {
struct qlist_node {
struct qlist_node *next;
};
+
+/*
+ * Generic mode either stores free meta in the object itself or in the redzone
+ * after the object. In the former case free meta offset is 0, in the latter
+ * case it has some sane value smaller than INT_MAX. Use INT_MAX as free meta
+ * offset when free meta isn't present.
+ */
+#define KASAN_NO_FREE_META (INT_MAX)
+
struct kasan_free_meta {
+#ifdef CONFIG_KASAN_GENERIC
/* This field is used while the object is in the quarantine.
* Otherwise it might be used for the allocator freelist.
*/
struct qlist_node quarantine_link;
-#ifdef CONFIG_KASAN_GENERIC
struct kasan_track free_track;
#endif
};

struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object);
+#ifdef CONFIG_KASAN_GENERIC

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object);
+#endif

#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index 0da3d37e1589..23f6bfb1e73f 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -135,7 +135,12 @@ static void qlink_free(struct qlist_node *qlink, struct kmem_cache *cache)
if (IS_ENABLED(CONFIG_SLAB))
local_irq_save(flags);

+ /*
+ * As the object now gets freed from the quaratine, assume that its
+ * free track is now longer valid.
+ */
*(u8 *)kasan_mem_to_shadow(object) = KASAN_KMALLOC_FREE;
+
___cache_free(cache, object, _THIS_IP_);

if (IS_ENABLED(CONFIG_SLAB))
@@ -168,6 +173,9 @@ void quarantine_put(struct kmem_cache *cache, void *object)

struct qlist_head temp = QLIST_INIT;

struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);

+ if (!meta)
+ return;
+

/*
* Note: irq must be disabled until after we move the batch to the

* global quarantine. Otherwise quarantine_remove_cache() can miss
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 7d86af340148..6a95ad2dee91 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -168,32 +168,35 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,

static void describe_object_stacks(struct kmem_cache *cache, void *object,

const void *addr, u8 tag)

{
- struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);
-
- if (cache->flags & SLAB_KASAN) {
- struct kasan_track *free_track;
+ struct kasan_alloc_meta *alloc_meta;
+ struct kasan_track *free_track;

+ alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (alloc_meta) {

print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");

- free_track = kasan_get_free_track(cache, object, tag);
- if (free_track) {
- print_track(free_track, "Freed");
- pr_err("\n");
- }
+ }
+
+ free_track = kasan_get_free_track(cache, object, tag);
+ if (free_track) {
+ print_track(free_track, "Freed");
+ pr_err("\n");
+ }

#ifdef CONFIG_KASAN_GENERIC
- if (alloc_meta->aux_stack[0]) {
- pr_err("Last call_rcu():\n");
- print_stack(alloc_meta->aux_stack[0]);
- pr_err("\n");
- }
- if (alloc_meta->aux_stack[1]) {
- pr_err("Second to last call_rcu():\n");
- print_stack(alloc_meta->aux_stack[1]);
- pr_err("\n");
- }
-#endif
+ if (!alloc_meta)
+ return;

+ if (alloc_meta->aux_stack[0]) {

+ pr_err("Last call_rcu():\n");

+ print_stack(alloc_meta->aux_stack[0]);

+ pr_err("\n");

}
+ if (alloc_meta->aux_stack[1]) {

+ pr_err("Second to last call_rcu():\n");

+ print_stack(alloc_meta->aux_stack[1]);

+ pr_err("\n");
+ }
+#endif

}

static void describe_object(struct kmem_cache *cache, void *object,

diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index 7604b46239d4..11dc8739e500 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -48,9 +48,10 @@ const char *get_bug_type(struct kasan_access_info *info)

object = nearest_obj(cache, page, (void *)addr);

alloc_meta = kasan_get_alloc_meta(cache, object);

- for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
- if (alloc_meta->free_pointer_tag[i] == tag)
- return "use-after-free";
+ if (alloc_meta)
+ for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
+ if (alloc_meta->free_pointer_tag[i] == tag)
+ return "use-after-free";
return "out-of-bounds";
}

diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index d1af6f6c6d12..be10d16bd129 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -170,6 +170,8 @@ void kasan_set_free_info(struct kmem_cache *cache,
u8 idx = 0;

alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return;

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;

@@ -187,6 +189,8 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
int i = 0;

alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return NULL;

[Andrey Konovalov]

The reason cache merging is disabled with KASAN is because KASAN puts its
metadata right after the allocated object. When the merged caches have
slightly different sizes, the metadata ends up in different places, which
KASAN doesn't support.

It might be possible to adjust the metadata allocation algorithm and make
it friendly to the cache merging code. Instead this change takes a simpler
approach and allows merging caches when no metadata is present. Which is
the case for hardware tag-based KASAN with kasan.mode=prod.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/Ia114847dfb2244f297d2cb82d592bf6a07455dba
---
include/linux/kasan.h | 26 ++++++++++++++++++++++++--
mm/kasan/common.c | 11 +++++++++++
mm/slab_common.c | 11 ++++++++---
3 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index d47601517dad..fb8ba4719e3b 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -79,17 +79,35 @@ struct kasan_cache {
};

#ifdef CONFIG_KASAN_HW_TAGS
+
DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
+
static inline kasan_enabled(void)
{
return static_branch_likely(&kasan_flag_enabled);
}
-#else
+
+slab_flags_t __kasan_never_merge(slab_flags_t flags);
+static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
+{
+ if (kasan_enabled())
+ return __kasan_never_merge(flags);
+ return flags;
+}
+
+#else /* CONFIG_KASAN_HW_TAGS */
+
static inline kasan_enabled(void)
{
return true;
}
-#endif
+
+static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
+{
+ return flags;
+}

+
+#endif /* CONFIG_KASAN_HW_TAGS */

void __kasan_alloc_pages(struct page *page, unsigned int order);
static inline void kasan_alloc_pages(struct page *page, unsigned int order)
@@ -238,6 +256,10 @@ static inline kasan_enabled(void)
{
return false;
}
+static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
+{
+ return flags;
+}
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

static inline void kasan_cache_create(struct kmem_cache *cache,

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 940b42231069..25b18c145b06 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -81,6 +81,17 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)
}
#endif /* CONFIG_KASAN_STACK */

+/*
+ * Only allow cache merging when stack collection is disabled and no metadata
+ * is present.
+ */
+slab_flags_t __kasan_never_merge(slab_flags_t flags)
+{
+ if (kasan_stack_collection_enabled())
+ return flags;
+ return flags & ~SLAB_KASAN;
+}
+
void __kasan_alloc_pages(struct page *page, unsigned int order)
{
u8 tag;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index f1b0c4a22f08..3042ee8ea9ce 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -18,6 +18,7 @@
#include <linux/seq_file.h>
#include <linux/proc_fs.h>
#include <linux/debugfs.h>
+#include <linux/kasan.h>
#include <asm/cacheflush.h>
#include <asm/tlbflush.h>
#include <asm/page.h>
@@ -49,12 +50,16 @@ static DECLARE_WORK(slab_caches_to_rcu_destroy_work,
slab_caches_to_rcu_destroy_workfn);

/*
- * Set of flags that will prevent slab merging
+ * Set of flags that will prevent slab merging.
+ * Use slab_never_merge() instead.
*/
#define SLAB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
SLAB_TRACE | SLAB_TYPESAFE_BY_RCU | SLAB_NOLEAKTRACE | \
SLAB_FAILSLAB | SLAB_KASAN)

+/* KASAN allows merging in some configurations and will remove SLAB_KASAN. */
+#define slab_never_merge() (kasan_never_merge(SLAB_NEVER_MERGE))
+
#define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \
SLAB_CACHE_DMA32 | SLAB_ACCOUNT)

@@ -164,7 +169,7 @@ static unsigned int calculate_alignment(slab_flags_t flags,
*/
int slab_unmergeable(struct kmem_cache *s)
{
- if (slab_nomerge || (s->flags & SLAB_NEVER_MERGE))
+ if (slab_nomerge || (s->flags & slab_never_merge()))
return 1;

if (s->ctor)
@@ -198,7 +203,7 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,
size = ALIGN(size, align);
flags = kmem_cache_flags(size, flags, name, NULL);

- if (flags & SLAB_NEVER_MERGE)
+ if (flags & slab_never_merge())
return NULL;

list_for_each_entry_reverse(s, &slab_caches, list) {
--
2.29.1.341.ge80a0c044ae-goog

[Andrey Konovalov]

This change updates KASAN documentation to reflect the addition of boot
parameters and also reworks and clarifies some of the existing sections,
in particular: defines what a memory granule is, mentions quarantine,
makes Kunit section more readable.

Signed-off-by: Andrey Konovalov <andre...@google.com>
---
Documentation/dev-tools/kasan.rst | 180 +++++++++++++++++++-----------
1 file changed, 113 insertions(+), 67 deletions(-)

diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
index 422f8ee1bb17..ae1e732ebdfa 100644
--- a/Documentation/dev-tools/kasan.rst
+++ b/Documentation/dev-tools/kasan.rst
@@ -6,6 +6,7 @@ Overview

KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to
find out-of-bound and use-after-free bugs. KASAN has three modes:
+
1. generic KASAN (similar to userspace ASan),
2. software tag-based KASAN (similar to userspace HWASan),
3. hardware tag-based KASAN (based on hardware memory tagging).
@@ -39,23 +40,13 @@ CONFIG_KASAN_INLINE. Outline and inline are compiler instrumentation types.
The former produces smaller binary while the latter is 1.1 - 2 times faster.

Both software KASAN modes work with both SLUB and SLAB memory allocators,
-hardware tag-based KASAN currently only support SLUB.
-For better bug detection and nicer reporting, enable CONFIG_STACKTRACE.
+while the hardware tag-based KASAN currently only support SLUB.
+
+For better error reports that include stack traces, enable CONFIG_STACKTRACE.

To augment reports with last allocation and freeing stack of the physical page,
it is recommended to enable also CONFIG_PAGE_OWNER and boot with page_owner=on.

-To disable instrumentation for specific files or directories, add a line
-similar to the following to the respective kernel Makefile:
-
-- For a single file (e.g. main.o)::
-
- KASAN_SANITIZE_main.o := n
-
-- For all files in one directory::
-
- KASAN_SANITIZE := n
-
Error reports
~~~~~~~~~~~~~

@@ -140,16 +131,20 @@ freed (in case of a use-after-free bug report). Next comes a description of
the accessed slab object and information about the accessed memory page.

In the last section the report shows memory state around the accessed address.
-Reading this part requires some understanding of how KASAN works.
-
-The state of each 8 aligned bytes of memory is encoded in one shadow byte.
-Those 8 bytes can be accessible, partially accessible, freed or be a redzone.
-We use the following encoding for each shadow byte: 0 means that all 8 bytes
-of the corresponding memory region are accessible; number N (1 <= N <= 7) means
-that the first N bytes are accessible, and other (8 - N) bytes are not;
-any negative value indicates that the entire 8-byte word is inaccessible.
-We use different negative values to distinguish between different kinds of
-inaccessible memory like redzones or freed memory (see mm/kasan/kasan.h).
+Internally KASAN tracks memory state separately for each memory granule, which
+is either 8 or 16 aligned bytes depending on KASAN mode. Each number in the
+memory state section of the report shows the state of one of the memory
+granules that surround the accessed address.
+
+For generic KASAN the size of each memory granule is 8. The state of each
+granule is encoded in one shadow byte. Those 8 bytes can be accessible,
+partially accessible, freed or be a part of a redzone. KASAN uses the following
+encoding for each shadow byte: 0 means that all 8 bytes of the corresponding
+memory region are accessible; number N (1 <= N <= 7) means that the first N
+bytes are accessible, and other (8 - N) bytes are not; any negative value
+indicates that the entire 8-byte word is inaccessible. KASAN uses different
+negative values to distinguish between different kinds of inaccessible memory
+like redzones or freed memory (see mm/kasan/kasan.h).

In the report above the arrows point to the shadow byte 03, which means that
the accessed address is partially accessible.
@@ -157,6 +152,55 @@ the accessed address is partially accessible.
For tag-based KASAN this last report section shows the memory tags around the
accessed address (see Implementation details section).

+Boot parameters
+~~~~~~~~~~~~~~~
+
+Hardware tag-based KASAN mode (see the section about different mode below) is
+intended for use in production as a security mitigation. Therefore it supports
+boot parameters that allow to disable KASAN competely or otherwise control
+particular KASAN features.
+
+The things that can be controlled are:
+
+1. Whether KASAN is enabled at all.
+2. Whether KASAN collects and saves alloc/free stacks.
+3. Whether KASAN panics on a detected bug or not.
+
+The ``kasam.mode`` boot parameter allows to choose one of three main modes:
+
+- ``kasan.mode=off`` - KASAN is disabled, no tag checks are performed
+- ``kasan.mode=prod`` - only essential production features are enabled
+- ``kasan.mode=full`` - all KASAN features are enabled
+
+The chosen mode provides default control values for the features mentioned
+above. However it's also possible to override the default values by providing:
+
+- ``kasan.stack=off`` or ``=on`` - enable alloc/free stacks collection
+ (default: ``on`` for ``mode=full``,
+ otherwise ``off``)
+- ``kasan.fault=report`` or ``=panic`` - only print KASAN report or also panic
+ (default: ``report``)
+
+If ``kasan.mode parameter`` is not provided, it defaults to ``full`` when
+``CONFIG_DEBUG_KERNEL`` is enabled, and to ``prod`` otherwise.
+
+For developers
+~~~~~~~~~~~~~~
+
+Software KASAN modes use compiler instrumentation to insert validity checks.
+Such instrumentation might be incompatible with some part of the kernel, and
+therefore needs to be disabled. To disable instrumentation for specific files
+or directories, add a line similar to the following to the respective kernel
+Makefile:
+
+- For a single file (e.g. main.o)::
+
+ KASAN_SANITIZE_main.o := n
+
+- For all files in one directory::
+
+ KASAN_SANITIZE := n
+

Implementation details
----------------------
@@ -164,10 +208,10 @@ Implementation details
Generic KASAN
~~~~~~~~~~~~~

-From a high level, our approach to memory error detection is similar to that
-of kmemcheck: use shadow memory to record whether each byte of memory is safe
-to access, and use compile-time instrumentation to insert checks of shadow
-memory on each memory access.
+From a high level perspective, KASAN's approach to memory error detection is
+similar to that of kmemcheck: use shadow memory to record whether each byte of
+memory is safe to access, and use compile-time instrumentation to insert checks
+of shadow memory on each memory access.

Generic KASAN dedicates 1/8th of kernel memory to its shadow memory (e.g. 16TB
to cover 128TB on x86_64) and uses direct mapping with a scale and offset to
@@ -194,7 +238,10 @@ function calls GCC directly inserts the code to check the shadow memory.
This option significantly enlarges kernel but it gives x1.1-x2 performance
boost over outline instrumented kernel.

-Generic KASAN prints up to 2 call_rcu() call stacks in reports, the last one
+Generic KASAN is the only mode that delays the reuse of freed object via
+quarantine (see mm/kasan/quarantine.c for implementation).
+
+Generic KASAN prints up to two call_rcu() call stacks in reports, the last one
and the second to last.

Software tag-based KASAN
@@ -302,15 +349,15 @@ therefore be wasteful. Furthermore, to ensure that different mappings
use different shadow pages, mappings would have to be aligned to
``KASAN_GRANULE_SIZE * PAGE_SIZE``.

-Instead, we share backing space across multiple mappings. We allocate
+Instead, KASAN shares backing space across multiple mappings. It allocates
a backing page when a mapping in vmalloc space uses a particular page
of the shadow region. This page can be shared by other vmalloc
mappings later on.

-We hook in to the vmap infrastructure to lazily clean up unused shadow
+KASAN hooks in to the vmap infrastructure to lazily clean up unused shadow
memory.

-To avoid the difficulties around swapping mappings around, we expect
+To avoid the difficulties around swapping mappings around, KASAN expects
that the part of the shadow region that covers the vmalloc space will
not be covered by the early shadow page, but will be left
unmapped. This will require changes in arch-specific code.
@@ -321,24 +368,31 @@ architectures that do not have a fixed module region.
CONFIG_KASAN_KUNIT_TEST & CONFIG_TEST_KASAN_MODULE
--------------------------------------------------

-``CONFIG_KASAN_KUNIT_TEST`` utilizes the KUnit Test Framework for testing.
-This means each test focuses on a small unit of functionality and
-there are a few ways these tests can be run.
+KASAN tests consist on two parts:
+
+1. Tests that are integrated with the KUnit Test Framework. Enabled with
+``CONFIG_KASAN_KUNIT_TEST``. These tests can be run and partially verified
+automatically in a few different ways, see the instructions below.

-Each test will print the KASAN report if an error is detected and then
-print the number of the test and the status of the test:
+2. Tests that are currently incompatible with Kunit. Enabled with
+``CONFIG_TEST_KASAN_MODULE`` and can only be run as a module. These tests can
+only be verified manually, by loading the kernel module and inspecting the
+kernel log for KASAN reports.

-pass::
+Each KUNIT-compatible KASAN test prints a KASAN report if an error is detected.
+Then the test prints its number and status.
+
+When a test passes::

ok 28 - kmalloc_double_kzfree

-or, if kmalloc failed::
+When a test fails due to a failed ``kmalloc``::

# kmalloc_large_oob_right: ASSERTION FAILED at lib/test_kasan.c:163
Expected ptr is not null, but is
not ok 4 - kmalloc_large_oob_right

-or, if a KASAN report was expected, but not found::
+When a test fails due to a missing KASAN report::

# kmalloc_double_kzfree: EXPECTATION FAILED at lib/test_kasan.c:629
Expected kasan_data->report_expected == kasan_data->report_found, but
@@ -346,46 +400,38 @@ or, if a KASAN report was expected, but not found::
kasan_data->report_found == 0
not ok 28 - kmalloc_double_kzfree

-All test statuses are tracked as they run and an overall status will
-be printed at the end::
+At the end the cumulative status of all KASAN tests is printed. On success::

ok 1 - kasan

-or::
+Or, if one of the tests failed::

not ok 1 - kasan

-(1) Loadable Module
-~~~~~~~~~~~~~~~~~~~~
+
+There are a few ways to run Kunit-compatible KASAN tests.
+
+1. Loadable module
+~~~~~~~~~~~~~~~~~~

With ``CONFIG_KUNIT`` enabled, ``CONFIG_KASAN_KUNIT_TEST`` can be built as
-a loadable module and run on any architecture that supports KASAN
-using something like insmod or modprobe. The module is called ``test_kasan``.
+a loadable module and run on any architecture that supports KASAN by loading
+the module with insmod or modprobe. The module is called ``test_kasan``.

-(2) Built-In
-~~~~~~~~~~~~~
+2. Built-In
+~~~~~~~~~~~

With ``CONFIG_KUNIT`` built-in, ``CONFIG_KASAN_KUNIT_TEST`` can be built-in
-on any architecure that supports KASAN. These and any other KUnit
-tests enabled will run and print the results at boot as a late-init
-call.
+on any architecure that supports KASAN. These and any other KUnit tests enabled
+will run and print the results at boot as a late-init call.

-(3) Using kunit_tool
-~~~~~~~~~~~~~~~~~~~~~
+3. Using kunit_tool
+~~~~~~~~~~~~~~~~~~~

-With ``CONFIG_KUNIT`` and ``CONFIG_KASAN_KUNIT_TEST`` built-in, we can also
-use kunit_tool to see the results of these along with other KUnit
-tests in a more readable way. This will not print the KASAN reports
-of tests that passed. Use `KUnit documentation <https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html>`_ for more up-to-date
-information on kunit_tool.
+With ``CONFIG_KUNIT`` and ``CONFIG_KASAN_KUNIT_TEST`` built-in, it's also
+possible use ``kunit_tool`` to see the results of these and other KUnit tests
+in a more readable way. This will not print the KASAN reports of the tests that
+passed. Use `KUnit documentation <https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html>`_
+for more up-to-date information on ``kunit_tool``.

.. _KUnit: https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html
-
-``CONFIG_TEST_KASAN_MODULE`` is a set of KASAN tests that could not be
-converted to KUnit. These tests can be run only as a module with
-``CONFIG_TEST_KASAN_MODULE`` built as a loadable module and
-``CONFIG_KASAN`` built-in. The type of error expected and the
-function being run is printed before the expression expected to give
-an error. Then the error is printed, if found, and that test
-should be interpretted to pass only if the error was the one expected
-by the test.
--
2.29.1.341.ge80a0c044ae-goog

[Evgenii Stepanov]

I think this was discussed before, but should this be kasan.stacktrace
or something like that?
In other places "kasan stack" refers to stack instrumentation, not
stack trace collection.
Ex.: CONFIG_KASAN_STACK

[Andrey Konovalov]

On Thu, Nov 5, 2020 at 9:49 PM Evgenii Stepanov <eug...@google.com> wrote:
>
> > The chosen mode provides default control values for the features mentioned
> > above. However it's also possible to override the default values by
> > providing:
> >
> > - kasan.stack=off/on - enable stacks collection
> > (default: on for mode=full, otherwise off)
>
> I think this was discussed before, but should this be kasan.stacktrace
> or something like that?
> In other places "kasan stack" refers to stack instrumentation, not
> stack trace collection.
> Ex.: CONFIG_KASAN_STACK

Forgot to update it here, but it's kasan.stacks now (with an s at the
end). kasan.stacktrace might be better, although it's somewhat long.
WDYT?

[Evgenii Stepanov]

I like kasan.stacktrace, but I would not insist.

[Andrey Konovalov]

=== Overview

Hardware tag-based KASAN mode [1] is intended to eventually be used in
production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.

This patchset adds a few boot parameters for hardware tag-based KASAN that

allow to disable or otherwise control particular KASAN features, as well
as provides some initial optimizations for running KASAN in production.

There's another planned patchset what will further optimize hardware
tag-based KASAN, provide proper benchmarking and tests, and will fully
enable tag-based KASAN for production use.

Hardware tag-based KASAN relies on arm64 Memory Tagging Extension (MTE)
[2] to perform memory and pointer tagging. Please see [3] and [4] for
detailed analysis of how MTE helps to fight memory safety problems.

The features that can be controlled are:

1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.

The patch titled "kasan: add and integrate kasan boot parameters" of this
series adds a few new boot parameters.

kasan.mode allows to choose one of three main modes:

- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled

The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:

- kasan.stacktrace=off/on - enable stacks collection

(default: on for mode=full, otherwise off)

https://github.com/xairy/linux/tree/up-boot-mte-v2

and on Gerrit here:

https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/3707

This patchset is based on v9 of "kasan: add hardware tag-based mode for

arm64" patchset [1].

For testing in QEMU hardware tag-based KASAN requires:

1. QEMU built from master [6] (use "-machine virt,mte=on -cpu max" arguments
to run).
2. GCC version 10.

[1] https://lkml.org/lkml/2020/11/10/1187

[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] https://arxiv.org/pdf/1802.09517.pdf
[4] https://github.com/microsoft/MSRC-Security-Research/blob/master/papers/2020/Security%20analysis%20of%20memory%20tagging.pdf
[5] https://source.android.com/devices/architecture/kernel/generic-kernel-image
[6] https://github.com/qemu/qemu

=== History

Changes v1 -> v2:
- Rebased onto v9 of the HW_TAGS patchset.
- Don't initialize static branches in kasan_init_hw_tags_cpu(), as
cpu_enable_mte() can't sleep; do in in kasan_init_hw_tags() instead.
- Rename kasan.stacks to kasan.stacktrace.

mm/kasan/hw_tags.c | 183 ++++++++++++++++----

mm/kasan/kasan.h | 113 ++++++++----
mm/kasan/quarantine.c | 13 +-
mm/kasan/report.c | 61 ++++---
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 13 +-
mm/kasan/shadow.c | 5 +-
mm/kasan/sw_tags.c | 17 +-
mm/mempool.c | 2 +-
mm/slab_common.c | 13 +-

19 files changed, 826 insertions(+), 366 deletions(-)

--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Move get_free_info() call into quarantine_put() to simplify the call site.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Iab0f04e7ebf8d83247024b7190c67c3c34c7940f
---
mm/kasan/common.c | 2 +-
mm/kasan/kasan.h | 5 ++---
mm/kasan/quarantine.c | 3 ++-

3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 2bb0ef6da6bd..5712c66c11c1 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c

@@ -308,7 +308,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

kasan_set_free_info(cache, object, tag);

- quarantine_put(get_free_info(cache, object), cache);
+ quarantine_put(cache, object);

return IS_ENABLED(CONFIG_KASAN_GENERIC);
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 21fe75c66f26..c2c40ec1544d 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h

@@ -214,12 +214,11 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

#if defined(CONFIG_KASAN_GENERIC) && \
(defined(CONFIG_SLAB) || defined(CONFIG_SLUB))
-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache);
+void quarantine_put(struct kmem_cache *cache, void *object);
void quarantine_reduce(void);
void quarantine_remove_cache(struct kmem_cache *cache);
#else
-static inline void quarantine_put(struct kasan_free_meta *info,
- struct kmem_cache *cache) { }

+static inline void quarantine_put(struct kmem_cache *cache, void *object) { }

static inline void quarantine_reduce(void) { }
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif

diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index 580ff5610fc1..a0792f0d6d0f 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c

@@ -161,11 +161,12 @@ static void qlist_free_all(struct qlist_head *q, struct kmem_cache *cache)
qlist_init(q);
}

-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache)

+void quarantine_put(struct kmem_cache *cache, void *object)
{

unsigned long flags;
struct qlist_head *q;

struct qlist_head temp = QLIST_INIT;

+ struct kasan_free_meta *info = get_free_info(cache, object);

/*
* Note: irq must be disabled until after we move the batch to the

--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Rename get_alloc_info() and get_free_info() to kasan_get_alloc_meta()
and kasan_get_free_meta() to better reflect what those do and avoid
confusion with kasan_set_free_info().

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ib6e4ba61c8b12112b403d3479a9799ac8fff8de1
---
mm/kasan/common.c | 16 ++++++++--------
mm/kasan/generic.c | 12 ++++++------
mm/kasan/hw_tags.c | 4 ++--
mm/kasan/kasan.h | 8 ++++----
mm/kasan/quarantine.c | 4 ++--
mm/kasan/report.c | 12 ++++++------
mm/kasan/report_sw_tags.c | 2 +-
mm/kasan/sw_tags.c | 4 ++--

8 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 5712c66c11c1..8fd04415d8f4 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c

@@ -175,14 +175,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
sizeof(struct kasan_free_meta) : 0);
}

-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object)
{

return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object)

+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

+ const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);

return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;

@@ -259,13 +259,13 @@ static u8 assign_tag(struct kmem_cache *cache, const void *object,
void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
const void *object)
{

- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;

if (!(cache->flags & SLAB_KASAN))

return (void *)object;

- alloc_info = get_alloc_info(cache, object);
- __memset(alloc_info, 0, sizeof(*alloc_info));

+ alloc_meta = kasan_get_alloc_meta(cache, object);

+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))

object = set_tag(object, assign_tag(cache, object, true, false));

@@ -345,7 +345,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
KASAN_KMALLOC_REDZONE);

if (cache->flags & SLAB_KASAN)

- kasan_set_track(&get_alloc_info(cache, object)->alloc_track, flags);

+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

return set_tag(object, tag);
}
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c

index adb254df1b1d..d259e4c3aefd 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c

@@ -329,7 +329,7 @@ void kasan_record_aux_stack(void *addr)
{
struct page *page = kasan_addr_to_page(addr);
struct kmem_cache *cache;
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;
void *object;

if (!(page && PageSlab(page)))

@@ -337,13 +337,13 @@ void kasan_record_aux_stack(void *addr)

cache = page->slab_cache;
object = nearest_obj(cache, page, addr);

- alloc_info = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

/*

* record the last two call_rcu() call stacks.

*/
- alloc_info->aux_stack[1] = alloc_info->aux_stack[0];
- alloc_info->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
+ alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];
+ alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
}

void kasan_set_free_info(struct kmem_cache *cache,

@@ -351,7 +351,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
{
struct kasan_free_meta *free_meta;

- free_meta = get_free_info(cache, object);
+ free_meta = kasan_get_free_meta(cache, object);
kasan_set_track(&free_meta->free_track, GFP_NOWAIT);

/*

@@ -365,5 +365,5 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;

- return &get_free_info(cache, object)->free_track;

+ return &kasan_get_free_meta(cache, object)->free_track;
}
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 0080b78ec843..70b88dd40cd8 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -66,7 +66,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
}

@@ -75,6 +75,6 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

{
struct kasan_alloc_meta *alloc_meta;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

return &alloc_meta->free_track[0];
}
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index c2c40ec1544d..db8a7a508121 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h

@@ -148,10 +148,10 @@ struct kasan_free_meta {
#endif
};

-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object);
-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object);
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object);
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object);

void kasan_poison_memory(const void *address, size_t size, u8 value);

diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index a0792f0d6d0f..0da3d37e1589 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -166,7 +166,7 @@ void quarantine_put(struct kmem_cache *cache, void *object)

unsigned long flags;
struct qlist_head *q;
struct qlist_head temp = QLIST_INIT;

- struct kasan_free_meta *info = get_free_info(cache, object);
+ struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);

/*
* Note: irq must be disabled until after we move the batch to the

@@ -179,7 +179,7 @@ void quarantine_put(struct kmem_cache *cache, void *object)
local_irq_save(flags);

q = this_cpu_ptr(&cpu_quarantine);
- qlist_put(q, &info->quarantine_link, cache->size);
+ qlist_put(q, &meta->quarantine_link, cache->size);
if (unlikely(q->bytes > QUARANTINE_PERCPU_SIZE)) {
qlist_move_all(q, &temp);

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index ce06005d4052..0cac53a57c14 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -164,12 +164,12 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,

static void describe_object(struct kmem_cache *cache, void *object,

const void *addr, u8 tag)
{

- struct kasan_alloc_meta *alloc_info = get_alloc_info(cache, object);

+ struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);

if (cache->flags & SLAB_KASAN) {

struct kasan_track *free_track;

- print_track(&alloc_info->alloc_track, "Allocated");

+ print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");

free_track = kasan_get_free_track(cache, object, tag);

if (free_track) {
@@ -178,14 +178,14 @@ static void describe_object(struct kmem_cache *cache, void *object,
}

#ifdef CONFIG_KASAN_GENERIC

- if (alloc_info->aux_stack[0]) {
+ if (alloc_meta->aux_stack[0]) {

pr_err("Last call_rcu():\n");

- print_stack(alloc_info->aux_stack[0]);

+ print_stack(alloc_meta->aux_stack[0]);

pr_err("\n");
}
- if (alloc_info->aux_stack[1]) {

+ if (alloc_meta->aux_stack[1]) {

pr_err("Second to last call_rcu():\n");

- print_stack(alloc_info->aux_stack[1]);

+ print_stack(alloc_meta->aux_stack[1]);

pr_err("\n");
}
#endif
diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index aebc44a29e83..317100fd95b9 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -46,7 +46,7 @@ const char *get_bug_type(struct kasan_access_info *info)
if (page && PageSlab(page)) {
cache = page->slab_cache;

object = nearest_obj(cache, page, (void *)addr);
- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

for (i = 0; i < KASAN_NR_FREE_STACKS; i++)

if (alloc_meta->free_pointer_tag[i] == tag)

diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index dfe707dd8d0d..3bffb489b144 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -174,7 +174,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

u8 idx = 0;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;

@@ -191,7 +191,7 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;
int i = 0;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {
--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Add set_alloc_info() helper and move kasan_set_track() into it. This will
simplify the code for one of the upcoming changes.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I0316193cbb4ecc9b87b7c2eee0dd79f8ec908c1a
---
mm/kasan/common.c | 7 ++++++-

1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 8fd04415d8f4..a880e5a547ed 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c

@@ -318,6 +318,11 @@ bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
return __kasan_slab_free(cache, object, ip, true);
}

+static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
+{

+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

+}
+

static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

size_t size, gfp_t flags, bool keep_tag)
{

@@ -345,7 +350,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

KASAN_KMALLOC_REDZONE);

if (cache->flags & SLAB_KASAN)

- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

+ set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);
}
--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

There's a config option CONFIG_KASAN_STACK that has to be enabled for
KASAN to use stack instrumentation and perform validity checks for
stack variables.

There's no need to unpoison stack when CONFIG_KASAN_STACK is not enabled.
Only call kasan_unpoison_task_stack[_below]() when CONFIG_KASAN_STACK is
enabled.

Note, that CONFIG_KASAN_STACK is an option that is currently always
defined when CONFIG_KASAN is enabled, and therefore has to be tested
with #if instead of #ifdef.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/If8a891e9fe01ea543e00b576852685afec0887e3
---

arch/arm64/kernel/sleep.S | 2 +-
arch/x86/kernel/acpi/wakeup_64.S | 2 +-

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index f22bdef82111..b9b9db335d87 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -76,8 +76,6 @@ static inline void kasan_disable_current(void) {}

void kasan_unpoison_memory(const void *address, size_t size);

-void kasan_unpoison_task_stack(struct task_struct *task);
-
void kasan_alloc_pages(struct page *page, unsigned int order);
void kasan_free_pages(struct page *page, unsigned int order);

@@ -122,8 +120,6 @@ void kasan_restore_multi_shot(bool enabled);

static inline void kasan_unpoison_memory(const void *address, size_t size) {}

-static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
-

static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

@@ -175,6 +171,12 @@ static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

#endif /* CONFIG_KASAN */

+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
+void kasan_unpoison_task_stack(struct task_struct *task);
+#else
+static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
+#endif
+
#ifdef CONFIG_KASAN_GENERIC

void kasan_cache_shrink(struct kmem_cache *cache);

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a880e5a547ed..a3e67d49b893 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c

@@ -58,6 +58,7 @@ void kasan_disable_current(void)
}

#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

+#if CONFIG_KASAN_STACK

static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)

{
void *base = task_stack_page(task);
@@ -84,6 +85,7 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)

kasan_unpoison_memory(base, watermark - base);
}
+#endif /* CONFIG_KASAN_STACK */

void kasan_alloc_pages(struct page *page, unsigned int order)

{
--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Even though hardware tag-based mode currently doesn't support checking
vmalloc allocations, it doesn't use shadow memory and works with
VMAP_STACK as is. Change VMAP_STACK definition accordingly.

Signed-off-by: Andrey Konovalov <andre...@google.com>

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

There's no need for __kasan_unpoison_stack() helper, as it's only
currently used in a single place. Removing it also removes unneeded
arithmetic.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ie5ba549d445292fe629b4a96735e4034957bcc50
---
mm/kasan/common.c | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a3e67d49b893..9008fc6b0810 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -59,18 +59,12 @@ void kasan_disable_current(void)

#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

#if CONFIG_KASAN_STACK
-static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
-{
- void *base = task_stack_page(task);
- size_t size = sp - base;
-
- kasan_unpoison_memory(base, size);
-}
-
/* Unpoison the entire stack for a task. */

void kasan_unpoison_task_stack(struct task_struct *task)
{

- __kasan_unpoison_stack(task, task_stack_page(task) + THREAD_SIZE);
+ void *base = task_stack_page(task);
+
+ kasan_unpoison_memory(base, THREAD_SIZE);
}

/* Unpoison the stack for the current task beyond a watermark sp value. */
--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Using kasan_reset_tag() currently results in a function call. As it's
called quite often from the allocator code, this leads to a noticeable
slowdown. Move it to include/linux/kasan.h and turn it into a static
inline function. Also remove the now unneeded reset_tag() internal KASAN
macro and use kasan_reset_tag() instead.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/I4d2061acfe91d480a75df00b07c22d8494ef14b5
---
include/linux/kasan.h | 5 ++++-
mm/kasan/common.c | 6 +++---
mm/kasan/hw_tags.c | 9 ++-------
mm/kasan/kasan.h | 4 ----
mm/kasan/report.c | 4 ++--
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 4 ++--
mm/kasan/shadow.c | 4 ++--
mm/kasan/sw_tags.c | 9 ++-------
9 files changed, 18 insertions(+), 29 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index b9b9db335d87..53c8e8b12fbc 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -193,7 +193,10 @@ static inline void kasan_record_aux_stack(void *ptr) {}

#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)

-void *kasan_reset_tag(const void *addr);
+static inline void *kasan_reset_tag(const void *addr)
+{
+ return (void *)arch_kasan_reset_tag(addr);
+}

bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 9008fc6b0810..a266b90636a1 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -174,14 +174,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
- return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object)
{

BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
- return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

void kasan_poison_slab(struct page *page)

@@ -278,7 +278,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

tag = get_tag(object);
tagged_object = object;

- object = reset_tag(object);
+ object = kasan_reset_tag(object);

if (unlikely(nearest_obj(cache, virt_to_head_page(object), object) !=
object)) {

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 70b88dd40cd8..49ea5f5c5643 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -30,20 +30,15 @@ void kasan_init_hw_tags(void)

pr_info("KernelAddressSanitizer initialized\n");
}

-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-

void kasan_poison_memory(const void *address, size_t size, u8 value)

{
- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), value);
}

void kasan_unpoison_memory(const void *address, size_t size)

{
- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index db8a7a508121..8a5501ef2339 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h

@@ -246,15 +246,11 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)
return addr;
}
#endif
-#ifndef arch_kasan_reset_tag
-#define arch_kasan_reset_tag(addr) ((void *)(addr))
-#endif
#ifndef arch_kasan_get_tag
#define arch_kasan_get_tag(addr) 0
#endif

#define set_tag(addr, tag) ((void *)arch_kasan_set_tag((addr), (tag)))
-#define reset_tag(addr) ((void *)arch_kasan_reset_tag(addr))
#define get_tag(addr) arch_kasan_get_tag(addr)

#ifdef CONFIG_KASAN_HW_TAGS

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 0cac53a57c14..25ca66c99e48 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c

@@ -328,7 +328,7 @@ void kasan_report_invalid_free(void *object, unsigned long ip)
unsigned long flags;
u8 tag = get_tag(object);

- object = reset_tag(object);
+ object = kasan_reset_tag(object);

#if IS_ENABLED(CONFIG_KUNIT)
if (current->kunit_test)
@@ -361,7 +361,7 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write,
disable_trace_on_warning();

tagged_addr = (void *)addr;
- untagged_addr = reset_tag(tagged_addr);
+ untagged_addr = kasan_reset_tag(tagged_addr);

info.access_addr = tagged_addr;
if (addr_has_metadata(untagged_addr))
diff --git a/mm/kasan/report_hw_tags.c b/mm/kasan/report_hw_tags.c
index da543eb832cd..57114f0e14d1 100644
--- a/mm/kasan/report_hw_tags.c
+++ b/mm/kasan/report_hw_tags.c

@@ -22,7 +22,7 @@ const char *get_bug_type(struct kasan_access_info *info)

void *find_first_bad_addr(void *addr, size_t size)

{
- return reset_tag(addr);
+ return kasan_reset_tag(addr);
}

void metadata_fetch_row(char *buffer, void *row)

diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index 317100fd95b9..7604b46239d4 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c

@@ -41,7 +41,7 @@ const char *get_bug_type(struct kasan_access_info *info)
int i;

tag = get_tag(info->access_addr);
- addr = reset_tag(info->access_addr);
+ addr = kasan_reset_tag(info->access_addr);
page = kasan_addr_to_page(addr);

if (page && PageSlab(page)) {
cache = page->slab_cache;

@@ -72,7 +72,7 @@ const char *get_bug_type(struct kasan_access_info *info)

void *find_first_bad_addr(void *addr, size_t size)

{
u8 tag = get_tag(addr);
- void *p = reset_tag(addr);
+ void *p = kasan_reset_tag(addr);
void *end = p + size;

while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p))

diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 616ac64c4a21..8e4fa9157a0b 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c

@@ -81,7 +81,7 @@ void kasan_poison_memory(const void *address, size_t size, u8 value)
* some of the callers (e.g. kasan_poison_object_data) pass tagged

* addresses to this function.
*/

- address = reset_tag(address);
+ address = kasan_reset_tag(address);

shadow_start = kasan_mem_to_shadow(address);
shadow_end = kasan_mem_to_shadow(address + size);

@@ -98,7 +98,7 @@ void kasan_unpoison_memory(const void *address, size_t size)
* some of the callers (e.g. kasan_unpoison_object_data) pass tagged

* addresses to this function.
*/

- address = reset_tag(address);
+ address = kasan_reset_tag(address);

kasan_poison_memory(address, size, tag);

diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index 3bffb489b144..d1af6f6c6d12 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Using random_tag() currently results in a function call. Move its

definition to mm/kasan/kasan.h and turn it into a static inline function
for hardware tag-based mode to avoid uneeded function calls.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/Iac5b2faf9a912900e16cca6834d621f5d4abf427
---
mm/kasan/hw_tags.c | 5 -----
mm/kasan/kasan.h | 34 +++++++++++++++++-----------------
2 files changed, 17 insertions(+), 22 deletions(-)

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 49ea5f5c5643..1476ac07666e 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -42,11 +42,6 @@ void kasan_unpoison_memory(const void *address, size_t size)

round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

-u8 random_tag(void)
-{
- return hw_get_random_tag();
-}
-

bool check_invalid_free(void *addr)
{
u8 ptr_tag = get_tag(addr);

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 8a5501ef2339..7498839a15d3 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h

@@ -188,6 +188,12 @@ static inline bool addr_has_metadata(const void *addr)

#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+void print_tags(u8 addr_tag, const void *addr);
+#else
+static inline void print_tags(u8 addr_tag, const void *addr) { }
+#endif
+
bool check_invalid_free(void *addr);

void *find_first_bad_addr(void *addr, size_t size);
@@ -223,23 +229,6 @@ static inline void quarantine_reduce(void) { }

static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif

-#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
-
-void print_tags(u8 addr_tag, const void *addr);
-
-u8 random_tag(void);
-
-#else
-
-static inline void print_tags(u8 addr_tag, const void *addr) { }
-
-static inline u8 random_tag(void)
-{
- return 0;
-}
-
-#endif
-
#ifndef arch_kasan_set_tag

static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)

{
@@ -279,6 +268,17 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)

#endif /* CONFIG_KASAN_HW_TAGS */

+#ifdef CONFIG_KASAN_SW_TAGS
+u8 random_tag(void);
+#elif defined(CONFIG_KASAN_HW_TAGS)
+#define random_tag() hw_get_random_tag()
+#else
+static inline u8 random_tag(void)

+{
+ return 0;
+}

+#endif

+
/*
* Exported functions for interfaces called from assembly or from generated
* code. Declarations here to avoid warning about missing declarations.
--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Using kasan_poison_memory() or check_invalid_free() currently results in
function calls. Move their definitions to mm/kasan/kasan.h and turn them

into static inline functions for hardware tag-based mode to avoid
unneeded function calls.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ia9d8191024a12d1374675b3d27197f10193f50bb
---
mm/kasan/hw_tags.c | 15 ---------------
mm/kasan/kasan.h | 28 ++++++++++++++++++++++++----

2 files changed, 24 insertions(+), 19 deletions(-)

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 1476ac07666e..0303e49904b4 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -30,27 +30,12 @@ void kasan_init_hw_tags(void)

pr_info("KernelAddressSanitizer initialized\n");
}

-void kasan_poison_memory(const void *address, size_t size, u8 value)
-{
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), value);
-}
-

void kasan_unpoison_memory(const void *address, size_t size)

{
hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

-bool check_invalid_free(void *addr)
-{
- u8 ptr_tag = get_tag(addr);
- u8 mem_tag = hw_get_mem_tag(addr);
-
- return (mem_tag == KASAN_TAG_INVALID) ||
- (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
-}
-

void kasan_set_free_info(struct kmem_cache *cache,
void *object, u8 tag)
{
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 7498839a15d3..ab7314418604 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -153,8 +153,6 @@ struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object);

-void kasan_poison_memory(const void *address, size_t size, u8 value);
-

#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

static inline const void *kasan_shadow_to_mem(const void *shadow_addr)

@@ -194,8 +192,6 @@ void print_tags(u8 addr_tag, const void *addr);

static inline void print_tags(u8 addr_tag, const void *addr) { }

#endif

-bool check_invalid_free(void *addr);
-

void *find_first_bad_addr(void *addr, size_t size);

const char *get_bug_type(struct kasan_access_info *info);

void metadata_fetch_row(char *buffer, void *row);

@@ -279,6 +275,30 @@ static inline u8 random_tag(void)
}
#endif

+#ifdef CONFIG_KASAN_HW_TAGS
+
+static inline void kasan_poison_memory(const void *address, size_t size, u8 value)
+{
+ hw_set_mem_tag_range(kasan_reset_tag(address),

+ round_up(size, KASAN_GRANULE_SIZE), value);
+}
+
+static inline bool check_invalid_free(void *addr)
+{
+ u8 ptr_tag = get_tag(addr);
+ u8 mem_tag = hw_get_mem_tag(addr);
+
+ return (mem_tag == KASAN_TAG_INVALID) ||
+ (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);

+}
+
+#else /* CONFIG_KASAN_HW_TAGS */
+

+void kasan_poison_memory(const void *address, size_t size, u8 value);
+bool check_invalid_free(void *addr);

+
+#endif /* CONFIG_KASAN_HW_TAGS */

[Andrey Konovalov]

Currently kasan_unpoison_memory() is used as both an external annotation
and as an internal memory poisoning helper. Rename external annotation to
kasan_unpoison_data() and inline the internal helper for hardware
tag-based mode to avoid undeeded function calls.

There's the external annotation kasan_unpoison_slab() that is currently
defined as static inline and uses kasan_unpoison_memory(). With this
change it's turned into a function call. Overall, this results in the
same number of calls for hardware tag-based mode as
kasan_unpoison_memory() is now inlined.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/Ia7c8b659f79209935cbaab3913bf7f082cc43a0e
---
include/linux/kasan.h | 16 ++++++----------
kernel/fork.c | 2 +-
mm/kasan/common.c | 10 ++++++++++
mm/kasan/hw_tags.c | 6 ------
mm/kasan/kasan.h | 7 +++++++
mm/slab_common.c | 2 +-

6 files changed, 25 insertions(+), 18 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 53c8e8b12fbc..f1a5042ae4fc 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -74,14 +74,15 @@ static inline void kasan_disable_current(void) {}

#ifdef CONFIG_KASAN

-void kasan_unpoison_memory(const void *address, size_t size);

-
void kasan_alloc_pages(struct page *page, unsigned int order);
void kasan_free_pages(struct page *page, unsigned int order);

void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
slab_flags_t *flags);

+void kasan_unpoison_data(const void *address, size_t size);
+void kasan_unpoison_slab(const void *ptr);
+
void kasan_poison_slab(struct page *page);

void kasan_unpoison_object_data(struct kmem_cache *cache, void *object);

void kasan_poison_object_data(struct kmem_cache *cache, void *object);

@@ -106,11 +107,6 @@ struct kasan_cache {

int free_meta_offset;
};

-size_t __ksize(const void *);

-static inline void kasan_unpoison_slab(const void *ptr)

-{
- kasan_unpoison_memory(ptr, __ksize(ptr));
-}
size_t kasan_metadata_size(struct kmem_cache *cache);

bool kasan_save_enable_multi_shot(void);

@@ -118,8 +114,6 @@ void kasan_restore_multi_shot(bool enabled);

#else /* CONFIG_KASAN */

-static inline void kasan_unpoison_memory(const void *address, size_t size) {}

-
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

@@ -127,6 +121,9 @@ static inline void kasan_cache_create(struct kmem_cache *cache,

unsigned int *size,
slab_flags_t *flags) {}

+static inline void kasan_unpoison_data(const void *address, size_t size) { }
+static inline void kasan_unpoison_slab(const void *ptr) { }

+

static inline void kasan_poison_slab(struct page *page) {}
static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
void *object) {}

@@ -166,7 +163,6 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
return false;
}

-static inline void kasan_unpoison_slab(const void *ptr) { }

static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

#endif /* CONFIG_KASAN */

diff --git a/kernel/fork.c b/kernel/fork.c
index 1c905e4290ab..883898487b3f 100644

--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -226,7 +226,7 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
continue;

/* Mark stack accessible for KASAN. */
- kasan_unpoison_memory(s->addr, THREAD_SIZE);
+ kasan_unpoison_data(s->addr, THREAD_SIZE);

/* Clear stale pointers from reused stack. */
memset(s->addr, 0, THREAD_SIZE);

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index a266b90636a1..4598c1364f19 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -184,6 +184,16 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

+void kasan_unpoison_data(const void *address, size_t size)
+{
+ kasan_unpoison_memory(address, size);
+}
+

+void kasan_unpoison_slab(const void *ptr)
+{

+ kasan_unpoison_memory(ptr, __ksize(ptr));
+}
+

void kasan_poison_slab(struct page *page)
{
unsigned long i;
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 0303e49904b4..838b29e44e32 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -30,12 +30,6 @@ void kasan_init_hw_tags(void)

pr_info("KernelAddressSanitizer initialized\n");
}

-void kasan_unpoison_memory(const void *address, size_t size)
-{
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), get_tag(address));

-}
-
void kasan_set_free_info(struct kmem_cache *cache,
void *object, u8 tag)
{
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h

index ab7314418604..2d3c99125996 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -283,6 +283,12 @@ static inline void kasan_poison_memory(const void *address, size_t size, u8 valu
round_up(size, KASAN_GRANULE_SIZE), value);
}

+static inline void kasan_unpoison_memory(const void *address, size_t size)
+{
+ hw_set_mem_tag_range(kasan_reset_tag(address),

+ round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
+}
+

static inline bool check_invalid_free(void *addr)
{
u8 ptr_tag = get_tag(addr);
@@ -295,6 +301,7 @@ static inline bool check_invalid_free(void *addr)

#else /* CONFIG_KASAN_HW_TAGS */

void kasan_poison_memory(const void *address, size_t size, u8 value);
+void kasan_unpoison_memory(const void *address, size_t size);
bool check_invalid_free(void *addr);

#endif /* CONFIG_KASAN_HW_TAGS */

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 53d0f8bb57ea..f1b0c4a22f08 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c

@@ -1176,7 +1176,7 @@ size_t ksize(const void *objp)
* We assume that ksize callers could use whole allocated area,
* so we need to unpoison this area.
*/
- kasan_unpoison_memory(objp, size);
+ kasan_unpoison_data(objp, size);
return size;
}
EXPORT_SYMBOL(ksize);
--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Hardware tag-based KASAN mode is intended to eventually be used in

production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.

This change adds a few boot parameters for hardware tag-based KASAN that
allow to disable or otherwise control particular KASAN features.

The features that can be controlled are:

1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.

With this change a new boot parameter kasan.mode allows to choose one of

three main modes:

- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled

The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:

- kasan.stacktrace=off/on - enable alloc/free stack collection

(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)

If kasan.mode parameter is not provided, it defaults to full when
CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.

It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by

the Android GKI (Generic Kernel Image) initiative [1].

[1] https://source.android.com/devices/architecture/kernel/generic-kernel-image

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/If7d37003875b2ed3e0935702c8015c223d6416a4
---
mm/kasan/common.c | 22 +++++--

mm/kasan/hw_tags.c | 152 +++++++++++++++++++++++++++++++++++++++++++++

mm/kasan/kasan.h | 16 +++++
mm/kasan/report.c | 14 ++++-

4 files changed, 197 insertions(+), 7 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 4598c1364f19..efad5ed6a3bd 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c

@@ -129,6 +129,11 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
unsigned int redzone_size;
int redzone_adjust;

+ if (!kasan_stack_collection_enabled()) {
+ *flags |= SLAB_KASAN;
+ return;

+ }
+
/* Add alloc meta. */

cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);

@@ -165,6 +170,8 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,

size_t kasan_metadata_size(struct kmem_cache *cache)
{

+ if (!kasan_stack_collection_enabled())
+ return 0;
return (cache->kasan_info.alloc_meta_offset ?
sizeof(struct kasan_alloc_meta) : 0) +
(cache->kasan_info.free_meta_offset ?
@@ -267,11 +274,13 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;
+ if (kasan_stack_collection_enabled()) {

+ if (!(cache->flags & SLAB_KASAN))
+ return (void *)object;

- alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ }

if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));

@@ -308,6 +317,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);

kasan_poison_memory(object, rounded_up_size, KASAN_KMALLOC_FREE);

+ if (!kasan_stack_collection_enabled())
+ return false;
+
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
unlikely(!(cache->flags & SLAB_KASAN)))
return false;

@@ -355,7 +367,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);

- if (cache->flags & SLAB_KASAN)
+ if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))

set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 838b29e44e32..2f6f0261af8c 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c

@@ -8,6 +8,8 @@

#define pr_fmt(fmt) "kasan: " fmt

+#include <linux/init.h>
+#include <linux/jump_label.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
#include <linux/memory.h>

@@ -17,9 +19,104 @@

#include "kasan.h"

+enum kasan_arg_mode {
+ KASAN_ARG_MODE_DEFAULT,
+ KASAN_ARG_MODE_OFF,
+ KASAN_ARG_MODE_PROD,
+ KASAN_ARG_MODE_FULL,
+};
+

+enum kasan_arg_stacktrace {
+ KASAN_ARG_STACKTRACE_DEFAULT,
+ KASAN_ARG_STACKTRACE_OFF,
+ KASAN_ARG_STACKTRACE_ON,

+};
+
+enum kasan_arg_fault {
+ KASAN_ARG_FAULT_DEFAULT,
+ KASAN_ARG_FAULT_REPORT,
+ KASAN_ARG_FAULT_PANIC,
+};
+
+static enum kasan_arg_mode kasan_arg_mode __ro_after_init;

+static enum kasan_arg_stacktrace kasan_arg_stacktrace __ro_after_init;

+static enum kasan_arg_fault kasan_arg_fault __ro_after_init;
+
+/* Whether KASAN is enabled at all. */
+DEFINE_STATIC_KEY_FALSE_RO(kasan_flag_enabled);
+EXPORT_SYMBOL(kasan_flag_enabled);
+
+/* Whether to collect alloc/free stack traces. */

+DEFINE_STATIC_KEY_FALSE_RO(kasan_flag_stacktrace);

+
+/* Whether panic or disable tag checking on fault. */
+bool kasan_flag_panic __ro_after_init;
+
+/* kasan.mode=off/prod/full */
+static int __init early_kasan_mode(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))
+ kasan_arg_mode = KASAN_ARG_MODE_OFF;
+ else if (!strcmp(arg, "prod"))
+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ else if (!strcmp(arg, "full"))
+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else
+ return -EINVAL;

+
+ return 0;
+}

+early_param("kasan.mode", early_kasan_mode);
+
+/* kasan.stack=off/on */

+static int __init early_kasan_flag_stacktrace(char *arg)

+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))

+ kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_OFF;

+ else if (!strcmp(arg, "on"))

+ kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_ON;

+ else
+ return -EINVAL;

+
+ return 0;
+}

+early_param("kasan.stacktrace", early_kasan_flag_stacktrace);

+
+/* kasan.fault=report/panic */
+static int __init early_kasan_fault(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "report"))
+ kasan_arg_fault = KASAN_ARG_FAULT_REPORT;
+ else if (!strcmp(arg, "panic"))
+ kasan_arg_fault = KASAN_ARG_FAULT_PANIC;
+ else
+ return -EINVAL;

+
+ return 0;
+}

+early_param("kasan.fault", early_kasan_fault);
+

/* kasan_init_hw_tags_cpu() is called for each CPU. */
void kasan_init_hw_tags_cpu(void)
{
+ /*
+ * There's no need to check that the hardware is MTE-capable here,
+ * as this function is only called for MTE-capable hardware.
+ */
+
+ /* If KASAN is disabled, do nothing. */

+ if (kasan_arg_mode == KASAN_ARG_MODE_OFF)
+ return;
+

hw_init_tags(KASAN_TAG_MAX);
hw_enable_tagging();
}
@@ -27,6 +124,61 @@ void kasan_init_hw_tags_cpu(void)
/* kasan_init_hw_tags() is called once on boot CPU. */
void kasan_init_hw_tags(void)
{
+ /* If hardware doesn't support MTE, do nothing. */
+ if (!system_supports_mte())
+ return;
+
+ /* If KASAN is disabled, do nothing. */

+ if (kasan_arg_mode == KASAN_ARG_MODE_OFF)
+ return;
+

+ /* Choose KASAN mode if kasan boot parameter is not provided. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_DEFAULT) {
+ if (IS_ENABLED(CONFIG_DEBUG_KERNEL))
+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else
+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ }
+

+ /* Preset parameter values based on the mode. */
+ switch (kasan_arg_mode) {
+ case KASAN_ARG_MODE_OFF:
+ return;
+ case KASAN_ARG_MODE_PROD:
+ static_branch_enable(&kasan_flag_enabled);
+ break;
+ case KASAN_ARG_MODE_FULL:
+ static_branch_enable(&kasan_flag_enabled);

+ static_branch_enable(&kasan_flag_stacktrace);

+ break;
+ default:
+ break;
+ }
+
+ /* Now, optionally override the presets. */
+

+ switch (kasan_arg_stacktrace) {
+ case KASAN_ARG_STACKTRACE_OFF:
+ static_branch_disable(&kasan_flag_stacktrace);
+ break;
+ case KASAN_ARG_STACKTRACE_ON:
+ static_branch_enable(&kasan_flag_stacktrace);

+ break;
+ default:
+ break;
+ }
+
+ switch (kasan_arg_fault) {
+ case KASAN_ARG_FAULT_REPORT:
+ kasan_flag_panic = false;
+ break;
+ case KASAN_ARG_FAULT_PANIC:
+ kasan_flag_panic = true;
+ break;
+ default:
+ break;
+ }
+

pr_info("KernelAddressSanitizer initialized\n");
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 2d3c99125996..5eff3d9f624e 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h

@@ -5,6 +5,22 @@
#include <linux/kasan.h>
#include <linux/stackdepot.h>

+#ifdef CONFIG_KASAN_HW_TAGS
+#include <linux/jump_label.h>

+DECLARE_STATIC_KEY_FALSE(kasan_flag_stacktrace);

+static inline bool kasan_stack_collection_enabled(void)
+{

+ return static_branch_unlikely(&kasan_flag_stacktrace);

+}
+#else
+static inline bool kasan_stack_collection_enabled(void)

+{
+ return true;
+}
+#endif

+
+extern bool kasan_flag_panic __ro_after_init;
+
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
#else

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 25ca66c99e48..7d86af340148 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c

@@ -99,6 +99,10 @@ static void end_report(unsigned long *flags)
panic_on_warn = 0;
panic("panic_on_warn set ...\n");
}
+#ifdef CONFIG_KASAN_HW_TAGS
+ if (kasan_flag_panic)
+ panic("kasan.fault=panic set ...\n");
+#endif
kasan_enable_current();
}

@@ -161,8 +165,8 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,
(void *)(object_addr + cache->object_size));
}

-static void describe_object(struct kmem_cache *cache, void *object,
- const void *addr, u8 tag)
+static void describe_object_stacks(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
{

struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);

@@ -190,7 +194,13 @@ static void describe_object(struct kmem_cache *cache, void *object,
}
#endif
}
+}

+static void describe_object(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
+{
+ if (kasan_stack_collection_enabled())

+ describe_object_stacks(cache, object, addr, tag);
describe_object_addr(cache, object, addr);
}

--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Declare the kasan_enabled static key in include/linux/kasan.h and in
include/linux/mm.h and check it in all kasan annotations. This allows to
avoid any slowdown caused by function calls when kasan_enabled is
disabled.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/I2589451d3c96c97abbcbf714baabe6161c6f153e
---
include/linux/kasan.h | 220 ++++++++++++++++++++++++++++++++----------
include/linux/mm.h | 22 +++--
mm/kasan/common.c | 60 ++++++------

3 files changed, 216 insertions(+), 86 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index f1a5042ae4fc..779f8e703982 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h

@@ -2,6 +2,7 @@
#ifndef _LINUX_KASAN_H
#define _LINUX_KASAN_H

+#include <linux/jump_label.h>
#include <linux/types.h>

struct kmem_cache;

@@ -74,56 +75,179 @@ static inline void kasan_disable_current(void) {}

#ifdef CONFIG_KASAN

-void kasan_alloc_pages(struct page *page, unsigned int order);
-void kasan_free_pages(struct page *page, unsigned int order);
+struct kasan_cache {
+ int alloc_meta_offset;
+ int free_meta_offset;
+};

-void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags);
+#ifdef CONFIG_KASAN_HW_TAGS
+DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
+static inline kasan_enabled(void)
+{
+ return static_branch_likely(&kasan_flag_enabled);
+}
+#else
+static inline kasan_enabled(void)

+{
+ return true;
+}
+#endif

-void kasan_unpoison_data(const void *address, size_t size);
-void kasan_unpoison_slab(const void *ptr);

+void __kasan_alloc_pages(struct page *page, unsigned int order);
+static inline void kasan_alloc_pages(struct page *page, unsigned int order)
+{
+ if (kasan_enabled())

+ __kasan_alloc_pages(page, order);
+}

-void kasan_poison_slab(struct page *page);
-void kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
-void kasan_poison_object_data(struct kmem_cache *cache, void *object);
-void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
- const void *object);

+void __kasan_free_pages(struct page *page, unsigned int order);
+static inline void kasan_free_pages(struct page *page, unsigned int order)
+{
+ if (kasan_enabled())

+ __kasan_free_pages(page, order);
+}

-void * __must_check kasan_kmalloc_large(const void *ptr, size_t size,
- gfp_t flags);
-void kasan_kfree_large(void *ptr, unsigned long ip);
-void kasan_poison_kfree(void *ptr, unsigned long ip);
-void * __must_check kasan_kmalloc(struct kmem_cache *s, const void *object,
- size_t size, gfp_t flags);
-void * __must_check kasan_krealloc(const void *object, size_t new_size,
- gfp_t flags);

+void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags);
+static inline void kasan_cache_create(struct kmem_cache *cache,
+ unsigned int *size, slab_flags_t *flags)
+{
+ if (kasan_enabled())

+ __kasan_cache_create(cache, size, flags);
+}

-void * __must_check kasan_slab_alloc(struct kmem_cache *s, void *object,
- gfp_t flags);
-bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip);
+size_t __kasan_metadata_size(struct kmem_cache *cache);

+static inline size_t kasan_metadata_size(struct kmem_cache *cache)
+{
+ if (kasan_enabled())

+ return __kasan_metadata_size(cache);
+ return 0;
+}

-struct kasan_cache {
- int alloc_meta_offset;
- int free_meta_offset;
-};
+void __kasan_unpoison_data(const void *addr, size_t size);
+static inline void kasan_unpoison_data(const void *addr, size_t size)

+{
+ if (kasan_enabled())

+ __kasan_unpoison_data(addr, size);
+}
+
+void __kasan_unpoison_slab(const void *ptr);

+static inline void kasan_unpoison_slab(const void *ptr)

+{
+ if (kasan_enabled())
+ __kasan_unpoison_slab(ptr);
+}
+
+void __kasan_poison_slab(struct page *page);

+static inline void kasan_poison_slab(struct page *page)
+{
+ if (kasan_enabled())
+ return __kasan_poison_slab(page);
+}
+
+void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object);
+static inline void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
+{
+ if (kasan_enabled())

+ return __kasan_unpoison_object_data(cache, object);
+}
+

+void __kasan_poison_object_data(struct kmem_cache *cache, void *object);
+static inline void kasan_poison_object_data(struct kmem_cache *cache, void *object)
+{
+ if (kasan_enabled())

+ __kasan_poison_object_data(cache, object);
+}
+

+void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
+ const void *object);
+static inline void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
+ const void *object)
+{
+ if (kasan_enabled())

+ return __kasan_init_slab_obj(cache, object);
+ return (void *)object;
+}
+

+bool __kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip);
+static inline bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned long ip)
+{
+ if (kasan_enabled())

+ return __kasan_slab_free(s, object, ip);
+ return false;
+}
+
+void * __must_check __kasan_slab_alloc(struct kmem_cache *s,
+ void *object, gfp_t flags);

+static inline void * __must_check kasan_slab_alloc(struct kmem_cache *s,
+ void *object, gfp_t flags)
+{

+ if (kasan_enabled())
+ return __kasan_slab_alloc(s, object, flags);
+ return object;
+}

-size_t kasan_metadata_size(struct kmem_cache *cache);
+void * __must_check __kasan_kmalloc(struct kmem_cache *s, const void *object,
+ size_t size, gfp_t flags);
+static inline void * __must_check kasan_kmalloc(struct kmem_cache *s, const void *object,

+ size_t size, gfp_t flags)
+{
+ if (kasan_enabled())

+ return __kasan_kmalloc(s, object, size, flags);
+ return (void *)object;
+}
+
+void * __must_check __kasan_kmalloc_large(const void *ptr,
+ size_t size, gfp_t flags);
+static inline void * __must_check kasan_kmalloc_large(const void *ptr,

+ size_t size, gfp_t flags)
+{
+ if (kasan_enabled())

+ return __kasan_kmalloc_large(ptr, size, flags);
+ return (void *)ptr;
+}
+
+void * __must_check __kasan_krealloc(const void *object,
+ size_t new_size, gfp_t flags);

+static inline void * __must_check kasan_krealloc(const void *object,
+ size_t new_size, gfp_t flags)
+{
+ if (kasan_enabled())

+ return __kasan_krealloc(object, new_size, flags);
+ return (void *)object;
+}
+

+void __kasan_poison_kfree(void *ptr, unsigned long ip);
+static inline void kasan_poison_kfree(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())

+ __kasan_poison_kfree(ptr, ip);
+}
+

+void __kasan_kfree_large(void *ptr, unsigned long ip);
+static inline void kasan_kfree_large(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())

+ __kasan_kfree_large(ptr, ip);
+}

bool kasan_save_enable_multi_shot(void);

void kasan_restore_multi_shot(bool enabled);

#else /* CONFIG_KASAN */

+static inline kasan_enabled(void)
+{
+ return false;
+}

static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

-
static inline void kasan_cache_create(struct kmem_cache *cache,
unsigned int *size,
slab_flags_t *flags) {}
-
-static inline void kasan_unpoison_data(const void *address, size_t size) { }

-static inline void kasan_unpoison_slab(const void *ptr) { }
-

+static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

+static inline void kasan_unpoison_data(const void *address, size_t size) {}
+static inline void kasan_unpoison_slab(const void *ptr) {}

static inline void kasan_poison_slab(struct page *page) {}
static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
void *object) {}

@@ -134,36 +258,32 @@ static inline void *kasan_init_slab_obj(struct kmem_cache *cache,

{
return (void *)object;
}
-
-static inline void *kasan_kmalloc_large(void *ptr, size_t size, gfp_t flags)
+static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
+ unsigned long ip)
{
- return ptr;
+ return false;
+}
+static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
+ gfp_t flags)
+{
+ return object;
}
-static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}
-static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
static inline void *kasan_kmalloc(struct kmem_cache *s, const void *object,
size_t size, gfp_t flags)
{
return (void *)object;
}
+static inline void *kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags)
+{
+ return (void *)ptr;
+}

static inline void *kasan_krealloc(const void *object, size_t new_size,

gfp_t flags)
{
return (void *)object;
}
-
-static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,
- gfp_t flags)
-{
- return object;
-}

-static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
- unsigned long ip)
-{

- return false;
-}
-

-static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }
+static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}
+static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}

#endif /* CONFIG_KASAN */

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 0793d03a4183..8d84a6b2fa3c 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -31,6 +31,7 @@
#include <linux/sizes.h>
#include <linux/sched.h>
#include <linux/pgtable.h>
+#include <linux/kasan.h>

struct mempolicy;
struct anon_vma;
@@ -1414,22 +1415,30 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
#endif /* CONFIG_NUMA_BALANCING */

#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+
static inline u8 page_kasan_tag(const struct page *page)
{
- return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;

+ if (kasan_enabled())

+ return (page->flags >> KASAN_TAG_PGSHIFT) & KASAN_TAG_MASK;
+ return 0xff;
}

static inline void page_kasan_tag_set(struct page *page, u8 tag)
{
- page->flags &= ~(KASAN_TAG_MASK << KASAN_TAG_PGSHIFT);
- page->flags |= (tag & KASAN_TAG_MASK) << KASAN_TAG_PGSHIFT;
+ if (kasan_enabled()) {
+ page->flags &= ~(KASAN_TAG_MASK << KASAN_TAG_PGSHIFT);
+ page->flags |= (tag & KASAN_TAG_MASK) << KASAN_TAG_PGSHIFT;
+ }
}

static inline void page_kasan_tag_reset(struct page *page)
{
- page_kasan_tag_set(page, 0xff);
+ if (kasan_enabled())
+ page_kasan_tag_set(page, 0xff);
}
-#else
+

+#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+

static inline u8 page_kasan_tag(const struct page *page)
{
return 0xff;
@@ -1437,7 +1446,8 @@ static inline u8 page_kasan_tag(const struct page *page)

static inline void page_kasan_tag_set(struct page *page, u8 tag) { }
static inline void page_kasan_tag_reset(struct page *page) { }
-#endif
+
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */

static inline struct zone *page_zone(const struct page *page)
{

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index efad5ed6a3bd..385863eaec2c 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c

@@ -81,7 +81,7 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)
}
#endif /* CONFIG_KASAN_STACK */

-void kasan_alloc_pages(struct page *page, unsigned int order)
+void __kasan_alloc_pages(struct page *page, unsigned int order)
{
u8 tag;
unsigned long i;
@@ -95,7 +95,7 @@ void kasan_alloc_pages(struct page *page, unsigned int order)
kasan_unpoison_memory(page_address(page), PAGE_SIZE << order);
}

-void kasan_free_pages(struct page *page, unsigned int order)
+void __kasan_free_pages(struct page *page, unsigned int order)
{
if (likely(!PageHighMem(page)))
kasan_poison_memory(page_address(page),
@@ -122,8 +122,8 @@ static inline unsigned int optimal_redzone(unsigned int object_size)
object_size <= (1 << 16) - 1024 ? 1024 : 2048;
}

-void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags)

+void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags)
{

unsigned int orig_size = *size;

unsigned int redzone_size;
@@ -168,7 +168,7 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
*flags |= SLAB_KASAN;
}

-size_t kasan_metadata_size(struct kmem_cache *cache)
+size_t __kasan_metadata_size(struct kmem_cache *cache)
{
if (!kasan_stack_collection_enabled())
return 0;

@@ -191,17 +191,17 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

-void kasan_unpoison_data(const void *address, size_t size)
+void __kasan_unpoison_data(const void *addr, size_t size)
{
- kasan_unpoison_memory(address, size);
+ kasan_unpoison_memory(addr, size);
}

-void kasan_unpoison_slab(const void *ptr)
+void __kasan_unpoison_slab(const void *ptr)
{
kasan_unpoison_memory(ptr, __ksize(ptr));
}

-void kasan_poison_slab(struct page *page)

+void __kasan_poison_slab(struct page *page)
{
unsigned long i;

@@ -211,12 +211,12 @@ void kasan_poison_slab(struct page *page)
KASAN_KMALLOC_REDZONE);
}

-void kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
+void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object)
{
kasan_unpoison_memory(object, cache->object_size);
}

-void kasan_poison_object_data(struct kmem_cache *cache, void *object)
+void __kasan_poison_object_data(struct kmem_cache *cache, void *object)
{
kasan_poison_memory(object,
round_up(cache->object_size, KASAN_GRANULE_SIZE),

@@ -269,7 +269,7 @@ static u8 assign_tag(struct kmem_cache *cache, const void *object,
#endif
}

-void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
+void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
const void *object)

{
struct kasan_alloc_meta *alloc_meta;
@@ -288,7 +288,7 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
return (void *)object;
}

-static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
+static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
unsigned long ip, bool quarantine)
{
u8 tag;

@@ -331,9 +331,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
return IS_ENABLED(CONFIG_KASAN_GENERIC);
}

-bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
+bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
{
- return __kasan_slab_free(cache, object, ip, true);
+ return ____kasan_slab_free(cache, object, ip, true);

}

static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)

@@ -341,7 +341,7 @@ static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
}

-static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
+static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,

size_t size, gfp_t flags, bool keep_tag)
{

- __kasan_slab_free(page->slab_cache, ptr, ip, false);

+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);
}
}

-void kasan_kfree_large(void *ptr, unsigned long ip)
+void __kasan_kfree_large(void *ptr, unsigned long ip)

{
if (ptr != page_address(virt_to_head_page(ptr)))
kasan_report_invalid_free(ptr, ip);

--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

kasan_poison_kfree() is currently only called for mempool allocations
that are backed by either kmem_cache_alloc() or kmalloc(). Therefore, the
page passed to kasan_poison_kfree() is always PageSlab() and there's no
need to do the check. Remove it.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/If31f88726745da8744c6bea96fb32584e6c2778c
---
mm/kasan/common.c | 11 +----------

1 file changed, 1 insertion(+), 10 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 385863eaec2c..819403548f2e 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -432,16 +432,7 @@ void __kasan_poison_kfree(void *ptr, unsigned long ip)
struct page *page;

page = virt_to_head_page(ptr);
-
- if (unlikely(!PageSlab(page))) {
- if (ptr != page_address(page)) {
- kasan_report_invalid_free(ptr, ip);
- return;
- }
- kasan_poison_memory(ptr, page_size(page), KASAN_FREE_PAGE);
- } else {

- ____kasan_slab_free(page->slab_cache, ptr, ip, false);

- }
+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);
}

void __kasan_kfree_large(void *ptr, unsigned long ip)

--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Rename kasan_poison_kfree() to kasan_slab_free_mempool() as it better
reflects what this annotation does.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I5026f87364e556b506ef1baee725144bb04b8810
---
include/linux/kasan.h | 16 ++++++++--------
mm/kasan/common.c | 16 ++++++++--------
mm/mempool.c | 2 +-

3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 779f8e703982..534ab3e2935a 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -177,6 +177,13 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object, unsigned
return false;
}

+void __kasan_slab_free_mempool(void *ptr, unsigned long ip);
+static inline void kasan_slab_free_mempool(void *ptr, unsigned long ip)
+{
+ if (kasan_enabled())

+ __kasan_slab_free_mempool(ptr, ip);
+}
+

void * __must_check __kasan_slab_alloc(struct kmem_cache *s,

void *object, gfp_t flags);

static inline void * __must_check kasan_slab_alloc(struct kmem_cache *s,

@@ -217,13 +224,6 @@ static inline void * __must_check kasan_krealloc(const void *object,
return (void *)object;
}

-void __kasan_poison_kfree(void *ptr, unsigned long ip);

-static inline void kasan_poison_kfree(void *ptr, unsigned long ip)

-{
- if (kasan_enabled())
- __kasan_poison_kfree(ptr, ip);
-}
-

void __kasan_kfree_large(void *ptr, unsigned long ip);

static inline void kasan_kfree_large(void *ptr, unsigned long ip)

{
@@ -263,6 +263,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
{
return false;
}
+static inline void kasan_slab_free_mempool(void *ptr, unsigned long ip) {}

static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object,

gfp_t flags)
{
@@ -282,7 +283,6 @@ static inline void *kasan_krealloc(const void *object, size_t new_size,
{
return (void *)object;

}
-static inline void kasan_poison_kfree(void *ptr, unsigned long ip) {}

static inline void kasan_kfree_large(void *ptr, unsigned long ip) {}

#endif /* CONFIG_KASAN */

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 819403548f2e..60793f8695a8 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -336,6 +336,14 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)

return ____kasan_slab_free(cache, object, ip, true);
}

+void __kasan_slab_free_mempool(void *ptr, unsigned long ip)
+{

+ struct page *page;
+
+ page = virt_to_head_page(ptr);

+ ____kasan_slab_free(page->slab_cache, ptr, ip, false);

+}
+

static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{
kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

@@ -427,14 +435,6 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag
flags, true);
}

-void __kasan_poison_kfree(void *ptr, unsigned long ip)
-{
- struct page *page;
-
- page = virt_to_head_page(ptr);

- ____kasan_slab_free(page->slab_cache, ptr, ip, false);
-}

-

void __kasan_kfree_large(void *ptr, unsigned long ip)

{
if (ptr != page_address(virt_to_head_page(ptr)))
diff --git a/mm/mempool.c b/mm/mempool.c
index f473cdddaff0..b1f39fa75ade 100644
--- a/mm/mempool.c
+++ b/mm/mempool.c
@@ -104,7 +104,7 @@ static inline void poison_element(mempool_t *pool, void *element)
static __always_inline void kasan_poison_element(mempool_t *pool, void *element)
{
if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
- kasan_poison_kfree(element, _RET_IP_);
+ kasan_slab_free_mempool(element, _RET_IP_);
else if (pool->alloc == mempool_alloc_pages)
kasan_free_pages(element, (unsigned long)pool->pool_data);
}
--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

For hardware tag-based mode kasan_poison_memory() already rounds up the
size. Do the same for software modes and remove round_up() from the common
code.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/Ib397128fac6eba874008662b4964d65352db4aa4
---
mm/kasan/common.c | 8 ++------
mm/kasan/shadow.c | 1 +
2 files changed, 3 insertions(+), 6 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 60793f8695a8..69ab880abacc 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -218,9 +218,7 @@ void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object)

void __kasan_poison_object_data(struct kmem_cache *cache, void *object)

{
- kasan_poison_memory(object,
- round_up(cache->object_size, KASAN_GRANULE_SIZE),
- KASAN_KMALLOC_REDZONE);
+ kasan_poison_memory(object, cache->object_size, KASAN_KMALLOC_REDZONE);
}

/*
@@ -293,7 +291,6 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
{
u8 tag;
void *tagged_object;
- unsigned long rounded_up_size;

tag = get_tag(object);
tagged_object = object;

@@ -314,8 +311,7 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
return true;
}

- rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
- kasan_poison_memory(object, rounded_up_size, KASAN_KMALLOC_FREE);
+ kasan_poison_memory(object, cache->object_size, KASAN_KMALLOC_FREE);

if (!kasan_stack_collection_enabled())
return false;

diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 8e4fa9157a0b..3f64c9ecbcc0 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -82,6 +82,7 @@ void kasan_poison_memory(const void *address, size_t size, u8 value)

* addresses to this function.
*/

address = kasan_reset_tag(address);
+ size = round_up(size, KASAN_GRANULE_SIZE);

shadow_start = kasan_mem_to_shadow(address);
shadow_end = kasan_mem_to_shadow(address + size);

--
2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

set_tag() already ignores the tag for the generic mode, so just call it
as is. Add a check for the generic mode to assign_tag(), and simplify its
call in ____kasan_kmalloc().

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I18905ca78fb4a3d60e1a34a4ca00247272480438
---
mm/kasan/common.c | 11 ++++++-----

1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 69ab880abacc..40ff3ce07a76 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -238,6 +238,9 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object)

static u8 assign_tag(struct kmem_cache *cache, const void *object,

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Currently it says that the memory gets poisoned by page_alloc code.
Clarify this by mentioning the specific callback that poisons the
memory.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Link: https://linux-review.googlesource.com/id/I1334dffb69b87d7986fab88a1a039cc3ea764725
---
mm/kasan/common.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 40ff3ce07a76..4360292ad7f3 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -436,5 +436,5 @@ void __kasan_kfree_large(void *ptr, unsigned long ip)
{
if (ptr != page_address(virt_to_head_page(ptr)))

kasan_report_invalid_free(ptr, ip);
- /* The object will be poisoned by page_alloc. */
+ /* The object will be poisoned by kasan_free_pages(). */
}
--

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

KASAN marks caches that are sanitized with the SLAB_KASAN cache flag.
Currently if the metadata that is appended after the object (stores e.g.
stack trace ids) doesn't fit into KMALLOC_MAX_SIZE (can only happen with
SLAB, see the comment in the patch), KASAN turns off sanitization
completely.

With this change sanitization of the object data is always enabled.
However the metadata is only stored when it fits. Instead of checking for
SLAB_KASAN flag accross the code to find out whether the metadata is
there, use cache->kasan_info.alloc/free_meta_offset. As 0 can be a valid
value for free_meta_offset, introduce KASAN_NO_FREE_META as an indicator
that the free metadata is missing.

Along the way rework __kasan_cache_create() and add claryfying comments.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/Icd947e2bea054cb5cfbdc6cf6652227d97032dcb
---
mm/kasan/common.c | 112 +++++++++++++++++++++++++-------------
mm/kasan/generic.c | 15 ++---
mm/kasan/hw_tags.c | 6 +-
mm/kasan/kasan.h | 13 ++++-
mm/kasan/quarantine.c | 8 +++
mm/kasan/report.c | 43 ++++++++-------
mm/kasan/report_sw_tags.c | 7 ++-
mm/kasan/sw_tags.c | 4 ++
8 files changed, 138 insertions(+), 70 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 4360292ad7f3..940b42231069 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -109,9 +109,6 @@ void __kasan_free_pages(struct page *page, unsigned int order)
*/

static inline unsigned int optimal_redzone(unsigned int object_size)

{
- if (!IS_ENABLED(CONFIG_KASAN_GENERIC))
- return 0;
-
return
object_size <= 64 - 16 ? 16 :
object_size <= 128 - 32 ? 32 :

@@ -125,47 +122,79 @@ static inline unsigned int optimal_redzone(unsigned int object_size)

void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,

slab_flags_t *flags)
{
- unsigned int orig_size = *size;
+ unsigned int ok_size;
unsigned int redzone_size;
- int redzone_adjust;
+ unsigned int optimal_size;
+
+ /*
+ * SLAB_KASAN is used to mark caches as ones that are sanitized by
+ * KASAN. Currently this is used in two places:
+ * 1. In slab_ksize() when calculating the size of the accessible
+ * memory within the object.
+ * 2. In slab_common.c to prevent merging of sanitized caches.
+ */
+ *flags |= SLAB_KASAN;

- if (!kasan_stack_collection_enabled()) {
- *flags |= SLAB_KASAN;
+ if (!kasan_stack_collection_enabled())
return;
- }

- /* Add alloc meta. */
+ ok_size = *size;
+
+ /* Add alloc meta into redzone. */

cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);

@@ -181,15 +210,21 @@ size_t __kasan_metadata_size(struct kmem_cache *cache)
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{

+ if (!cache->kasan_info.alloc_meta_offset)
+ return NULL;

return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

+#ifdef CONFIG_KASAN_GENERIC

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);

+ if (cache->kasan_info.free_meta_offset == KASAN_NO_FREE_META)
+ return NULL;

return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

+#endif

void __kasan_unpoison_data(const void *addr, size_t size)

{
@@ -276,11 +311,9 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

if (kasan_stack_collection_enabled()) {

- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;

-
alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ if (alloc_meta)
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));
}

/* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */

@@ -319,8 +352,7 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object,
if (!kasan_stack_collection_enabled())
return false;

- if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
- unlikely(!(cache->flags & SLAB_KASAN)))
+ if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine))
return false;

kasan_set_free_info(cache, object, tag);

@@ -345,7 +377,11 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip)

static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{

- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

+ struct kasan_alloc_meta *alloc_meta;
+

+ alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (alloc_meta)
+ kasan_set_track(&alloc_meta->alloc_track, flags);
}

static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,

@@ -372,7 +408,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,

kasan_poison_memory((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);

- if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))
+ if (kasan_stack_collection_enabled())

set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);

diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index d259e4c3aefd..97e39516f8fe 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -338,10 +338,10 @@ void kasan_record_aux_stack(void *addr)

cache = page->slab_cache;
object = nearest_obj(cache, page, addr);

alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (!alloc_meta)
+ return;

- /*
- * record the last two call_rcu() call stacks.
- */

+ /* Record the last two call_rcu() call stacks. */

alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];

alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
}

@@ -352,11 +352,11 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_free_meta *free_meta;

free_meta = kasan_get_free_meta(cache, object);
- kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
+ if (!free_meta)
+ return;

- /*
- * the object was freed and has free track set
- */
+ kasan_set_track(&free_meta->free_track, GFP_NOWAIT);
+ /* The object was freed and has free track set. */
*(u8 *)kasan_mem_to_shadow(object) = KASAN_KMALLOC_FREETRACK;
}

@@ -365,5 +365,6 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;

+ /* Free meta must be present with KASAN_KMALLOC_FREETRACK. */
return &kasan_get_free_meta(cache, object)->free_track;
}

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 2f6f0261af8c..c3d2a21d925d 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -188,7 +188,8 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

alloc_meta = kasan_get_alloc_meta(cache, object);
- kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
+ if (alloc_meta)
+ kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);

}

struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

@@ -197,5 +198,8 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;

alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (!alloc_meta)
+ return NULL;
+
return &alloc_meta->free_track[0];
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 5eff3d9f624e..88892c05eb7d 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h

@@ -154,20 +154,31 @@ struct kasan_alloc_meta {
struct qlist_node {
struct qlist_node *next;
};
+
+/*
+ * Generic mode either stores free meta in the object itself or in the redzone
+ * after the object. In the former case free meta offset is 0, in the latter
+ * case it has some sane value smaller than INT_MAX. Use INT_MAX as free meta
+ * offset when free meta isn't present.
+ */
+#define KASAN_NO_FREE_META (INT_MAX)
+
struct kasan_free_meta {
+#ifdef CONFIG_KASAN_GENERIC
/* This field is used while the object is in the quarantine.
* Otherwise it might be used for the allocator freelist.
*/
struct qlist_node quarantine_link;
-#ifdef CONFIG_KASAN_GENERIC
struct kasan_track free_track;
#endif
};

struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object);
+#ifdef CONFIG_KASAN_GENERIC

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object);
+#endif

#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index 0da3d37e1589..23f6bfb1e73f 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c

@@ -135,7 +135,12 @@ static void qlink_free(struct qlist_node *qlink, struct kmem_cache *cache)
if (IS_ENABLED(CONFIG_SLAB))
local_irq_save(flags);

+ /*
+ * As the object now gets freed from the quaratine, assume that its
+ * free track is now longer valid.
+ */
*(u8 *)kasan_mem_to_shadow(object) = KASAN_KMALLOC_FREE;
+
___cache_free(cache, object, _THIS_IP_);

if (IS_ENABLED(CONFIG_SLAB))

@@ -168,6 +173,9 @@ void quarantine_put(struct kmem_cache *cache, void *object)

struct qlist_head temp = QLIST_INIT;

struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);

+ if (!meta)
+ return;
+

/*
* Note: irq must be disabled until after we move the batch to the

* global quarantine. Otherwise quarantine_remove_cache() can miss

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 7d86af340148..6a95ad2dee91 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -168,32 +168,35 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,

static void describe_object_stacks(struct kmem_cache *cache, void *object,

const void *addr, u8 tag)

{
- struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);
-
- if (cache->flags & SLAB_KASAN) {
- struct kasan_track *free_track;
+ struct kasan_alloc_meta *alloc_meta;
+ struct kasan_track *free_track;

+ alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (alloc_meta) {

print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");

+ if (alloc_meta->aux_stack[0]) {

+ pr_err("Last call_rcu():\n");

+ print_stack(alloc_meta->aux_stack[0]);

+ pr_err("\n");

}
+ if (alloc_meta->aux_stack[1]) {

+ pr_err("Second to last call_rcu():\n");

+ print_stack(alloc_meta->aux_stack[1]);

+ pr_err("\n");
+ }
+#endif
}

static void describe_object(struct kmem_cache *cache, void *object,

diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index 7604b46239d4..11dc8739e500 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -48,9 +48,10 @@ const char *get_bug_type(struct kasan_access_info *info)

object = nearest_obj(cache, page, (void *)addr);

alloc_meta = kasan_get_alloc_meta(cache, object);

- for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
- if (alloc_meta->free_pointer_tag[i] == tag)
- return "use-after-free";
+ if (alloc_meta)
+ for (i = 0; i < KASAN_NR_FREE_STACKS; i++)
+ if (alloc_meta->free_pointer_tag[i] == tag)
+ return "use-after-free";
return "out-of-bounds";
}

diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index d1af6f6c6d12..be10d16bd129 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -170,6 +170,8 @@ void kasan_set_free_info(struct kmem_cache *cache,
u8 idx = 0;

alloc_meta = kasan_get_alloc_meta(cache, object);
+ if (!alloc_meta)
+ return;

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;

@@ -187,6 +189,8 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
int i = 0;

alloc_meta = kasan_get_alloc_meta(cache, object);

+ if (!alloc_meta)
+ return NULL;

[Andrey Konovalov]

The reason cache merging is disabled with KASAN is because KASAN puts its
metadata right after the allocated object. When the merged caches have
slightly different sizes, the metadata ends up in different places, which
KASAN doesn't support.

It might be possible to adjust the metadata allocation algorithm and make
it friendly to the cache merging code. Instead this change takes a simpler
approach and allows merging caches when no metadata is present. Which is
the case for hardware tag-based KASAN with kasan.mode=prod.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Link: https://linux-review.googlesource.com/id/Ia114847dfb2244f297d2cb82d592bf6a07455dba
---
include/linux/kasan.h | 26 ++++++++++++++++++++++++--
mm/kasan/common.c | 11 +++++++++++
mm/slab_common.c | 11 ++++++++---

3 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 534ab3e2935a..c754eca356f7 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -81,17 +81,35 @@ struct kasan_cache {

};

#ifdef CONFIG_KASAN_HW_TAGS
+
DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
+
static inline kasan_enabled(void)
{
return static_branch_likely(&kasan_flag_enabled);
}
-#else
+
+slab_flags_t __kasan_never_merge(slab_flags_t flags);
+static inline slab_flags_t kasan_never_merge(slab_flags_t flags)

+{
+ if (kasan_enabled())

+ return __kasan_never_merge(flags);
+ return flags;
+}
+

+#else /* CONFIG_KASAN_HW_TAGS */
+

static inline kasan_enabled(void)
{
return true;
}
-#endif
+
+static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
+{
+ return flags;
+}

+
+#endif /* CONFIG_KASAN_HW_TAGS */

void __kasan_alloc_pages(struct page *page, unsigned int order);
static inline void kasan_alloc_pages(struct page *page, unsigned int order)
@@ -240,6 +258,10 @@ static inline kasan_enabled(void)

{
return false;
}
+static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
+{
+ return flags;

+}
static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

static inline void kasan_cache_create(struct kmem_cache *cache,

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 940b42231069..25b18c145b06 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -81,6 +81,17 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)
}
#endif /* CONFIG_KASAN_STACK */

+/*
+ * Only allow cache merging when stack collection is disabled and no metadata
+ * is present.
+ */
+slab_flags_t __kasan_never_merge(slab_flags_t flags)

+{
+ if (kasan_stack_collection_enabled())

+ return flags;
+ return flags & ~SLAB_KASAN;
+}
+

void __kasan_alloc_pages(struct page *page, unsigned int order)
{
u8 tag;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index f1b0c4a22f08..3042ee8ea9ce 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

This change updates KASAN documentation to reflect the addition of boot
parameters and also reworks and clarifies some of the existing sections,
in particular: defines what a memory granule is, mentions quarantine,
makes Kunit section more readable.

Signed-off-by: Andrey Konovalov <andre...@google.com>
---
Documentation/dev-tools/kasan.rst | 180 +++++++++++++++++++-----------
1 file changed, 113 insertions(+), 67 deletions(-)

diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst

index 422f8ee1bb17..f2da2b09e5c7 100644

+The ``kasam.mode`` boot parameter allows to choose one of three main modes:
+
+- ``kasan.mode=off`` - KASAN is disabled, no tag checks are performed
+- ``kasan.mode=prod`` - only essential production features are enabled
+- ``kasan.mode=full`` - all KASAN features are enabled
+
+The chosen mode provides default control values for the features mentioned
+above. However it's also possible to override the default values by providing:
+
+- ``kasan.stacktrace=off`` or ``=on`` - enable alloc/free stack collection

2.29.2.222.g5d2a92d10f8-goog

[Andrey Konovalov]

Can I get everyone's Ack on these boot parameters?

[Marco Elver]

These are missing types, as noticed by Andrew:
https://marc.info/?l=linux-mm-commits&m=160505097028591&w=2

> +{
> + return static_branch_likely(&kasan_flag_enabled);
> +}

I think this should be __always_inline, as not inlining is a bug.

Also, I believe that all the below wrappers need to become
__always_inline, as we really cannot tolerate them not being inlined.

> +#else
> +static inline kasan_enabled(void)
> +{
> + return true;
> +}

(Some of these could be on 1 line, but I don't mind.)

[Marco Elver]

Rather than changing this to require using slab_never_merge() which
removes SLAB_KASAN, could we not just have a function
kasan_never_merge() that returns KASAN-specific flags that should never
result in merging -- because as-is now, making kasan_never_merge()
remove the SLAB_KASAN flag seems the wrong way around.

Could we not just do this:

#define SLAB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
SLAB_TRACE | SLAB_TYPESAFE_BY_RCU | SLAB_NOLEAKTRACE | \

SLAB_FAILSLAB | kasan_never_merge())

??

Of course that might be problematic if this always needs to be a
compile-time constant, but currently that's not a requirement.

> +/* KASAN allows merging in some configurations and will remove SLAB_KASAN. */
> +#define slab_never_merge() (kasan_never_merge(SLAB_NEVER_MERGE))

Braces unnecessary.

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> This change updates KASAN documentation to reflect the addition of boot
> parameters and also reworks and clarifies some of the existing sections,
> in particular: defines what a memory granule is, mentions quarantine,
> makes Kunit section more readable.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> ---
> Documentation/dev-tools/kasan.rst | 180 +++++++++++++++++++-----------
> 1 file changed, 113 insertions(+), 67 deletions(-)
>
> diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
> index 422f8ee1bb17..f2da2b09e5c7 100644
> --- a/Documentation/dev-tools/kasan.rst
> +++ b/Documentation/dev-tools/kasan.rst
> @@ -6,6 +6,7 @@ Overview
>
> KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to

s/memory error/memory safety error/

to be precise and consistent with various other docs and literature we
have, if you deem it appropriate to change in this patch.

I think ReST automatically creates a link if you write it as

... (see `Implementation details`_ section).

>
> +Boot parameters
> +~~~~~~~~~~~~~~~
> +
> +Hardware tag-based KASAN mode (see the section about different mode below) is
> +intended for use in production as a security mitigation. Therefore it supports
> +boot parameters that allow to disable KASAN competely or otherwise control
> +particular KASAN features.
> +
> +The things that can be controlled are:
> +
> +1. Whether KASAN is enabled at all.
> +2. Whether KASAN collects and saves alloc/free stacks.
> +3. Whether KASAN panics on a detected bug or not.
> +
> +The ``kasam.mode`` boot parameter allows to choose one of three main modes:

s/kasam/kasan/

> +- ``kasan.mode=off`` - KASAN is disabled, no tag checks are performed
> +- ``kasan.mode=prod`` - only essential production features are enabled
> +- ``kasan.mode=full`` - all KASAN features are enabled
> +
> +The chosen mode provides default control values for the features mentioned
> +above. However it's also possible to override the default values by providing:
> +
> +- ``kasan.stacktrace=off`` or ``=on`` - enable alloc/free stack collection
> + (default: ``on`` for ``mode=full``,
> + otherwise ``off``)
> +- ``kasan.fault=report`` or ``=panic`` - only print KASAN report or also panic
> + (default: ``report``)

This is indented with tabs instead of spaces.

> +
> +If ``kasan.mode parameter`` is not provided, it defaults to ``full`` when

s/``kasan.mode parameter``/``kasan.mode`` parameter/ ?

s/in to/into/

> memory.
>
> -To avoid the difficulties around swapping mappings around, we expect
> +To avoid the difficulties around swapping mappings around, KASAN expects
> that the part of the shadow region that covers the vmalloc space will
> not be covered by the early shadow page, but will be left
> unmapped. This will require changes in arch-specific code.
> @@ -321,24 +368,31 @@ architectures that do not have a fixed module region.
> CONFIG_KASAN_KUNIT_TEST & CONFIG_TEST_KASAN_MODULE
> --------------------------------------------------
>
> -``CONFIG_KASAN_KUNIT_TEST`` utilizes the KUnit Test Framework for testing.
> -This means each test focuses on a small unit of functionality and
> -there are a few ways these tests can be run.
> +KASAN tests consist on two parts:
> +
> +1. Tests that are integrated with the KUnit Test Framework. Enabled with
> +``CONFIG_KASAN_KUNIT_TEST``. These tests can be run and partially verified
> +automatically in a few different ways, see the instructions below.
>
> -Each test will print the KASAN report if an error is detected and then
> -print the number of the test and the status of the test:
> +2. Tests that are currently incompatible with Kunit. Enabled with

s/Kunit/KUnit/

> +``CONFIG_TEST_KASAN_MODULE`` and can only be run as a module. These tests can
> +only be verified manually, by loading the kernel module and inspecting the
> +kernel log for KASAN reports.
>
> -pass::
> +Each KUNIT-compatible KASAN test prints a KASAN report if an error is detected.

s/KUNIT/KUnit/ like elsewhere.

s/Kunit/KUnit/

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Move get_free_info() call into quarantine_put() to simplify the call site.
>
> No functional changes.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/Iab0f04e7ebf8d83247024b7190c67c3c34c7940f
> ---
> mm/kasan/common.c | 2 +-
> mm/kasan/kasan.h | 5 ++---
> mm/kasan/quarantine.c | 3 ++-
> 3 files changed, 5 insertions(+), 5 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Rename get_alloc_info() and get_free_info() to kasan_get_alloc_meta()
> and kasan_get_free_meta() to better reflect what those do and avoid
> confusion with kasan_set_free_info().
>
> No functional changes.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/Ib6e4ba61c8b12112b403d3479a9799ac8fff8de1
> ---
> mm/kasan/common.c | 16 ++++++++--------
> mm/kasan/generic.c | 12 ++++++------
> mm/kasan/hw_tags.c | 4 ++--
> mm/kasan/kasan.h | 8 ++++----
> mm/kasan/quarantine.c | 4 ++--
> mm/kasan/report.c | 12 ++++++------
> mm/kasan/report_sw_tags.c | 2 +-
> mm/kasan/sw_tags.c | 4 ++--
> 8 files changed, 31 insertions(+), 31 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Add set_alloc_info() helper and move kasan_set_track() into it. This will
> simplify the code for one of the upcoming changes.
>
> No functional changes.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/I0316193cbb4ecc9b87b7c2eee0dd79f8ec908c1a
> ---

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> There's a config option CONFIG_KASAN_STACK that has to be enabled for
> KASAN to use stack instrumentation and perform validity checks for
> stack variables.
>
> There's no need to unpoison stack when CONFIG_KASAN_STACK is not enabled.
> Only call kasan_unpoison_task_stack[_below]() when CONFIG_KASAN_STACK is
> enabled.
>
> Note, that CONFIG_KASAN_STACK is an option that is currently always
> defined when CONFIG_KASAN is enabled, and therefore has to be tested
> with #if instead of #ifdef.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Link: https://linux-review.googlesource.com/id/If8a891e9fe01ea543e00b576852685afec0887e3
> ---
> arch/arm64/kernel/sleep.S | 2 +-
> arch/x86/kernel/acpi/wakeup_64.S | 2 +-
> include/linux/kasan.h | 10 ++++++----
> mm/kasan/common.c | 2 ++
> 4 files changed, 10 insertions(+), 6 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Even though hardware tag-based mode currently doesn't support checking
> vmalloc allocations, it doesn't use shadow memory and works with
> VMAP_STACK as is. Change VMAP_STACK definition accordingly.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Link: https://linux-review.googlesource.com/id/I3552cbc12321dec82cd7372676e9372a2eb452ac
> ---

Shouldn't this be in the other series?

FWIW,

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> There's no need for __kasan_unpoison_stack() helper, as it's only
> currently used in a single place. Removing it also removes unneeded
> arithmetic.
>
> No functional changes.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/Ie5ba549d445292fe629b4a96735e4034957bcc50
> ---
> mm/kasan/common.c | 12 +++---------
> 1 file changed, 3 insertions(+), 9 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

Thanks for spotting this simplification.

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Using kasan_reset_tag() currently results in a function call. As it's
> called quite often from the allocator code, this leads to a noticeable
> slowdown. Move it to include/linux/kasan.h and turn it into a static
> inline function. Also remove the now unneeded reset_tag() internal KASAN
> macro and use kasan_reset_tag() instead.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Link: https://linux-review.googlesource.com/id/I4d2061acfe91d480a75df00b07c22d8494ef14b5
> ---
> include/linux/kasan.h | 5 ++++-
> mm/kasan/common.c | 6 +++---
> mm/kasan/hw_tags.c | 9 ++-------
> mm/kasan/kasan.h | 4 ----
> mm/kasan/report.c | 4 ++--
> mm/kasan/report_hw_tags.c | 2 +-
> mm/kasan/report_sw_tags.c | 4 ++--
> mm/kasan/shadow.c | 4 ++--
> mm/kasan/sw_tags.c | 9 ++-------
> 9 files changed, 18 insertions(+), 29 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Using random_tag() currently results in a function call. Move its
> definition to mm/kasan/kasan.h and turn it into a static inline function
> for hardware tag-based mode to avoid uneeded function calls.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Link: https://linux-review.googlesource.com/id/Iac5b2faf9a912900e16cca6834d621f5d4abf427
> ---
> mm/kasan/hw_tags.c | 5 -----
> mm/kasan/kasan.h | 34 +++++++++++++++++-----------------
> 2 files changed, 17 insertions(+), 22 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

But see style comments below.

Shouldn't this also be a function?

+static inline u8 random_tag(void) { return hw_get_random_tag(); }

Or is there a reason why this was made a macro?

> +#else
> +static inline u8 random_tag(void)
> +{
> + return 0;
> +}

Could just be on 1 line:

+static inline u8 random_tag(void) { return 0; }

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Currently kasan_unpoison_memory() is used as both an external annotation
> and as an internal memory poisoning helper. Rename external annotation to
> kasan_unpoison_data() and inline the internal helper for hardware
> tag-based mode to avoid undeeded function calls.

I don't understand why this needs to be renamed again. The users of
kasan_unpoison_memory() outweigh those of kasan_unpoison_slab(), of
which there seems to be only 1!

So can't we just get rid of kasan_unpoison_slab() and just open-code it
in mm/mempool.c:kasan_unpoison_element()? That function is already
kasan-prefixed, so we can even place a small comment there (which would
also be an improvement over current interface, since
kasan_unpoison_slab() is not documented and its existence not quite
justified).

... this change would become unnecessary.

> /* Clear stale pointers from reused stack. */
> memset(s->addr, 0, THREAD_SIZE);
> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
> index a266b90636a1..4598c1364f19 100644
> --- a/mm/kasan/common.c
> +++ b/mm/kasan/common.c
> @@ -184,6 +184,16 @@ struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
> return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
> }
>
> +void kasan_unpoison_data(const void *address, size_t size)
> +{
> + kasan_unpoison_memory(address, size);
> +}
> +
> +void kasan_unpoison_slab(const void *ptr)
> +{
> + kasan_unpoison_memory(ptr, __ksize(ptr));
> +}
> +

This function is so simple, I think just open-coding

kasan_unpoison_memory(ptr, __ksize(ptr))

wherever required is much simpler, also bearing in mind the changes that
are coming to the rest of this series.

... this change would become unnecessary.

Thanks,
-- Marco

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, 'Andrey Konovalov' via kasan-dev wrote:
> Using kasan_poison_memory() or check_invalid_free() currently results in
> function calls. Move their definitions to mm/kasan/kasan.h and turn them
> into static inline functions for hardware tag-based mode to avoid
> unneeded function calls.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/Ia9d8191024a12d1374675b3d27197f10193f50bb
> ---
> mm/kasan/hw_tags.c | 15 ---------------
> mm/kasan/kasan.h | 28 ++++++++++++++++++++++++----
> 2 files changed, 24 insertions(+), 19 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/e14ac53d7c43b4381ad94665c63a154dffc04b6b.1605046662.git.andreyknvl%40google.com.

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, 'Andrey Konovalov' via kasan-dev wrote:

This should include <linux/static_key.h> -- although the rest of the
kernel seems to also inconsistently use on or the other. Since the name,
as referred to also by macros are "static keys", perhaps the
static_key.h header is more appropriate...

> #include <linux/kasan.h>
> #include <linux/kernel.h>
> #include <linux/memory.h>
> @@ -17,9 +19,104 @@
>
> #include "kasan.h"
>
> +enum kasan_arg_mode {
> + KASAN_ARG_MODE_DEFAULT,
> + KASAN_ARG_MODE_OFF,
> + KASAN_ARG_MODE_PROD,
> + KASAN_ARG_MODE_FULL,
> +};
> +
> +enum kasan_arg_stacktrace {
> + KASAN_ARG_STACKTRACE_DEFAULT,

It seems KASAN_ARG_STACKTRACE_DEFAULT is never used explicitly. Could
the switch statements just be changed to not have a 'default' but
instead refer to *DEFAULT where appropriate?

Is this an __init function, since it sets __ro_after_init vars?

> {
> + /* If hardware doesn't support MTE, do nothing. */
> + if (!system_supports_mte())
> + return;
> +
> + /* If KASAN is disabled, do nothing. */
> + if (kasan_arg_mode == KASAN_ARG_MODE_OFF)
> + return;

This is checked twice, once here and the in the switch. I think remove
the one here ^^^.

> + /* Choose KASAN mode if kasan boot parameter is not provided. */
> + if (kasan_arg_mode == KASAN_ARG_MODE_DEFAULT) {
> + if (IS_ENABLED(CONFIG_DEBUG_KERNEL))
> + kasan_arg_mode = KASAN_ARG_MODE_FULL;
> + else
> + kasan_arg_mode = KASAN_ARG_MODE_PROD;
> + }
> +
> + /* Preset parameter values based on the mode. */
> + switch (kasan_arg_mode) {
> + case KASAN_ARG_MODE_OFF:
> + return;
> + case KASAN_ARG_MODE_PROD:
> + static_branch_enable(&kasan_flag_enabled);
> + break;
> + case KASAN_ARG_MODE_FULL:
> + static_branch_enable(&kasan_flag_enabled);
> + static_branch_enable(&kasan_flag_stacktrace);
> + break;
> + default:

I'd suggest removing the 'default' cases in all the switches, so that we
get warnings in case we add new options and they aren't handled.

Here, having KASAN_ARG_MODE_DEFAULT is probably redundant, but so is
'default' ;-)

Would be good to get rid of the 'default' cases here.

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:
> kasan_poison_kfree() is currently only called for mempool allocations
> that are backed by either kmem_cache_alloc() or kmalloc(). Therefore, the
> page passed to kasan_poison_kfree() is always PageSlab() and there's no
> need to do the check. Remove it.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/If31f88726745da8744c6bea96fb32584e6c2778c
> ---
> mm/kasan/common.c | 11 +----------
> 1 file changed, 1 insertion(+), 10 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Rename kasan_poison_kfree() to kasan_slab_free_mempool() as it better
> reflects what this annotation does.

This function is again so simple, and now it seems it's mempool
specific, can't we just remove it and open-code it in mempool.c?

This is already a kasan-prefixed function, so if
kasan_slab_free_mempool() is only ever called in this function, we
should just call kasan_slab_free() here directly with the 2 extra args
it requires open-coded.

> else if (pool->alloc == mempool_alloc_pages)
> kasan_free_pages(element, (unsigned long)pool->pool_data);
> }

Thanks,
-- Marco

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> For hardware tag-based mode kasan_poison_memory() already rounds up the
> size. Do the same for software modes and remove round_up() from the common
> code.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/Ib397128fac6eba874008662b4964d65352db4aa4
> ---
> mm/kasan/common.c | 8 ++------
> mm/kasan/shadow.c | 1 +
> 2 files changed, 3 insertions(+), 6 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> set_tag() already ignores the tag for the generic mode, so just call it
> as is. Add a check for the generic mode to assign_tag(), and simplify its
> call in ____kasan_kmalloc().
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/I18905ca78fb4a3d60e1a34a4ca00247272480438
> ---
> mm/kasan/common.c | 11 ++++++-----
> 1 file changed, 6 insertions(+), 5 deletions(-)

Reviewed-by: Marco Elver <el...@google.com>

> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
> index 69ab880abacc..40ff3ce07a76 100644
> --- a/mm/kasan/common.c
> +++ b/mm/kasan/common.c
> @@ -238,6 +238,9 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object)
> static u8 assign_tag(struct kmem_cache *cache, const void *object,
> bool init, bool keep_tag)
> {
> + if (IS_ENABLED(CONFIG_KASAN_GENERIC))
> + return 0xff;
> +

Hopefully the compiler is clever enough to start inlining this function.

> /*
> * 1. When an object is kmalloc()'ed, two hooks are called:
> * kasan_slab_alloc() and kasan_kmalloc(). We assign the
> @@ -280,8 +283,8 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache,
> __memset(alloc_meta, 0, sizeof(*alloc_meta));
> }
>
> - if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
> - object = set_tag(object, assign_tag(cache, object, true, false));
> + /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */
> + object = set_tag(object, assign_tag(cache, object, true, false));
>
> return (void *)object;
> }
> @@ -362,9 +365,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object,
> KASAN_GRANULE_SIZE);
> redzone_end = round_up((unsigned long)object + cache->object_size,
> KASAN_GRANULE_SIZE);
> -
> - if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
> - tag = assign_tag(cache, object, false, keep_tag);
> + tag = assign_tag(cache, object, false, keep_tag);
>

The definition of 'tag' at the start of ____kasan_kmalloc() no longer
needs an initializer.

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> Currently it says that the memory gets poisoned by page_alloc code.
> Clarify this by mentioning the specific callback that poisons the
> memory.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Reviewed-by: Dmitry Vyukov <dvy...@google.com>
> Link: https://linux-review.googlesource.com/id/I1334dffb69b87d7986fab88a1a039cc3ea764725
> ---
> mm/kasan/common.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Marco Elver <el...@google.com>

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

s/that is can/that it can/

> + * be touched after it was freed, or
> + * 2. Object has a constructor, which means it's expected to
> + * retain its content until the next allocation, or
> + * 3. Object is too small.
> + * Otherwise cache->kasan_info.free_meta_offset = 0 is implied.
> */
> - if (*size <= cache->kasan_info.alloc_meta_offset ||
> - *size <= cache->kasan_info.free_meta_offset) {
> - cache->kasan_info.alloc_meta_offset = 0;
> - cache->kasan_info.free_meta_offset = 0;
> - *size = orig_size;
> - return;
> + if (cache->flags & SLAB_TYPESAFE_BY_RCU || cache->ctor ||

Braces around

(cache->flags & SLAB_TYPESAFE_BY_RCU)

> + cache->object_size < sizeof(struct kasan_free_meta)) {
> + ok_size = *size;
> +
> + cache->kasan_info.free_meta_offset = *size;
> + *size += sizeof(struct kasan_free_meta);
> +
> + /* If free meta doesn't fit, don't add it. */
> + if (*size > KMALLOC_MAX_SIZE) {
> + cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
> + *size = ok_size;
> + }
> }
>
> - *flags |= SLAB_KASAN;
> + redzone_size = optimal_redzone(cache->object_size);

redzone_size seems to used once, maybe just change the below to ...

> + /* Calculate size with optimal redzone. */
> + optimal_size = cache->object_size + redzone_size;

+ optimal_size = cache->object_size + optimal_redzone(cache->object_size);

?

> + /* Limit it with KMALLOC_MAX_SIZE (relevant for SLAB only). */
> + if (optimal_size > KMALLOC_MAX_SIZE)
> + optimal_size = KMALLOC_MAX_SIZE;
> + /* Use optimal size if the size with added metas is not large enough. */

Uses the optimal size if it's not "too large" rather than "not large
enough", right? As it is worded now makes me think this is a fallback,
whereas ideally it's the common case, right?

Braces not needed.

add {}

> + for (i = 0; i < KASAN_NR_FREE_STACKS; i++)

add {}

> + if (alloc_meta->free_pointer_tag[i] == tag)
> + return "use-after-free";

while the 2 pairs of {} aren't necessary, it definitely helps
readability in this case, as we have a multi-line then and for blocks.

> return "out-of-bounds";
> }
>
> diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
> index d1af6f6c6d12..be10d16bd129 100644
> --- a/mm/kasan/sw_tags.c
> +++ b/mm/kasan/sw_tags.c
> @@ -170,6 +170,8 @@ void kasan_set_free_info(struct kmem_cache *cache,
> u8 idx = 0;
>
> alloc_meta = kasan_get_alloc_meta(cache, object);
> + if (!alloc_meta)
> + return;
>
> #ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
> idx = alloc_meta->free_track_idx;
> @@ -187,6 +189,8 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
> int i = 0;
>
> alloc_meta = kasan_get_alloc_meta(cache, object);
> + if (!alloc_meta)
> + return NULL;
>
> #ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
> for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {

Thanks,
-- Marco

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> The reason cache merging is disabled with KASAN is because KASAN puts its
> metadata right after the allocated object. When the merged caches have
> slightly different sizes, the metadata ends up in different places, which
> KASAN doesn't support.
>
> It might be possible to adjust the metadata allocation algorithm and make
> it friendly to the cache merging code. Instead this change takes a simpler
> approach and allows merging caches when no metadata is present. Which is
> the case for hardware tag-based KASAN with kasan.mode=prod.
>
> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Link: https://linux-review.googlesource.com/id/Ia114847dfb2244f297d2cb82d592bf6a07455dba
> ---
> include/linux/kasan.h | 26 ++++++++++++++++++++++++--
> mm/kasan/common.c | 11 +++++++++++
> mm/slab_common.c | 11 ++++++++---
> 3 files changed, 43 insertions(+), 5 deletions(-)
>

[...]

>
> +/*
> + * Only allow cache merging when stack collection is disabled and no metadata
> + * is present.
> + */
> +slab_flags_t __kasan_never_merge(slab_flags_t flags)

I'm getting

mm/kasan/common.c:88:14: warning: no previous prototype for ‘__kasan_never_merge’ [-Wmissing-prototypes]

for a KASAN x86 build with W=1. Presumably because if !KASAN_HW_TAGS
then this is never needed and defined static inline in the header.

> +{
> + if (kasan_stack_collection_enabled())
> + return flags;
> + return flags & ~SLAB_KASAN;
> +}

Thanks,
-- Marco

[Andrey Konovalov]

No reason, will turn into a function in v10.

>
> > +#else
> > +static inline u8 random_tag(void)
> > +{
> > + return 0;
> > +}
>
> Could just be on 1 line:
>
> +static inline u8 random_tag(void) { return 0; }

Will do in v10.

Thanks!

[Andrey Konovalov]

On Wed, Nov 11, 2020 at 5:20 PM Marco Elver <el...@google.com> wrote:
>
> On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:
> > Even though hardware tag-based mode currently doesn't support checking
> > vmalloc allocations, it doesn't use shadow memory and works with
> > VMAP_STACK as is. Change VMAP_STACK definition accordingly.
> >
> > Signed-off-by: Andrey Konovalov <andre...@google.com>
> > Link: https://linux-review.googlesource.com/id/I3552cbc12321dec82cd7372676e9372a2eb452ac
> > ---
>
> Shouldn't this be in the other series?

I don't think it makes much difference considering the series will go
in together.

>
> FWIW,
>
> Reviewed-by: Marco Elver <el...@google.com>

Thanks!

[Andrey Konovalov]

Will do in v3, thanks!

[Andrey Konovalov]

Will fix all in v10/v3.

[Andrey Konovalov]

On Wed, Nov 11, 2020 at 7:53 PM Marco Elver <el...@google.com> wrote:
>
> On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:
> > Rename kasan_poison_kfree() to kasan_slab_free_mempool() as it better
> > reflects what this annotation does.
>
> This function is again so simple, and now it seems it's mempool
> specific, can't we just remove it and open-code it in mempool.c?

The simplification introduced in the previous patch is based on a
false assumption and will be reverted. Thus will keep this as a
separate function.

[Catalin Marinas]

On Tue, Nov 10, 2020 at 11:20:08PM +0100, Andrey Konovalov wrote:
> There's a config option CONFIG_KASAN_STACK that has to be enabled for
> KASAN to use stack instrumentation and perform validity checks for
> stack variables.
>
> There's no need to unpoison stack when CONFIG_KASAN_STACK is not enabled.
> Only call kasan_unpoison_task_stack[_below]() when CONFIG_KASAN_STACK is
> enabled.
>
> Note, that CONFIG_KASAN_STACK is an option that is currently always
> defined when CONFIG_KASAN is enabled, and therefore has to be tested
> with #if instead of #ifdef.
>

> Signed-off-by: Andrey Konovalov <andre...@google.com>

> Link: https://linux-review.googlesource.com/id/If8a891e9fe01ea543e00b576852685afec0887e3
> ---
> arch/arm64/kernel/sleep.S | 2 +-
> arch/x86/kernel/acpi/wakeup_64.S | 2 +-
> include/linux/kasan.h | 10 ++++++----
> mm/kasan/common.c | 2 ++
> 4 files changed, 10 insertions(+), 6 deletions(-)
>

> diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
> index ba40d57757d6..bdadfa56b40e 100644
> --- a/arch/arm64/kernel/sleep.S
> +++ b/arch/arm64/kernel/sleep.S
> @@ -133,7 +133,7 @@ SYM_FUNC_START(_cpu_resume)
> */
> bl cpu_do_resume
>
> -#ifdef CONFIG_KASAN
> +#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
> mov x0, sp
> bl kasan_unpoison_task_stack_below
> #endif

I don't understand why CONFIG_KASAN_STACK is not a bool (do you plan to
add more values to it?) but for arm64:

Acked-by: Catalin Marinas <catalin...@arm.com>

[Catalin Marinas]

On Tue, Nov 10, 2020 at 11:20:09PM +0100, Andrey Konovalov wrote:
> Even though hardware tag-based mode currently doesn't support checking
> vmalloc allocations, it doesn't use shadow memory and works with
> VMAP_STACK as is. Change VMAP_STACK definition accordingly.
>

> Signed-off-by: Andrey Konovalov <andre...@google.com>
> Link: https://linux-review.googlesource.com/id/I3552cbc12321dec82cd7372676e9372a2eb452ac
> ---

> arch/Kconfig | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 56b6ccc0e32d..7e7d14fae568 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -914,16 +914,16 @@ config VMAP_STACK
> default y
> bool "Use a virtually-mapped stack"
> depends on HAVE_ARCH_VMAP_STACK
> - depends on !KASAN || KASAN_VMALLOC
> + depends on !KASAN || KASAN_HW_TAGS || KASAN_VMALLOC

From the arm64 perspective:

Acked-by: Catalin Marinas <catalin...@arm.com>

[Catalin Marinas]

On Tue, Nov 10, 2020 at 11:20:15PM +0100, Andrey Konovalov wrote:
> Hardware tag-based KASAN mode is intended to eventually be used in
> production as a security mitigation. Therefore there's a need for finer
> control over KASAN features and for an existence of a kill switch.
>
> This change adds a few boot parameters for hardware tag-based KASAN that
> allow to disable or otherwise control particular KASAN features.
>

> The features that can be controlled are:

>
> 1. Whether KASAN is enabled at all.

> 2. Whether KASAN collects and saves alloc/free stacks.

> 3. Whether KASAN panics on a detected bug or not.
>

> With this change a new boot parameter kasan.mode allows to choose one of
> three main modes:
>
> - kasan.mode=off - KASAN is disabled, no tag checks are performed
> - kasan.mode=prod - only essential production features are enabled
> - kasan.mode=full - all KASAN features are enabled

Alternative naming if we want to avoid "production" (in case someone
considers MTE to be expensive in a production system):

- kasan.mode=off
- kasan.mode=on
- kasan.mode=debug

Anyway, whatever you prefer is fine by me:

Acked-by: Catalin Marinas <catalin...@arm.com>

[Marco Elver]

I believe this was what it was in RFC, and we had a long discussion on
what might be the most intuitive options. Since KASAN is still a
debugging tool for the most part, an "on" mode might imply we get all
the debugging facilities of regular KASAN. However, this is not the
case and misleading. Hence, we decided to be more explicit and avoid
"on".

[Catalin Marinas]

Even better, kasan.mode=fast ;).

--
Catalin

[Andrey Konovalov]

I don't remember if there's an actual reason. Perhaps this is
something that can be reworked later, but I don't want to get into
this in this series.

> Acked-by: Catalin Marinas <catalin...@arm.com>

Thanks!

[Andrey Konovalov]

On Wed, Nov 11, 2020 at 6:49 PM Marco Elver <el...@google.com> wrote:
>
> On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:
> > Currently kasan_unpoison_memory() is used as both an external annotation
> > and as an internal memory poisoning helper. Rename external annotation to
> > kasan_unpoison_data() and inline the internal helper for hardware
> > tag-based mode to avoid undeeded function calls.
>
> I don't understand why this needs to be renamed again. The users of
> kasan_unpoison_memory() outweigh those of kasan_unpoison_slab(), of
> which there seems to be only 1!

The idea is to make kasan_(un)poison_memory() functions inlinable for
internal use. It doesn't have anything to do with the number of times
they are used.

Perhaps we can drop the kasan_ prefix for the internal implementations
though, and keep using kasan_unpoison_memory() externally.

> So can't we just get rid of kasan_unpoison_slab() and just open-code it
> in mm/mempool.c:kasan_unpoison_element()? That function is already
> kasan-prefixed, so we can even place a small comment there (which would
> also be an improvement over current interface, since
> kasan_unpoison_slab() is not documented and its existence not quite
> justified).

We can, but this is a change unrelated to this patch.

[Andrey Konovalov]

On Wed, Nov 11, 2020 at 7:29 PM Marco Elver <el...@google.com> wrote:
>
> > +#include <linux/init.h>
> > +#include <linux/jump_label.h>
>
> This should include <linux/static_key.h> -- although the rest of the
> kernel seems to also inconsistently use on or the other. Since the name,
> as referred to also by macros are "static keys", perhaps the
> static_key.h header is more appropriate...

Will fix.

> > +enum kasan_arg_stacktrace {
> > + KASAN_ARG_STACKTRACE_DEFAULT,
>
> It seems KASAN_ARG_STACKTRACE_DEFAULT is never used explicitly. Could
> the switch statements just be changed to not have a 'default' but
> instead refer to *DEFAULT where appropriate?

We need to either cover all cases explicitly, or use default in each
switch, otherwise there's a warning. I guess covering everything
explicitly is a better approach, in case more values are added in the
future, as we'll get warnings for those if they aren't covered in
switches. Will do.

[Marco Elver]

On Thu, 12 Nov 2020 at 20:45, Andrey Konovalov <andre...@google.com> wrote:
>
> On Wed, Nov 11, 2020 at 6:49 PM Marco Elver <el...@google.com> wrote:
> >
> > On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:
> > > Currently kasan_unpoison_memory() is used as both an external annotation
> > > and as an internal memory poisoning helper. Rename external annotation to
> > > kasan_unpoison_data() and inline the internal helper for hardware
> > > tag-based mode to avoid undeeded function calls.
> >
> > I don't understand why this needs to be renamed again. The users of
> > kasan_unpoison_memory() outweigh those of kasan_unpoison_slab(), of
> > which there seems to be only 1!
>
> The idea is to make kasan_(un)poison_memory() functions inlinable for
> internal use. It doesn't have anything to do with the number of times
> they are used.
>
> Perhaps we can drop the kasan_ prefix for the internal implementations
> though, and keep using kasan_unpoison_memory() externally.

Whatever avoids changing the external interface, because it seems
really pointless. I can see why it's done, but it's a side-effect of
the various wrappers being added.

I'd much rather prefer we do it right from the beginning, and cleaning
up things very much is related to this series vs. just making things
uglier and hoping somebody will clean it up later.

> > So can't we just get rid of kasan_unpoison_slab() and just open-code it
> > in mm/mempool.c:kasan_unpoison_element()? That function is already
> > kasan-prefixed, so we can even place a small comment there (which would
> > also be an improvement over current interface, since
> > kasan_unpoison_slab() is not documented and its existence not quite
> > justified).
>
> We can, but this is a change unrelated to this patch.

Not quite, we're trying to optimize KASAN which is related -- this
patch as-is would obviously change, but replaced by a patch
simplifying things. This change as-is makes 2 changes outside of
KASAN, whereas if we removed it it would only be 1 and we end up with
less cruft.

[Andrey Konovalov]

Well, it uses sync, so technically it's not as fast as it could be with async :)

[Andrey Konovalov]

On Thu, Nov 12, 2020 at 12:06 AM Marco Elver <el...@google.com> wrote:
>
> > + /* Limit it with KMALLOC_MAX_SIZE (relevant for SLAB only). */
> > + if (optimal_size > KMALLOC_MAX_SIZE)
> > + optimal_size = KMALLOC_MAX_SIZE;
> > + /* Use optimal size if the size with added metas is not large enough. */
>
> Uses the optimal size if it's not "too large" rather than "not large
> enough", right?

Not really. If the redzone composed from metas is begger than optimal
redzone - we're good. If it's not large enough to reach optimal
redzone - we need to make it bigger.

> As it is worded now makes me think this is a fallback,
> whereas ideally it's the common case, right?

It's hard to say which case is more common, as optimal redzone size
varies and depends on the object size.

[...]

Will fix the rest of the comments, thanks!

[Andrey Konovalov]

On Thu, Nov 12, 2020 at 8:52 PM Marco Elver <el...@google.com> wrote:
>
> On Thu, 12 Nov 2020 at 20:45, Andrey Konovalov <andre...@google.com> wrote:
> >
> > On Wed, Nov 11, 2020 at 6:49 PM Marco Elver <el...@google.com> wrote:
> > >
> > > On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:
> > > > Currently kasan_unpoison_memory() is used as both an external annotation
> > > > and as an internal memory poisoning helper. Rename external annotation to
> > > > kasan_unpoison_data() and inline the internal helper for hardware
> > > > tag-based mode to avoid undeeded function calls.
> > >
> > > I don't understand why this needs to be renamed again. The users of
> > > kasan_unpoison_memory() outweigh those of kasan_unpoison_slab(), of
> > > which there seems to be only 1!
> >
> > The idea is to make kasan_(un)poison_memory() functions inlinable for
> > internal use. It doesn't have anything to do with the number of times
> > they are used.
> >
> > Perhaps we can drop the kasan_ prefix for the internal implementations
> > though, and keep using kasan_unpoison_memory() externally.
>
> Whatever avoids changing the external interface, because it seems
> really pointless. I can see why it's done, but it's a side-effect of
> the various wrappers being added.

It looks like unposion_memory() is already taken. Any suggestions for
internal KASAN poisoning function names?

[Marco Elver]

I still don't like that one of these functions just forwards to the
other, but we could use it to also give the external function a more
descriptive name.

I propose 2 options:

1. Name the internal helpers *poison_range().
2. Name the external function kasan_unpoison_range() instead of
kasan_unpoison_data().

Anything "memory" (or "data") in the allocators might not be too
helpful w.r.t. descriptive function names (i.e. stripping "memory"
from function names won't lessen descriptiveness given our context).
Perhaps kasan_poison_range() for the external function might in fact
improve the external interface (without looking at its arguments).

If we need to keep the internal helpers, I'd probably go so far as to
suggest renaming them to simply kasan_{poison,unpoison}(), and then
building the external kasan_{poison,unpoison}_foo() on top. But maybe
that's too much for now if it doesn't fit here. :-)

Thanks,
-- Marco

[Andrey Konovalov]

On Wed, Nov 11, 2020 at 4:13 PM Marco Elver <el...@google.com> wrote:
>
> On Tue, Nov 10, 2020 at 11:20PM +0100, Andrey Konovalov wrote:

> > The reason cache merging is disabled with KASAN is because KASAN puts its
> > metadata right after the allocated object. When the merged caches have
> > slightly different sizes, the metadata ends up in different places, which
> > KASAN doesn't support.
> >
> > It might be possible to adjust the metadata allocation algorithm and make
> > it friendly to the cache merging code. Instead this change takes a simpler
> > approach and allows merging caches when no metadata is present. Which is
> > the case for hardware tag-based KASAN with kasan.mode=prod.
> >

> > Signed-off-by: Andrey Konovalov <andre...@google.com>

> > Link: https://linux-review.googlesource.com/id/Ia114847dfb2244f297d2cb82d592bf6a07455dba
> > ---
> > include/linux/kasan.h | 26 ++++++++++++++++++++++++--
> > mm/kasan/common.c | 11 +++++++++++
> > mm/slab_common.c | 11 ++++++++---

> > 3 files changed, 43 insertions(+), 5 deletions(-)
> >
> > diff --git a/include/linux/kasan.h b/include/linux/kasan.h
> > index 534ab3e2935a..c754eca356f7 100644
> > --- a/include/linux/kasan.h
> > +++ b/include/linux/kasan.h
> > @@ -81,17 +81,35 @@ struct kasan_cache {
> > };
> >
> > #ifdef CONFIG_KASAN_HW_TAGS
> > +
> > DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled);
> > +
> > static inline kasan_enabled(void)
> > {
> > return static_branch_likely(&kasan_flag_enabled);
> > }
> > -#else
> > +
> > +slab_flags_t __kasan_never_merge(slab_flags_t flags);
> > +static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
> > +{
> > + if (kasan_enabled())
> > + return __kasan_never_merge(flags);
> > + return flags;
> > +}
> > +
> > +#else /* CONFIG_KASAN_HW_TAGS */
> > +
> > static inline kasan_enabled(void)
> > {
> > return true;
> > }
> > -#endif
> > +
> > +static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
> > +{
> > + return flags;
> > +}
> > +
> > +#endif /* CONFIG_KASAN_HW_TAGS */

> >
> > void __kasan_alloc_pages(struct page *page, unsigned int order);

> > static inline void kasan_alloc_pages(struct page *page, unsigned int order)

> > @@ -240,6 +258,10 @@ static inline kasan_enabled(void)
> > {
> > return false;
> > }
> > +static inline slab_flags_t kasan_never_merge(slab_flags_t flags)
> > +{
> > + return flags;

> > +}
> > static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
> > static inline void kasan_free_pages(struct page *page, unsigned int order) {}

> > static inline void kasan_cache_create(struct kmem_cache *cache,

> > diff --git a/mm/kasan/common.c b/mm/kasan/common.c
> > index 940b42231069..25b18c145b06 100644
> > --- a/mm/kasan/common.c
> > +++ b/mm/kasan/common.c
> > @@ -81,6 +81,17 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)
> > }
> > #endif /* CONFIG_KASAN_STACK */
> >

> > +/*
> > + * Only allow cache merging when stack collection is disabled and no metadata
> > + * is present.
> > + */
> > +slab_flags_t __kasan_never_merge(slab_flags_t flags)

> > +{
> > + if (kasan_stack_collection_enabled())
> > + return flags;
> > + return flags & ~SLAB_KASAN;
> > +}

> > +

> > void __kasan_alloc_pages(struct page *page, unsigned int order)
> > {
> > u8 tag;

> > diff --git a/mm/slab_common.c b/mm/slab_common.c
> > index f1b0c4a22f08..3042ee8ea9ce 100644
> > --- a/mm/slab_common.c
> > +++ b/mm/slab_common.c
> > @@ -18,6 +18,7 @@
> > #include <linux/seq_file.h>
> > #include <linux/proc_fs.h>
> > #include <linux/debugfs.h>
> > +#include <linux/kasan.h>
> > #include <asm/cacheflush.h>
> > #include <asm/tlbflush.h>
> > #include <asm/page.h>
> > @@ -49,12 +50,16 @@ static DECLARE_WORK(slab_caches_to_rcu_destroy_work,
> > slab_caches_to_rcu_destroy_workfn);
> >
> > /*
> > - * Set of flags that will prevent slab merging
> > + * Set of flags that will prevent slab merging.
> > + * Use slab_never_merge() instead.
> > */
> > #define SLAB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
> > SLAB_TRACE | SLAB_TYPESAFE_BY_RCU | SLAB_NOLEAKTRACE | \
> > SLAB_FAILSLAB | SLAB_KASAN)
>
> Rather than changing this to require using slab_never_merge() which
> removes SLAB_KASAN, could we not just have a function
> kasan_never_merge() that returns KASAN-specific flags that should never
> result in merging -- because as-is now, making kasan_never_merge()
> remove the SLAB_KASAN flag seems the wrong way around.
>
> Could we not just do this:
>
> #define SLAB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
> SLAB_TRACE | SLAB_TYPESAFE_BY_RCU | SLAB_NOLEAKTRACE | \
> SLAB_FAILSLAB | kasan_never_merge())
>
> ??

The issue here was that SLAB_KASAN is defined in slab.h, which
includes kasan.h, so we can't have a static inline definition of this
function for generic and software tag-based modes. So we can do this,
as long as we're fine with having kasan_never_merge() to be an actual
function call for all KASAN modes. I guess it's not a problem, so
let's do it this way.

>
> Of course that might be problematic if this always needs to be a
> compile-time constant, but currently that's not a requirement.
>
> > +/* KASAN allows merging in some configurations and will remove SLAB_KASAN. */
> > +#define slab_never_merge() (kasan_never_merge(SLAB_NEVER_MERGE))
>
> Braces unnecessary.

[Marco Elver]

On Tue, Nov 10, 2020 at 11:20PM +0100, 'Andrey Konovalov' via kasan-dev wrote:
[...]
> +/* kasan.mode=off/prod/full */
> +static int __init early_kasan_mode(char *arg)
> +{
> + if (!arg)
> + return -EINVAL;
> +
> + if (!strcmp(arg, "off"))
> + kasan_arg_mode = KASAN_ARG_MODE_OFF;
> + else if (!strcmp(arg, "prod"))
> + kasan_arg_mode = KASAN_ARG_MODE_PROD;
> + else if (!strcmp(arg, "full"))
> + kasan_arg_mode = KASAN_ARG_MODE_FULL;
> + else
> + return -EINVAL;

> +
> + return 0;
> +}

> +early_param("kasan.mode", early_kasan_mode);
> +
> +/* kasan.stack=off/on */
> +static int __init early_kasan_flag_stacktrace(char *arg)
> +{
> + if (!arg)
> + return -EINVAL;
> +
> + if (!strcmp(arg, "off"))
> + kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_OFF;
> + else if (!strcmp(arg, "on"))
> + kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_ON;
> + else
> + return -EINVAL;

> +
> + return 0;
> +}

> +early_param("kasan.stacktrace", early_kasan_flag_stacktrace);
> +
> +/* kasan.fault=report/panic */
> +static int __init early_kasan_fault(char *arg)
> +{
> + if (!arg)
> + return -EINVAL;
> +
> + if (!strcmp(arg, "report"))
> + kasan_arg_fault = KASAN_ARG_FAULT_REPORT;
> + else if (!strcmp(arg, "panic"))
> + kasan_arg_fault = KASAN_ARG_FAULT_PANIC;
> + else
> + return -EINVAL;

> +
> + return 0;
> +}

[...]

The above could be simplified, see suggestion below.

Thanks,
-- Marco

------ >8 ------

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index c91f2c06ecb5..71fc481ad21d 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -64,10 +64,8 @@ static int __init early_kasan_mode(char *arg)
kasan_arg_mode = KASAN_ARG_MODE_PROD;
else if (!strcmp(arg, "full"))
kasan_arg_mode = KASAN_ARG_MODE_FULL;
- else
- return -EINVAL;

- return 0;
+ return -EINVAL;
}
early_param("kasan.mode", early_kasan_mode);

@@ -81,10 +79,8 @@ static int __init early_kasan_flag_stacktrace(char *arg)
kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_OFF;
else if (!strcmp(arg, "on"))
kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_ON;
- else
- return -EINVAL;

- return 0;
+ return -EINVAL;
}
early_param("kasan.stacktrace", early_kasan_flag_stacktrace);

@@ -98,10 +94,8 @@ static int __init early_kasan_fault(char *arg)
kasan_arg_fault = KASAN_ARG_FAULT_REPORT;
else if (!strcmp(arg, "panic"))
kasan_arg_fault = KASAN_ARG_FAULT_PANIC;
- else
- return -EINVAL;

- return 0;
+ return -EINVAL;
}
early_param("kasan.fault", early_kasan_fault);

[Marco Elver]

Ah that clearly doesn't work. Hmm, never mind this suggestion, sorry.

Thanks,
-- Marco

[Andrey Konovalov]

=== Overview

Hardware tag-based KASAN mode [1] is intended to eventually be used in

production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.

This patchset adds a few boot parameters for hardware tag-based KASAN that
allow to disable or otherwise control particular KASAN features, as well
as provides some initial optimizations for running KASAN in production.

There's another planned patchset what will further optimize hardware
tag-based KASAN, provide proper benchmarking and tests, and will fully
enable tag-based KASAN for production use.

Hardware tag-based KASAN relies on arm64 Memory Tagging Extension (MTE)
[2] to perform memory and pointer tagging. Please see [3] and [4] for
detailed analysis of how MTE helps to fight memory safety problems.

The features that can be controlled are:

1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.

The patch titled "kasan: add and integrate kasan boot parameters" of this
series adds a few new boot parameters.

kasan.mode allows to choose one of three main modes:

- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled

The chosen mode provides default control values for the features mentioned

above. However it's also possible to override the default values by
providing:

- kasan.stacktrace=off/on - enable stacks collection
(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)

If kasan.mode parameter is not provided, it defaults to full when

CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.

It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by
the Android GKI (Generic Kernel Image) initiative.

=== Benchmarks

For now I've only performed a few simple benchmarks such as measuring
kernel boot time and slab memory usage after boot. There's an upcoming
patchset which will optimize KASAN further and include more detailed
benchmarking results.

The benchmarks were performed in QEMU and the results below exclude the
slowdown caused by QEMU memory tagging emulation (as it's different from
the slowdown that will be introduced by hardware and is therefore
irrelevant).

KASAN_HW_TAGS=y + kasan.mode=off introduces no performance or memory
impact compared to KASAN_HW_TAGS=n.

kasan.mode=prod (manually excluding tagging) introduces 3% of performance
and no memory impact (except memory used by hardware to store tags)
compared to kasan.mode=off.

kasan.mode=full has about 40% performance and 30% memory impact over
kasan.mode=prod. Both come from alloc/free stack collection.

=== Notes

This patchset is available here:

https://github.com/xairy/linux/tree/up-boot-mte-v3

This patchset is based on v10 of "kasan: add hardware tag-based mode for
arm64" patchset [1].

For testing in QEMU hardware tag-based KASAN requires:

1. QEMU built from master [6] (use "-machine virt,mte=on -cpu max" arguments
to run).
2. GCC version 10.

[1] https://lkml.org/lkml/2020/11/13/1154
[2] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/enhancing-memory-safety
[3] https://arxiv.org/pdf/1802.09517.pdf
[4] https://github.com/microsoft/MSRC-Security-Research/blob/master/papers/2020/Security%20analysis%20of%20memory%20tagging.pdf
[5] https://source.android.com/devices/architecture/kernel/generic-kernel-image
[6] https://github.com/qemu/qemu

=== History

Changes v2 -> v3:
- Rebase onto v10 of the HW_TAGS series.
- Add missing return type for kasan_enabled().
- Always define random_tag() as a function.
- Mark kasan wrappers as __always_inline.
- Don't "kasan: simplify kasan_poison_kfree" as it's based on a false
assumption, add a comment instead.
- Address documentation comments.
- Use <linux/static_key.h> instead of <linux/jump_label.h>.
- Rework switches in mm/kasan/hw_tags.c.
- Don't init tag in ____kasan_kmalloc().
- Correctly check SLAB_TYPESAFE_BY_RCU flag in mm/kasan/common.c.
- Readability fixes for "kasan: clean up metadata allocation and usage".
- Change kasan_never_merge() to return SLAB_KASAN instead of excluding it
from flags.
- (Vincenzo) Address concerns from checkpatch.pl (courtesy of Marco Elver).

Changes v1 -> v2:
- Rebased onto v9 of the HW_TAGS patchset.
- Don't initialize static branches in kasan_init_hw_tags_cpu(), as
cpu_enable_mte() can't sleep; do in in kasan_init_hw_tags() instead.
- Rename kasan.stacks to kasan.stacktrace.

Changes RFC v2 -> v1:
- Rebrand the patchset from fully enabling production use to partially
addressing that; another optimization and testing patchset will be
required.
- Rebase onto v8 of KASAN_HW_TAGS series.
- Fix "ASYNC" -> "async" typo.
- Rework depends condition for VMAP_STACK and update config text.
- Remove unneeded reset_tag() macro, use kasan_reset_tag() instead.
- Rename kasan.stack to kasan.stacks to avoid confusion with stack
instrumentation.
- Introduce kasan_stack_collection_enabled() and kasan_is_enabled()
helpers.
- Simplify kasan_stack_collection_enabled() usage.
- Rework SLAB_KASAN flag and metadata allocation (see the corresponding
patch for details).
- Allow cache merging with KASAN_HW_TAGS when kasan.stacks is off.
- Use sync mode dy default for both prod and full KASAN modes.
- Drop kasan.trap=sync/async boot parameter, as async mode isn't supported
yet.
- Choose prod or full mode depending on CONFIG_DEBUG_KERNEL when no
kasan.mode boot parameter is provided.
- Drop krealloc optimization changes, those will be included in a separate
patchset.
- Update KASAN documentation to mention boot parameters.

Changes RFC v1 -> RFC v2:
- Rework boot parameters.
- Drop __init from empty kasan_init_tags() definition.
- Add cpu_supports_mte() helper that can be used during early boot and use
it in kasan_init_tags()
- Lots of new KASAN optimization commits.

Andrey Konovalov (19):
kasan: simplify quarantine_put call site
kasan: rename get_alloc/free_info
kasan: introduce set_alloc_info
kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK
kasan: allow VMAP_STACK for HW_TAGS mode
kasan: remove __kasan_unpoison_stack
kasan: inline kasan_reset_tag for tag-based modes
kasan: inline random_tag for HW_TAGS
kasan: open-code kasan_unpoison_slab
kasan: inline (un)poison_range and check_invalid_free
kasan: add and integrate kasan boot parameters
kasan, mm: check kasan_enabled in annotations
kasan, mm: rename kasan_poison_kfree
kasan: don't round_up too much
kasan: simplify assign_tag and set_tag calls
kasan: clarify comment in __kasan_kfree_large
kasan: clean up metadata allocation and usage
kasan, mm: allow cache merging with no metadata
kasan: update documentation

Documentation/dev-tools/kasan.rst | 186 ++++++++++++--------
arch/Kconfig | 8 +-

arch/arm64/kernel/sleep.S | 2 +-
arch/x86/kernel/acpi/wakeup_64.S | 2 +-

include/linux/kasan.h | 245 ++++++++++++++++++++------
include/linux/mm.h | 22 ++-
mm/kasan/common.c | 283 ++++++++++++++++++------------
mm/kasan/generic.c | 27 +--
mm/kasan/hw_tags.c | 185 +++++++++++++++----
mm/kasan/kasan.h | 120 +++++++++----
mm/kasan/quarantine.c | 13 +-
mm/kasan/report.c | 61 ++++---
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 15 +-
mm/kasan/shadow.c | 5 +-
mm/kasan/sw_tags.c | 17 +-
mm/mempool.c | 4 +-
mm/slab_common.c | 3 +-
18 files changed, 824 insertions(+), 376 deletions(-)

--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Move get_free_info() call into quarantine_put() to simplify the call site.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Reviewed-by: Marco Elver <el...@google.com>
Link: https://linux-review.googlesource.com/id/Iab0f04e7ebf8d83247024b7190c67c3c34c7940f
---
mm/kasan/common.c | 2 +-
mm/kasan/kasan.h | 5 ++---
mm/kasan/quarantine.c | 3 ++-
3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 998aede4d172..e11fac2ee30c 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -317,7 +317,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

kasan_set_free_info(cache, object, tag);

- quarantine_put(get_free_info(cache, object), cache);
+ quarantine_put(cache, object);

return IS_ENABLED(CONFIG_KASAN_GENERIC);
}
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 64560cc71191..13c511e85d5f 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -216,12 +216,11 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

#if defined(CONFIG_KASAN_GENERIC) && \
(defined(CONFIG_SLAB) || defined(CONFIG_SLUB))
-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache);
+void quarantine_put(struct kmem_cache *cache, void *object);
void quarantine_reduce(void);
void quarantine_remove_cache(struct kmem_cache *cache);
#else
-static inline void quarantine_put(struct kasan_free_meta *info,
- struct kmem_cache *cache) { }
+static inline void quarantine_put(struct kmem_cache *cache, void *object) { }

static inline void quarantine_reduce(void) { }
static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif

diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index 580ff5610fc1..a0792f0d6d0f 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -161,11 +161,12 @@ static void qlist_free_all(struct qlist_head *q, struct kmem_cache *cache)
qlist_init(q);
}

-void quarantine_put(struct kasan_free_meta *info, struct kmem_cache *cache)
+void quarantine_put(struct kmem_cache *cache, void *object)
{
unsigned long flags;
struct qlist_head *q;

struct qlist_head temp = QLIST_INIT;

+ struct kasan_free_meta *info = get_free_info(cache, object);

/*
* Note: irq must be disabled until after we move the batch to the

--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Rename get_alloc_info() and get_free_info() to kasan_get_alloc_meta()
and kasan_get_free_meta() to better reflect what those do and avoid
confusion with kasan_set_free_info().

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>
Reviewed-by: Marco Elver <el...@google.com>

Link: https://linux-review.googlesource.com/id/Ib6e4ba61c8b12112b403d3479a9799ac8fff8de1
---
mm/kasan/common.c | 16 ++++++++--------
mm/kasan/generic.c | 12 ++++++------
mm/kasan/hw_tags.c | 4 ++--
mm/kasan/kasan.h | 8 ++++----
mm/kasan/quarantine.c | 4 ++--
mm/kasan/report.c | 12 ++++++------
mm/kasan/report_sw_tags.c | 2 +-
mm/kasan/sw_tags.c | 4 ++--
8 files changed, 31 insertions(+), 31 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index e11fac2ee30c..8197399b0a1f 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -181,14 +181,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
sizeof(struct kasan_free_meta) : 0);
}

-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object)
{
return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object)
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
@@ -265,13 +265,13 @@ static u8 assign_tag(struct kmem_cache *cache, const void *object,

void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,

const void *object)
{
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;

if (!(cache->flags & SLAB_KASAN))

return (void *)object;

- alloc_info = get_alloc_info(cache, object);
- __memset(alloc_info, 0, sizeof(*alloc_info));

+ alloc_meta = kasan_get_alloc_meta(cache, object);

+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));
@@ -357,7 +357,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
KASAN_KMALLOC_REDZONE);

if (cache->flags & SLAB_KASAN)

- kasan_set_track(&get_alloc_info(cache, object)->alloc_track, flags);
+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

return set_tag(object, tag);
}
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c

index da3608187c25..9c6b77f8c4a4 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -329,7 +329,7 @@ void kasan_record_aux_stack(void *addr)
{
struct page *page = kasan_addr_to_page(addr);
struct kmem_cache *cache;
- struct kasan_alloc_meta *alloc_info;
+ struct kasan_alloc_meta *alloc_meta;
void *object;

if (is_kfence_address(addr) || !(page && PageSlab(page)))
@@ -337,13 +337,13 @@ void kasan_record_aux_stack(void *addr)

cache = page->slab_cache;
object = nearest_obj(cache, page, addr);

- alloc_info = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

/*

* record the last two call_rcu() call stacks.

*/
- alloc_info->aux_stack[1] = alloc_info->aux_stack[0];
- alloc_info->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
+ alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];
+ alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
}

void kasan_set_free_info(struct kmem_cache *cache,
@@ -351,7 +351,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
{
struct kasan_free_meta *free_meta;

- free_meta = get_free_info(cache, object);
+ free_meta = kasan_get_free_meta(cache, object);
kasan_set_track(&free_meta->free_track, GFP_NOWAIT);

/*
@@ -365,5 +365,5 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,

{
if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_KMALLOC_FREETRACK)
return NULL;

- return &get_free_info(cache, object)->free_track;
+ return &kasan_get_free_meta(cache, object)->free_track;
}
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 3f9232464ed4..68e77363e58b 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -75,7 +75,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

kasan_set_track(&alloc_meta->free_track[0], GFP_NOWAIT);
}

@@ -84,6 +84,6 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

return &alloc_meta->free_track[0];
}
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 13c511e85d5f..0eab7e4cecb8 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -149,10 +149,10 @@ struct kasan_free_meta {
#endif
};

-struct kasan_alloc_meta *get_alloc_info(struct kmem_cache *cache,
- const void *object);
-struct kasan_free_meta *get_free_info(struct kmem_cache *cache,
- const void *object);
+struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
+ const void *object);
+struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
+ const void *object);

void poison_range(const void *address, size_t size, u8 value);
void unpoison_range(const void *address, size_t size);
diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index a0792f0d6d0f..0da3d37e1589 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -166,7 +166,7 @@ void quarantine_put(struct kmem_cache *cache, void *object)

unsigned long flags;
struct qlist_head *q;
struct qlist_head temp = QLIST_INIT;

- struct kasan_free_meta *info = get_free_info(cache, object);
+ struct kasan_free_meta *meta = kasan_get_free_meta(cache, object);

/*
* Note: irq must be disabled until after we move the batch to the

@@ -179,7 +179,7 @@ void quarantine_put(struct kmem_cache *cache, void *object)
local_irq_save(flags);

q = this_cpu_ptr(&cpu_quarantine);
- qlist_put(q, &info->quarantine_link, cache->size);
+ qlist_put(q, &meta->quarantine_link, cache->size);
if (unlikely(q->bytes > QUARANTINE_PERCPU_SIZE)) {
qlist_move_all(q, &temp);

diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index a69c2827a125..df16bef0d810 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -164,12 +164,12 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,

static void describe_object(struct kmem_cache *cache, void *object,

const void *addr, u8 tag)
{

- struct kasan_alloc_meta *alloc_info = get_alloc_info(cache, object);
+ struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);

if (cache->flags & SLAB_KASAN) {

struct kasan_track *free_track;

- print_track(&alloc_info->alloc_track, "Allocated");
+ print_track(&alloc_meta->alloc_track, "Allocated");
pr_err("\n");

free_track = kasan_get_free_track(cache, object, tag);

if (free_track) {
@@ -178,14 +178,14 @@ static void describe_object(struct kmem_cache *cache, void *object,
}

#ifdef CONFIG_KASAN_GENERIC
- if (alloc_info->aux_stack[0]) {
+ if (alloc_meta->aux_stack[0]) {

pr_err("Last call_rcu():\n");

- print_stack(alloc_info->aux_stack[0]);

+ print_stack(alloc_meta->aux_stack[0]);

pr_err("\n");
}
- if (alloc_info->aux_stack[1]) {

+ if (alloc_meta->aux_stack[1]) {

pr_err("Second to last call_rcu():\n");

- print_stack(alloc_info->aux_stack[1]);

+ print_stack(alloc_meta->aux_stack[1]);

pr_err("\n");
}
#endif
diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index aebc44a29e83..317100fd95b9 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -46,7 +46,7 @@ const char *get_bug_type(struct kasan_access_info *info)
if (page && PageSlab(page)) {
cache = page->slab_cache;
object = nearest_obj(cache, page, (void *)addr);
- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

for (i = 0; i < KASAN_NR_FREE_STACKS; i++)

if (alloc_meta->free_pointer_tag[i] == tag)

diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index a518483f3965..6d7648cc3b98 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -174,7 +174,7 @@ void kasan_set_free_info(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;
u8 idx = 0;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
idx = alloc_meta->free_track_idx;

@@ -191,7 +191,7 @@ struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
struct kasan_alloc_meta *alloc_meta;
int i = 0;

- alloc_meta = get_alloc_info(cache, object);

+ alloc_meta = kasan_get_alloc_meta(cache, object);

#ifdef CONFIG_KASAN_SW_TAGS_IDENTIFY
for (i = 0; i < KASAN_NR_FREE_STACKS; i++) {

--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Add set_alloc_info() helper and move kasan_set_track() into it. This will
simplify the code for one of the upcoming changes.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Dmitry Vyukov <dvy...@google.com>
Reviewed-by: Marco Elver <el...@google.com>

Link: https://linux-review.googlesource.com/id/I0316193cbb4ecc9b87b7c2eee0dd79f8ec908c1a
---
mm/kasan/common.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 8197399b0a1f..0a420f1dbc54 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -327,6 +327,11 @@ bool kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip)
return __kasan_slab_free(cache, object, ip, true);
}

+static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
+{

+ kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);

+}
+

static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

size_t size, gfp_t flags, bool keep_tag)
{

@@ -357,7 +362,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,

KASAN_KMALLOC_REDZONE);

if (cache->flags & SLAB_KASAN)

- kasan_set_track(&kasan_get_alloc_meta(cache, object)->alloc_track, flags);
+ set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);
}
--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

There's a config option CONFIG_KASAN_STACK that has to be enabled for
KASAN to use stack instrumentation and perform validity checks for
stack variables.

There's no need to unpoison stack when CONFIG_KASAN_STACK is not enabled.
Only call kasan_unpoison_task_stack[_below]() when CONFIG_KASAN_STACK is
enabled.

Note, that CONFIG_KASAN_STACK is an option that is currently always
defined when CONFIG_KASAN is enabled, and therefore has to be tested
with #if instead of #ifdef.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Marco Elver <el...@google.com>
Acked-by: Catalin Marinas <catalin...@arm.com>
Link: https://linux-review.googlesource.com/id/If8a891e9fe01ea543e00b576852685afec0887e3
---

arch/arm64/kernel/sleep.S | 2 +-
arch/x86/kernel/acpi/wakeup_64.S | 2 +-

include/linux/kasan.h | 10 ++++++----
mm/kasan/common.c | 2 ++
4 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index ba40d57757d6..bdadfa56b40e 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -133,7 +133,7 @@ SYM_FUNC_START(_cpu_resume)
*/
bl cpu_do_resume

-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
mov x0, sp
bl kasan_unpoison_task_stack_below
#endif

diff --git a/arch/x86/kernel/acpi/wakeup_64.S b/arch/x86/kernel/acpi/wakeup_64.S
index c8daa92f38dc..5d3a0b8fd379 100644
--- a/arch/x86/kernel/acpi/wakeup_64.S
+++ b/arch/x86/kernel/acpi/wakeup_64.S
@@ -112,7 +112,7 @@ SYM_FUNC_START(do_suspend_lowlevel)
movq pt_regs_r14(%rax), %r14
movq pt_regs_r15(%rax), %r15

-#ifdef CONFIG_KASAN
+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK

/*
* The suspend path may have poisoned some areas deeper in the stack,
* which we now need to unpoison.
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 0c89e6fdd29e..f2109bf0c5f9 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -76,8 +76,6 @@ static inline void kasan_disable_current(void) {}

void kasan_unpoison_range(const void *address, size_t size);

-void kasan_unpoison_task_stack(struct task_struct *task);
-
void kasan_alloc_pages(struct page *page, unsigned int order);
void kasan_free_pages(struct page *page, unsigned int order);

@@ -122,8 +120,6 @@ void kasan_restore_multi_shot(bool enabled);

static inline void kasan_unpoison_range(const void *address, size_t size) {}

-static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
-

static inline void kasan_alloc_pages(struct page *page, unsigned int order) {}
static inline void kasan_free_pages(struct page *page, unsigned int order) {}

@@ -175,6 +171,12 @@ static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

#endif /* CONFIG_KASAN */

+#if defined(CONFIG_KASAN) && CONFIG_KASAN_STACK
+void kasan_unpoison_task_stack(struct task_struct *task);
+#else
+static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
+#endif
+
#ifdef CONFIG_KASAN_GENERIC

void kasan_cache_shrink(struct kmem_cache *cache);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 0a420f1dbc54..7648a2452a01 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -64,6 +64,7 @@ void kasan_unpoison_range(const void *address, size_t size)
unpoison_range(address, size);
}

+#if CONFIG_KASAN_STACK
static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
{
void *base = task_stack_page(task);
@@ -90,6 +91,7 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark)

unpoison_range(base, watermark - base);
}
+#endif /* CONFIG_KASAN_STACK */

void kasan_alloc_pages(struct page *page, unsigned int order)

{
--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Even though hardware tag-based mode currently doesn't support checking
vmalloc allocations, it doesn't use shadow memory and works with
VMAP_STACK as is. Change VMAP_STACK definition accordingly.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Marco Elver <el...@google.com>
Acked-by: Catalin Marinas <catalin...@arm.com>

Link: https://linux-review.googlesource.com/id/I3552cbc12321dec82cd7372676e9372a2eb452ac
---
arch/Kconfig | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/Kconfig b/arch/Kconfig

index 9ebdab3d0ca2..546869c3269d 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -921,16 +921,16 @@ config VMAP_STACK

default y
bool "Use a virtually-mapped stack"
depends on HAVE_ARCH_VMAP_STACK
- depends on !KASAN || KASAN_VMALLOC
+ depends on !KASAN || KASAN_HW_TAGS || KASAN_VMALLOC

help
Enable this if you want the use virtually-mapped kernel stacks
with guard pages. This causes kernel stack overflows to be
caught immediately rather than causing difficult-to-diagnose
corruption.

- To use this with KASAN, the architecture must support backing
- virtual mappings with real shadow memory, and KASAN_VMALLOC must
- be enabled.
+ To use this with software KASAN modes, the architecture must support
+ backing virtual mappings with real shadow memory, and KASAN_VMALLOC
+ must be enabled.

config ARCH_OPTIONAL_KERNEL_RWX
def_bool n
--

2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

There's no need for __kasan_unpoison_stack() helper, as it's only
currently used in a single place. Removing it also removes unneeded
arithmetic.

No functional changes.

Signed-off-by: Andrey Konovalov <andre...@google.com>

Reviewed-by: Dmitry Vyukov <dvy...@google.com>
Reviewed-by: Marco Elver <el...@google.com>

Link: https://linux-review.googlesource.com/id/Ie5ba549d445292fe629b4a96735e4034957bcc50
---
mm/kasan/common.c | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 7648a2452a01..fabd843eff3d 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -65,18 +65,12 @@ void kasan_unpoison_range(const void *address, size_t size)
}

#if CONFIG_KASAN_STACK
-static void __kasan_unpoison_stack(struct task_struct *task, const void *sp)
-{
- void *base = task_stack_page(task);
- size_t size = sp - base;
-
- unpoison_range(base, size);
-}
-
/* Unpoison the entire stack for a task. */
void kasan_unpoison_task_stack(struct task_struct *task)
{
- __kasan_unpoison_stack(task, task_stack_page(task) + THREAD_SIZE);
+ void *base = task_stack_page(task);
+
+ unpoison_range(base, THREAD_SIZE);
}

/* Unpoison the stack for the current task beyond a watermark sp value. */
--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Using kasan_reset_tag() currently results in a function call. As it's
called quite often from the allocator code, this leads to a noticeable
slowdown. Move it to include/linux/kasan.h and turn it into a static
inline function. Also remove the now unneeded reset_tag() internal KASAN
macro and use kasan_reset_tag() instead.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Marco Elver <el...@google.com>
Link: https://linux-review.googlesource.com/id/I4d2061acfe91d480a75df00b07c22d8494ef14b5
---
include/linux/kasan.h | 5 ++++-
mm/kasan/common.c | 6 +++---
mm/kasan/hw_tags.c | 9 ++-------
mm/kasan/kasan.h | 4 ----
mm/kasan/report.c | 4 ++--
mm/kasan/report_hw_tags.c | 2 +-
mm/kasan/report_sw_tags.c | 4 ++--
mm/kasan/shadow.c | 4 ++--
mm/kasan/sw_tags.c | 9 ++-------
9 files changed, 18 insertions(+), 29 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index f2109bf0c5f9..1594177f86bb 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -193,7 +193,10 @@ static inline void kasan_record_aux_stack(void *ptr) {}

#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)

-void *kasan_reset_tag(const void *addr);
+static inline void *kasan_reset_tag(const void *addr)
+{
+ return (void *)arch_kasan_reset_tag(addr);
+}

bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index fabd843eff3d..1ac4f435c679 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -180,14 +180,14 @@ size_t kasan_metadata_size(struct kmem_cache *cache)
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
- return (void *)reset_tag(object) + cache->kasan_info.alloc_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.alloc_meta_offset;
}

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object)
{
BUILD_BUG_ON(sizeof(struct kasan_free_meta) > 32);
- return (void *)reset_tag(object) + cache->kasan_info.free_meta_offset;
+ return kasan_reset_tag(object) + cache->kasan_info.free_meta_offset;
}

void kasan_poison_slab(struct page *page)
@@ -284,7 +284,7 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,

tag = get_tag(object);
tagged_object = object;
- object = reset_tag(object);
+ object = kasan_reset_tag(object);

if (is_kfence_address(object))
return false;
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 68e77363e58b..a34476764f1d 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -31,18 +31,13 @@ void __init kasan_init_hw_tags(void)
pr_info("KernelAddressSanitizer initialized\n");
}

-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-

void poison_range(const void *address, size_t size, u8 value)

{
/* Skip KFENCE memory if called explicitly outside of sl*b. */
if (is_kfence_address(address))
return;

- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), value);
}

@@ -52,7 +47,7 @@ void unpoison_range(const void *address, size_t size)
if (is_kfence_address(address))
return;

- hw_set_mem_tag_range(reset_tag(address),
+ hw_set_mem_tag_range(kasan_reset_tag(address),
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 0eab7e4cecb8..5e8cd2080369 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -248,15 +248,11 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)
return addr;
}
#endif
-#ifndef arch_kasan_reset_tag
-#define arch_kasan_reset_tag(addr) ((void *)(addr))
-#endif
#ifndef arch_kasan_get_tag
#define arch_kasan_get_tag(addr) 0
#endif

#define set_tag(addr, tag) ((void *)arch_kasan_set_tag((addr), (tag)))
-#define reset_tag(addr) ((void *)arch_kasan_reset_tag(addr))
#define get_tag(addr) arch_kasan_get_tag(addr)

#ifdef CONFIG_KASAN_HW_TAGS
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index df16bef0d810..76a0e3ae2049 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -328,7 +328,7 @@ void kasan_report_invalid_free(void *object, unsigned long ip)
unsigned long flags;
u8 tag = get_tag(object);

- object = reset_tag(object);
+ object = kasan_reset_tag(object);

#if IS_ENABLED(CONFIG_KUNIT)
if (current->kunit_test)
@@ -361,7 +361,7 @@ static void __kasan_report(unsigned long addr, size_t size, bool is_write,
disable_trace_on_warning();

tagged_addr = (void *)addr;
- untagged_addr = reset_tag(tagged_addr);
+ untagged_addr = kasan_reset_tag(tagged_addr);

info.access_addr = tagged_addr;
if (addr_has_metadata(untagged_addr))
diff --git a/mm/kasan/report_hw_tags.c b/mm/kasan/report_hw_tags.c
index da543eb832cd..57114f0e14d1 100644
--- a/mm/kasan/report_hw_tags.c
+++ b/mm/kasan/report_hw_tags.c
@@ -22,7 +22,7 @@ const char *get_bug_type(struct kasan_access_info *info)

void *find_first_bad_addr(void *addr, size_t size)
{
- return reset_tag(addr);
+ return kasan_reset_tag(addr);
}

void metadata_fetch_row(char *buffer, void *row)
diff --git a/mm/kasan/report_sw_tags.c b/mm/kasan/report_sw_tags.c
index 317100fd95b9..7604b46239d4 100644
--- a/mm/kasan/report_sw_tags.c
+++ b/mm/kasan/report_sw_tags.c
@@ -41,7 +41,7 @@ const char *get_bug_type(struct kasan_access_info *info)
int i;

tag = get_tag(info->access_addr);
- addr = reset_tag(info->access_addr);
+ addr = kasan_reset_tag(info->access_addr);
page = kasan_addr_to_page(addr);

if (page && PageSlab(page)) {
cache = page->slab_cache;

@@ -72,7 +72,7 @@ const char *get_bug_type(struct kasan_access_info *info)
void *find_first_bad_addr(void *addr, size_t size)
{
u8 tag = get_tag(addr);
- void *p = reset_tag(addr);
+ void *p = kasan_reset_tag(addr);
void *end = p + size;

while (p < end && tag == *(u8 *)kasan_mem_to_shadow(p))
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index d8a122f887a0..37153bd1c126 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -82,7 +82,7 @@ void poison_range(const void *address, size_t size, u8 value)
* some of the callers (e.g. kasan_poison_object_data) pass tagged
* addresses to this function.
*/
- address = reset_tag(address);
+ address = kasan_reset_tag(address);

/* Skip KFENCE memory if called explicitly outside of sl*b. */
if (is_kfence_address(address))
@@ -103,7 +103,7 @@ void unpoison_range(const void *address, size_t size)
* some of the callers (e.g. kasan_unpoison_object_data) pass tagged
* addresses to this function.
*/
- address = reset_tag(address);
+ address = kasan_reset_tag(address);

/*
* Skip KFENCE memory if called explicitly outside of sl*b. Also note
diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index 6d7648cc3b98..e17de2619bbf 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -67,11 +67,6 @@ u8 random_tag(void)
return (u8)(state % (KASAN_TAG_MAX + 1));
}

-void *kasan_reset_tag(const void *addr)
-{
- return reset_tag(addr);
-}
-
bool check_memory_region(unsigned long addr, size_t size, bool write,
unsigned long ret_ip)
{
@@ -107,7 +102,7 @@ bool check_memory_region(unsigned long addr, size_t size, bool write,
if (tag == KASAN_TAG_KERNEL)
return true;

- untagged_addr = reset_tag((const void *)addr);
+ untagged_addr = kasan_reset_tag((const void *)addr);
if (unlikely(untagged_addr <
kasan_shadow_to_mem((void *)KASAN_SHADOW_START))) {
return !kasan_report(addr, size, write, ret_ip);
@@ -126,7 +121,7 @@ bool check_memory_region(unsigned long addr, size_t size, bool write,
bool check_invalid_free(void *addr)
{
u8 tag = get_tag(addr);
- u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(reset_tag(addr)));
+ u8 shadow_byte = READ_ONCE(*(u8 *)kasan_mem_to_shadow(kasan_reset_tag(addr)));

return (shadow_byte == KASAN_TAG_INVALID) ||
(tag != KASAN_TAG_KERNEL && tag != shadow_byte);
--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Using random_tag() currently results in a function call. Move its
definition to mm/kasan/kasan.h and turn it into a static inline function
for hardware tag-based mode to avoid uneeded function calls.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Reviewed-by: Marco Elver <el...@google.com>

Link: https://linux-review.googlesource.com/id/Iac5b2faf9a912900e16cca6834d621f5d4abf427
---
mm/kasan/hw_tags.c | 5 -----
mm/kasan/kasan.h | 31 ++++++++++++++-----------------
2 files changed, 14 insertions(+), 22 deletions(-)

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index a34476764f1d..3cdd87d189f6 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -51,11 +51,6 @@ void unpoison_range(const void *address, size_t size)
round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
}

-u8 random_tag(void)
-{
- return hw_get_random_tag();
-}
-
bool check_invalid_free(void *addr)
{
u8 ptr_tag = get_tag(addr);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 5e8cd2080369..7876a2547b7d 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -190,6 +190,12 @@ static inline bool addr_has_metadata(const void *addr)

#endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */

+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+void print_tags(u8 addr_tag, const void *addr);
+#else
+static inline void print_tags(u8 addr_tag, const void *addr) { }
+#endif
+
bool check_invalid_free(void *addr);

void *find_first_bad_addr(void *addr, size_t size);
@@ -225,23 +231,6 @@ static inline void quarantine_reduce(void) { }

static inline void quarantine_remove_cache(struct kmem_cache *cache) { }
#endif

-#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
-
-void print_tags(u8 addr_tag, const void *addr);
-
-u8 random_tag(void);
-
-#else
-
-static inline void print_tags(u8 addr_tag, const void *addr) { }
-
-static inline u8 random_tag(void)
-{
- return 0;
-}
-
-#endif
-
#ifndef arch_kasan_set_tag

static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)

{
@@ -281,6 +270,14 @@ static inline const void *arch_kasan_set_tag(const void *addr, u8 tag)

#endif /* CONFIG_KASAN_HW_TAGS */

+#ifdef CONFIG_KASAN_SW_TAGS
+u8 random_tag(void);
+#elif defined(CONFIG_KASAN_HW_TAGS)
+static inline u8 random_tag(void) { return hw_get_random_tag(); }
+#else
+static inline u8 random_tag(void) { return 0; }
+#endif
+
/*
* Exported functions for interfaces called from assembly or from generated
* code. Declarations here to avoid warning about missing declarations.
--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

There's the external annotation kasan_unpoison_slab() that is currently
defined as static inline and uses kasan_unpoison_range(). Open-code this
function in mempool.c. Otherwise with an upcoming change this function
will result in an unnecessary function call.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/Ia7c8b659f79209935cbaab3913bf7f082cc43a0e
---
include/linux/kasan.h | 6 ------
mm/mempool.c | 2 +-
2 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 1594177f86bb..872bf145ddde 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -106,11 +106,6 @@ struct kasan_cache {
int free_meta_offset;
};

-size_t __ksize(const void *);
-static inline void kasan_unpoison_slab(const void *ptr)
-{
- kasan_unpoison_range(ptr, __ksize(ptr));
-}
size_t kasan_metadata_size(struct kmem_cache *cache);

bool kasan_save_enable_multi_shot(void);
@@ -166,7 +161,6 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object,
return false;
}

-static inline void kasan_unpoison_slab(const void *ptr) { }

static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; }

#endif /* CONFIG_KASAN */

diff --git a/mm/mempool.c b/mm/mempool.c
index f473cdddaff0..583a9865b181 100644
--- a/mm/mempool.c
+++ b/mm/mempool.c
@@ -112,7 +112,7 @@ static __always_inline void kasan_poison_element(mempool_t *pool, void *element)
static void kasan_unpoison_element(mempool_t *pool, void *element)
{
if (pool->alloc == mempool_alloc_slab || pool->alloc == mempool_kmalloc)
- kasan_unpoison_slab(element);
+ kasan_unpoison_range(element, __ksize(element));
else if (pool->alloc == mempool_alloc_pages)
kasan_alloc_pages(element, (unsigned long)pool->pool_data);
}
--
2.29.2.299.gdc1121823c-goog

[Andrey Konovalov]

Using (un)poison_range() or check_invalid_free() currently results in
function calls. Move their definitions to mm/kasan/kasan.h and turn them
into static inline functions for hardware tag-based mode to avoid
unneeded function calls.

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/Ia9d8191024a12d1374675b3d27197f10193f50bb
---
mm/kasan/hw_tags.c | 30 ------------------------------
mm/kasan/kasan.h | 45 ++++++++++++++++++++++++++++++++++++++++-----
2 files changed, 40 insertions(+), 35 deletions(-)

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 3cdd87d189f6..863fed4edd3f 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -10,7 +10,6 @@

#include <linux/kasan.h>
#include <linux/kernel.h>
-#include <linux/kfence.h>
#include <linux/memory.h>
#include <linux/mm.h>
#include <linux/string.h>
@@ -31,35 +30,6 @@ void __init kasan_init_hw_tags(void)

pr_info("KernelAddressSanitizer initialized\n");
}

-void poison_range(const void *address, size_t size, u8 value)
-{
- /* Skip KFENCE memory if called explicitly outside of sl*b. */
- if (is_kfence_address(address))
- return;
-
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), value);
-}
-
-void unpoison_range(const void *address, size_t size)
-{
- /* Skip KFENCE memory if called explicitly outside of sl*b. */
- if (is_kfence_address(address))
- return;
-
- hw_set_mem_tag_range(kasan_reset_tag(address),
- round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
-}
-
-bool check_invalid_free(void *addr)
-{
- u8 ptr_tag = get_tag(addr);
- u8 mem_tag = hw_get_mem_tag(addr);
-
- return (mem_tag == KASAN_TAG_INVALID) ||
- (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);
-}
-
void kasan_set_free_info(struct kmem_cache *cache,
void *object, u8 tag)
{
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 7876a2547b7d..8aa83b7ad79e 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -3,6 +3,7 @@
#define __MM_KASAN_KASAN_H

#include <linux/kasan.h>
+#include <linux/kfence.h>
#include <linux/stackdepot.h>

#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
@@ -154,9 +155,6 @@ struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,

struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,

const void *object);

-void poison_range(const void *address, size_t size, u8 value);
-void unpoison_range(const void *address, size_t size);
-
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

static inline const void *kasan_shadow_to_mem(const void *shadow_addr)
@@ -196,8 +194,6 @@ void print_tags(u8 addr_tag, const void *addr);

static inline void print_tags(u8 addr_tag, const void *addr) { }

#endif

-bool check_invalid_free(void *addr);
-

void *find_first_bad_addr(void *addr, size_t size);

const char *get_bug_type(struct kasan_access_info *info);

void metadata_fetch_row(char *buffer, void *row);
@@ -278,6 +274,45 @@ static inline u8 random_tag(void) { return hw_get_random_tag(); }

static inline u8 random_tag(void) { return 0; }

#endif

+#ifdef CONFIG_KASAN_HW_TAGS
+
+static inline void poison_range(const void *address, size_t size, u8 value)
+{
+ /* Skip KFENCE memory if called explicitly outside of sl*b. */
+ if (is_kfence_address(address))
+ return;
+
+ hw_set_mem_tag_range(kasan_reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), value);
+}
+
+static inline void unpoison_range(const void *address, size_t size)
+{
+ /* Skip KFENCE memory if called explicitly outside of sl*b. */
+ if (is_kfence_address(address))
+ return;
+
+ hw_set_mem_tag_range(kasan_reset_tag(address),
+ round_up(size, KASAN_GRANULE_SIZE), get_tag(address));
+}
+
+static inline bool check_invalid_free(void *addr)
+{
+ u8 ptr_tag = get_tag(addr);
+ u8 mem_tag = hw_get_mem_tag(addr);
+
+ return (mem_tag == KASAN_TAG_INVALID) ||
+ (ptr_tag != KASAN_TAG_KERNEL && ptr_tag != mem_tag);

+}
+
+#else /* CONFIG_KASAN_HW_TAGS */
+

+void poison_range(const void *address, size_t size, u8 value);
+void unpoison_range(const void *address, size_t size);
+bool check_invalid_free(void *addr);

+
+#endif /* CONFIG_KASAN_HW_TAGS */

[Andrey Konovalov]

Hardware tag-based KASAN mode is intended to eventually be used in

production as a security mitigation. Therefore there's a need for finer
control over KASAN features and for an existence of a kill switch.

This change adds a few boot parameters for hardware tag-based KASAN that
allow to disable or otherwise control particular KASAN features.

The features that can be controlled are:

1. Whether KASAN is enabled at all.
2. Whether KASAN collects and saves alloc/free stacks.
3. Whether KASAN panics on a detected bug or not.

With this change a new boot parameter kasan.mode allows to choose one of

three main modes:

- kasan.mode=off - KASAN is disabled, no tag checks are performed
- kasan.mode=prod - only essential production features are enabled
- kasan.mode=full - all KASAN features are enabled

The chosen mode provides default control values for the features mentioned
above. However it's also possible to override the default values by
providing:

- kasan.stacktrace=off/on - enable alloc/free stack collection

(default: on for mode=full, otherwise off)
- kasan.fault=report/panic - only report tag fault or also panic
(default: report)

If kasan.mode parameter is not provided, it defaults to full when
CONFIG_DEBUG_KERNEL is enabled, and to prod otherwise.

It is essential that switching between these modes doesn't require
rebuilding the kernel with different configs, as this is required by

the Android GKI (Generic Kernel Image) initiative [1].

[1] https://source.android.com/devices/architecture/kernel/generic-kernel-image

Signed-off-by: Andrey Konovalov <andre...@google.com>
Link: https://linux-review.googlesource.com/id/If7d37003875b2ed3e0935702c8015c223d6416a4
---
mm/kasan/common.c | 22 +++++--
mm/kasan/hw_tags.c | 151 +++++++++++++++++++++++++++++++++++++++++++++
mm/kasan/kasan.h | 16 +++++
mm/kasan/report.c | 14 ++++-
4 files changed, 196 insertions(+), 7 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 1ac4f435c679..a11e3e75eb08 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -135,6 +135,11 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
unsigned int redzone_size;
int redzone_adjust;

+ if (!kasan_stack_collection_enabled()) {
+ *flags |= SLAB_KASAN;
+ return;
+ }
+
/* Add alloc meta. */
cache->kasan_info.alloc_meta_offset = *size;
*size += sizeof(struct kasan_alloc_meta);
@@ -171,6 +176,8 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,

size_t kasan_metadata_size(struct kmem_cache *cache)
{
+ if (!kasan_stack_collection_enabled())
+ return 0;
return (cache->kasan_info.alloc_meta_offset ?
sizeof(struct kasan_alloc_meta) : 0) +
(cache->kasan_info.free_meta_offset ?
@@ -263,11 +270,13 @@ void * __must_check kasan_init_slab_obj(struct kmem_cache *cache,
{
struct kasan_alloc_meta *alloc_meta;

- if (!(cache->flags & SLAB_KASAN))
- return (void *)object;
+ if (kasan_stack_collection_enabled()) {
+ if (!(cache->flags & SLAB_KASAN))
+ return (void *)object;

- alloc_meta = kasan_get_alloc_meta(cache, object);
- __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ alloc_meta = kasan_get_alloc_meta(cache, object);
+ __memset(alloc_meta, 0, sizeof(*alloc_meta));

+ }

if (IS_ENABLED(CONFIG_KASAN_SW_TAGS) || IS_ENABLED(CONFIG_KASAN_HW_TAGS))
object = set_tag(object, assign_tag(cache, object, true, false));

@@ -307,6 +316,9 @@ static bool __kasan_slab_free(struct kmem_cache *cache, void *object,
rounded_up_size = round_up(cache->object_size, KASAN_GRANULE_SIZE);
poison_range(object, rounded_up_size, KASAN_KMALLOC_FREE);

+ if (!kasan_stack_collection_enabled())
+ return false;
+
if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine) ||
unlikely(!(cache->flags & SLAB_KASAN)))
return false;
@@ -357,7 +369,7 @@ static void *__kasan_kmalloc(struct kmem_cache *cache, const void *object,
poison_range((void *)redzone_start, redzone_end - redzone_start,
KASAN_KMALLOC_REDZONE);

- if (cache->flags & SLAB_KASAN)
+ if (kasan_stack_collection_enabled() && (cache->flags & SLAB_KASAN))

set_alloc_info(cache, (void *)object, flags);

return set_tag(object, tag);

diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 863fed4edd3f..30ce88935e9d 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -8,18 +8,115 @@

#define pr_fmt(fmt) "kasan: " fmt

+#include <linux/init.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
#include <linux/memory.h>
#include <linux/mm.h>
+#include <linux/static_key.h>
#include <linux/string.h>
#include <linux/types.h>

#include "kasan.h"

+enum kasan_arg_mode {
+ KASAN_ARG_MODE_DEFAULT,
+ KASAN_ARG_MODE_OFF,
+ KASAN_ARG_MODE_PROD,
+ KASAN_ARG_MODE_FULL,
+};
+
+enum kasan_arg_stacktrace {
+ KASAN_ARG_STACKTRACE_DEFAULT,
+ KASAN_ARG_STACKTRACE_OFF,
+ KASAN_ARG_STACKTRACE_ON,
+};
+
+enum kasan_arg_fault {
+ KASAN_ARG_FAULT_DEFAULT,
+ KASAN_ARG_FAULT_REPORT,
+ KASAN_ARG_FAULT_PANIC,
+};
+
+static enum kasan_arg_mode kasan_arg_mode __ro_after_init;
+static enum kasan_arg_stacktrace kasan_arg_stacktrace __ro_after_init;
+static enum kasan_arg_fault kasan_arg_fault __ro_after_init;
+
+/* Whether KASAN is enabled at all. */
+DEFINE_STATIC_KEY_FALSE_RO(kasan_flag_enabled);
+EXPORT_SYMBOL(kasan_flag_enabled);
+
+/* Whether to collect alloc/free stack traces. */
+DEFINE_STATIC_KEY_FALSE_RO(kasan_flag_stacktrace);
+
+/* Whether panic or disable tag checking on fault. */
+bool kasan_flag_panic __ro_after_init;
+

+early_param("kasan.fault", early_kasan_fault);
+
/* kasan_init_hw_tags_cpu() is called for each CPU. */
void kasan_init_hw_tags_cpu(void)
{
+ /*
+ * There's no need to check that the hardware is MTE-capable here,
+ * as this function is only called for MTE-capable hardware.
+ */
+
+ /* If KASAN is disabled, do nothing. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_OFF)
+ return;
+
hw_init_tags(KASAN_TAG_MAX);
hw_enable_tagging();
}
@@ -27,6 +124,60 @@ void kasan_init_hw_tags_cpu(void)
/* kasan_init_hw_tags() is called once on boot CPU. */
void __init kasan_init_hw_tags(void)
{
+ /* If hardware doesn't support MTE, do nothing. */
+ if (!system_supports_mte())
+ return;
+
+ /* Choose KASAN mode if kasan boot parameter is not provided. */
+ if (kasan_arg_mode == KASAN_ARG_MODE_DEFAULT) {
+ if (IS_ENABLED(CONFIG_DEBUG_KERNEL))

+ kasan_arg_mode = KASAN_ARG_MODE_FULL;
+ else

+ kasan_arg_mode = KASAN_ARG_MODE_PROD;
+ }
+
+ /* Preset parameter values based on the mode. */
+ switch (kasan_arg_mode) {
+ case KASAN_ARG_MODE_DEFAULT:
+ /* Shouldn't happen as per the check above. */
+ WARN_ON(1);
+ return;
+ case KASAN_ARG_MODE_OFF:
+ /* If KASAN is disabled, do nothing. */
+ return;
+ case KASAN_ARG_MODE_PROD:
+ static_branch_enable(&kasan_flag_enabled);
+ break;
+ case KASAN_ARG_MODE_FULL:
+ static_branch_enable(&kasan_flag_enabled);
+ static_branch_enable(&kasan_flag_stacktrace);
+ break;
+ }
+
+ /* Now, optionally override the presets. */
+
+ switch (kasan_arg_stacktrace) {
+ case KASAN_ARG_STACKTRACE_DEFAULT:
+ break;
+ case KASAN_ARG_STACKTRACE_OFF:
+ static_branch_disable(&kasan_flag_stacktrace);
+ break;
+ case KASAN_ARG_STACKTRACE_ON:
+ static_branch_enable(&kasan_flag_stacktrace);
+ break;
+ }
+
+ switch (kasan_arg_fault) {
+ case KASAN_ARG_FAULT_DEFAULT:
+ break;
+ case KASAN_ARG_FAULT_REPORT:
+ kasan_flag_panic = false;
+ break;
+ case KASAN_ARG_FAULT_PANIC:
+ kasan_flag_panic = true;
+ break;
+ }
+

pr_info("KernelAddressSanitizer initialized\n");
}

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 8aa83b7ad79e..d01a5ac34f70 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -6,6 +6,22 @@
#include <linux/kfence.h>
#include <linux/stackdepot.h>

+#ifdef CONFIG_KASAN_HW_TAGS
+#include <linux/static_key.h>
+DECLARE_STATIC_KEY_FALSE(kasan_flag_stacktrace);
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return static_branch_unlikely(&kasan_flag_stacktrace);
+}
+#else
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return true;
+}
+#endif
+
+extern bool kasan_flag_panic __ro_after_init;
+
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
#define KASAN_GRANULE_SIZE (1UL << KASAN_SHADOW_SCALE_SHIFT)
#else
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 76a0e3ae2049..ffa6076b1710 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -99,6 +99,10 @@ static void end_report(unsigned long *flags)
panic_on_warn = 0;
panic("panic_on_warn set ...\n");
}
+#ifdef CONFIG_KASAN_HW_TAGS
+ if (kasan_flag_panic)
+ panic("kasan.fault=panic set ...\n");
+#endif
kasan_enable_current();
}

@@ -161,8 +165,8 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,
(void *)(object_addr + cache->object_size));
}

-static void describe_object(struct kmem_cache *cache, void *object,
- const void *addr, u8 tag)
+static void describe_object_stacks(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
{

struct kasan_alloc_meta *alloc_meta = kasan_get_alloc_meta(cache, object);

@@ -190,7 +194,13 @@ static void describe_object(struct kmem_cache *cache, void *object,
}
#endif
}
+}

+static void describe_object(struct kmem_cache *cache, void *object,
+ const void *addr, u8 tag)
+{
+ if (kasan_stack_collection_enabled())
+ describe_object_stacks(cache, object, addr, tag);
describe_object_addr(cache, object, addr);
}

--
2.29.2.299.gdc1121823c-goog