[PATCH RFC v3 00/12] KCOV: entry/exit records, memory access records, and delay injection

[Jann Horn]

This series consists of three parts that add new KCOV features.
In short:

- Part 1: Adds function entry/exit records
- Part 2: Adds information about memory accesses (instruction address,
data address, access type, data value, timing)
- Part 3: Adds an API for delay injection (spin-waiting at a specific
point on one thread until a specific event happens on another
thread)

Together, these make it possible to build:

- tooling to force specific ordering of parallel execution (entry/exit
records provide stable identifiers for recorded memory access events
that can then be targeted for delay injection)
- tooling to force a specific ordering on a subset of events ("A should
happen before B"), where other parts of the testcase are left
executing in parallel
- visualization of recorded execution traces of small samples for
manual review (though the large amount of data makes recording the
execution of larger programs prohibitively expensive); this is the
only usecase I have for recording data values, and the main usecase
for recording timing information

At a higher level, I think this could be useful for the following use
cases:

- manual testing of possible bugs (especially race condition bugs)
- unit tests for race conditions
- fuzzing for race conditions
- maybe also for other fuzzing (automatically discovering how syscalls
interact)
- maybe also for providing more clues for analyzing normal fuzzer
crashes

=== Part 1: function entry/exit records ===
This series adds a KCOV feature that userspace can use to keep track of
the current call stack. When userspace enables the new mode
KCOV_TRACE_PC_EXT, collected instruction addresses are tagged with one
of three types:

- function entry
- non-entry basic block
- function exit

This requires corresponding LLVM support, which was added in LLVM commit
https://github.com/llvm/llvm-project/commit/dc5c6d008f487eea8f5d646011f9b3dca6caebd7
a few months ago; I believe this will be part of LLVM 23.

A simple example of how to use KCOV_TRACE_PC_EXT:
```
user@vm:~/kcov/u$ cat kcov-u.c

typeof(x) __res = (x); \
if (__res == (typeof(x))-1) \
err(1, "SYSCHK(" #x ")"); \
__res; \
})

static void indent(int depth) {
for (int i=0; i<depth; i++)
printf(" ");
}

int main(void) {
int fd = SYSCHK(open("/sys/kernel/debug/kcov", O_RDWR));
SYSCHK(ioctl(fd, KCOV_INIT_TRACE, COVER_SIZE));
unsigned long *cover = (unsigned long*)SYSCHK(
mmap(NULL, COVER_SIZE * sizeof(unsigned long), PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0));
SYSCHK(ioctl(fd, KCOV_ENABLE, KCOV_TRACE_PC_EXT));
usleep(1000); // fault in stuff
__atomic_store_n(&cover[0], 0, __ATOMIC_RELAXED); // start recording
usleep(1000);
unsigned long cover_num = __atomic_load_n(&cover[0], __ATOMIC_RELAXED); // end

int depth = 0;
for (unsigned long i = 0; i < cover_num; i++) {
unsigned long record = cover[1+i];
unsigned long pc = record | ~KCOV_RECORD_IP_MASK;
switch (record & KCOV_RECORDFLAG_TYPEMASK) {
case KCOV_RECORDFLAG_TYPE_NORMAL:
indent(depth);
printf("BB 0x%lx\n", pc);
break;
case KCOV_RECORDFLAG_TYPE_ENTRY:
indent(depth);
printf("ENTER 0x%lx\n", pc);
depth++;
break;
case KCOV_RECORDFLAG_TYPE_EXIT:
if (depth == 0)
errx(1, "exit at depth 0");
depth--;
indent(depth);
printf("EXIT 0x%lx\n", pc);
break;
default: errx(1, "unknown record type in 0x%016lx", record);
}
}
}
user@vm:~/kcov/u$ cat symbolize.py
import sys
syms = []
with open('/proc/kallsyms') as f:
for line in f:
parts = line.strip().split(' ')
if len(parts) < 3:
continue
syms.append((int(parts[0], 16), parts[2]))

for line in sys.stdin:
parts = line.rstrip().split('0x')
if len(parts) != 2:
continue
record_pc = int(parts[1], 16)
for i in range(0, len(syms)-1):
if syms[i+1][0] > record_pc:
print(parts[0] + syms[i][1] + '+' + hex(record_pc - syms[i][0]))
break
user@vm:~/kcov/u$ gcc -o kcov-u kcov-u.c -Wall
user@vm:~/kcov/u$ sudo ./kcov-u | sudo ./symbolize.py
ENTER __audit_syscall_entry+0x2c
BB __audit_syscall_entry+0xa4
BB __audit_syscall_entry+0xd2
BB __audit_syscall_entry+0x1ab
ENTER ktime_get_coarse_real_ts64+0x1a
BB ktime_get_coarse_real_ts64+0x3f
BB ktime_get_coarse_real_ts64+0x96
EXIT ktime_get_coarse_real_ts64+0x9b
EXIT __audit_syscall_entry+0x12b
ENTER __x64_sys_clock_nanosleep+0x18
ENTER __se_sys_clock_nanosleep+0x33
BB __se_sys_clock_nanosleep+0x10e
ENTER get_timespec64+0x29
ENTER _copy_from_user+0x17
BB _copy_from_user+0x5d
EXIT _copy_from_user+0x62
BB get_timespec64+0xaf
EXIT get_timespec64+0xd5
BB __se_sys_clock_nanosleep+0x1c0
ENTER common_nsleep+0x1f
ENTER hrtimer_nanosleep+0x2f
ENTER hrtimer_setup_sleeper_on_stack+0x20
BB hrtimer_setup_sleeper_on_stack+0x2a
BB hrtimer_setup_sleeper_on_stack+0x7e
EXIT hrtimer_setup_sleeper_on_stack+0x14c
ENTER do_nanosleep+0x2d
BB do_nanosleep+0x3b
ENTER hrtimer_start_range_ns+0x28
BB hrtimer_start_range_ns+0x67
ENTER remove_hrtimer+0x22
BB remove_hrtimer+0x4b
EXIT remove_hrtimer+0x1ea
BB hrtimer_start_range_ns+0x173
ENTER __hrtimer_cb_get_time+0x11
BB __hrtimer_cb_get_time+0x32
ENTER ktime_get+0x17
BB ktime_get+0x33
BB ktime_get+0x58
ENTER kvm_clock_get_cycles+0xc
BB kvm_clock_get_cycles+0x48
EXIT kvm_clock_get_cycles+0x4d
BB ktime_get+0xb7
BB ktime_get+0x149
EXIT ktime_get+0x151
EXIT __hrtimer_cb_get_time+0x84
BB hrtimer_start_range_ns+0x3bc
BB hrtimer_start_range_ns+0x5a4
ENTER enqueue_hrtimer+0x20
BB enqueue_hrtimer+0x2a
BB enqueue_hrtimer+0x5b
ENTER timerqueue_add+0x1c
BB timerqueue_add+0x41
BB timerqueue_add+0xb2
BB timerqueue_add+0xb2
BB timerqueue_add+0xf9
EXIT timerqueue_add+0x150
EXIT enqueue_hrtimer+0xaf
BB hrtimer_start_range_ns+0x714
ENTER hrtimer_reprogram+0x1b
BB hrtimer_reprogram+0x65
BB hrtimer_reprogram+0x13a
BB hrtimer_reprogram+0x1dc
ENTER tick_program_event+0x25
BB tick_program_event+0x65
ENTER clockevents_program_event+0x20
BB clockevents_program_event+0x7e
ENTER ktime_get+0x17
BB ktime_get+0x33
BB ktime_get+0x58
ENTER kvm_clock_get_cycles+0xc
BB kvm_clock_get_cycles+0x48
EXIT kvm_clock_get_cycles+0x4d
BB ktime_get+0xb7
BB ktime_get+0x149
EXIT ktime_get+0x151
BB clockevents_program_event+0x219
EXIT clockevents_program_event+0x22b
EXIT tick_program_event+0x89
EXIT hrtimer_reprogram+0x211
EXIT hrtimer_start_range_ns+0x74d
BB do_nanosleep+0x9c
ENTER sched_clock+0xc
BB sched_clock+0x40
EXIT sched_clock+0x45
ENTER arch_scale_cpu_capacity+0x13
BB arch_scale_cpu_capacity+0x1a
EXIT arch_scale_cpu_capacity+0x24
ENTER __cgroup_account_cputime+0x1b
ENTER css_rstat_updated+0x2c
BB css_rstat_updated+0x77
BB css_rstat_updated+0xbe
EXIT css_rstat_updated+0x1bc
BB __cgroup_account_cputime+0x81
EXIT __cgroup_account_cputime+0x86
ENTER sched_clock+0xc
BB sched_clock+0x40
EXIT sched_clock+0x45
ENTER sched_clock+0xc
BB sched_clock+0x40
EXIT sched_clock+0x45
ENTER __msecs_to_jiffies+0x13
BB __msecs_to_jiffies+0x25
EXIT __msecs_to_jiffies+0x4c
ENTER prandom_u32_state+0x15
EXIT prandom_u32_state+0xbe
ENTER hrtimer_try_to_cancel+0x1e
BB hrtimer_try_to_cancel+0x6a
BB hrtimer_try_to_cancel+0x1da
EXIT hrtimer_try_to_cancel+0x1be
BB do_nanosleep+0xbf
BB do_nanosleep+0x166
BB do_nanosleep+0x177
BB do_nanosleep+0x275
EXIT do_nanosleep+0x2d5
BB hrtimer_nanosleep+0x182
EXIT hrtimer_nanosleep+0x194
EXIT common_nsleep+0x77
EXIT __se_sys_clock_nanosleep+0x15d
EXIT __x64_sys_clock_nanosleep+0x62
ENTER __audit_syscall_exit+0x1d
BB __audit_syscall_exit+0x5c
ENTER audit_reset_context+0x1e
BB audit_reset_context+0x52
EXIT audit_reset_context+0x5f6
EXIT __audit_syscall_exit+0x168
ENTER fpregs_assert_state_consistent+0x11
BB fpregs_assert_state_consistent+0x48
BB fpregs_assert_state_consistent+0xa6
EXIT fpregs_assert_state_consistent+0xcc
ENTER switch_fpu_return+0xe
ENTER fpregs_restore_userregs+0x12
BB fpregs_restore_userregs+0x4c
BB fpregs_restore_userregs+0xb8
EXIT fpregs_restore_userregs+0x107
EXIT switch_fpu_return+0x18
```

=== part 2: memory access records (CONFIG_KCOV_MEMORY) ===
A new mode KCOV mode KCOV_TRACE_MEMORY_ACCESS generates the same
records as KCOV_TRACE_PC_EXT, but additionally generates records of
type "struct memory_access_record" when a memory access happens.
Information about memory accesses is obtained in two ways:

- from instrument_*() hooks
- from KASAN hooks in generic outline mode

The memory_access_record records in multiple traces can be analyzed
together to discover which memory regions could be relevant for
concurrency bugs.

=== part 3: delay injection ===
A new set of KCOV ioctls can be used to inject spin-waits at specific
points in the execution, identified by call stacks obtained from
KCOV_TRACE_MEMORY_ACCESS.

The main ioctl for configuring this feature for a KCOV instance is
KCOV_SET_DI, which essentially takes a list of call stacks, each
associated with an action, which is one of:

- wait on bit N in a kcov state bitmap before this access
- wake up bit N in a kcov state bitmap before this access
- wake up bit N in a kcov state bitmap after this access

=== userspace users ===
I have written two userspace programs that use this API:
A GUI which lets you interactively experiment with execution orderings
of parallel execution, and a testing harness that can exercise ~all
A-B-A orderings of a given testcase automatically.

See:
https://github.com/googleprojectzero/MAccConc

Signed-off-by: Jann Horn <ja...@google.com>
---
Changes in v3:
- in part 1: remove sched hack and replace it with
"kcov: summarize entry/exit while disabled" (suggested by peterz)
- add memory access records and delay injection
- Link to v2: https://lore.kernel.org/r/20260318-kcov-extrec...@google.com

Changes in v2:
- patch 2: change commit message (dvyukov)
- patch 2: add __always_inline (dvyukov)
- patch 2: add comment in __sanitizer_cov_trace_pc_entry
- replaced patch 3 with patches 3+4
- store extended record format flag as part of kcov_mode (dvyukov)
- clarify comment in __sanitizer_cov_trace_pc_exit (dvyukov)
- Link to v1: https://lore.kernel.org/r/20260311-kcov-extrec...@google.com

---
Jann Horn (12):
kcov: wire up compiler instrumentation for CONFIG_KCOV_EXT_RECORDS
kcov: refactor mode check out of check_kcov_mode()
kcov: introduce extended PC coverage collection mode
kcov: summarize entry/exit while disabled
kasan: refactor write/is_write arguments to flags
kcov: introduce memory access tracing
kasan: provide memory access information to KCOV
kcov: log freeing of SLUB objects and pages
kcov: record return address on function entry
kcov: log old value
kcov: introduce delay injection
Documentation/kcov: add documentation for EXT_RECORDS and KCOV_MEMORY

Documentation/dev-tools/kcov.rst | 70 +++++
arch/arm64/kernel/traps.c | 2 +-
arch/arm64/mm/fault.c | 2 +-
include/linux/instrumented.h | 30 ++
include/linux/kasan.h | 7 +-
include/linux/kcov.h | 31 +-
include/uapi/linux/kcov.h | 86 ++++++
kernel/kcov.c | 618 +++++++++++++++++++++++++++++++++++++--
lib/Kconfig.debug | 27 ++
lib/Kconfig.kasan | 9 +
mm/kasan/common.c | 4 +-
mm/kasan/generic.c | 35 ++-
mm/kasan/kasan.h | 6 +-
mm/kasan/report.c | 3 +-
mm/kasan/report_generic.c | 8 +-
mm/kasan/shadow.c | 25 +-
mm/kasan/sw_tags.c | 20 +-
mm/page_alloc.c | 3 +
scripts/Makefile.kasan | 17 ++
scripts/Makefile.kcov | 2 +
tools/objtool/check.c | 4 +
21 files changed, 931 insertions(+), 78 deletions(-)
---
base-commit: 73ae59e975966d24e32926247ddb45a537ebe184
change-id: 20260311-kcov-extrecord-6e0d9a2b0a8c

Best regards,
--
Jann Horn <ja...@google.com>

[Jann Horn]

This is the first half of CONFIG_KCOV_EXT_RECORDS.

Set the appropriate compiler flags to call separate hooks for function
entry/exit, and provide these hooks, but don't make it visible in the KCOV
UAPI yet.

With -fsanitize-coverage=trace-pc-entry-exit, the compiler behavior changes
as follows:

- The __sanitizer_cov_trace_pc() call on function entry is replaced with a
call to __sanitizer_cov_trace_pc_entry(); so for now,
__sanitizer_cov_trace_pc_entry() must be treated the same way as
__sanitizer_cov_trace_pc().
- On function exit, an extra call to __sanitizer_cov_trace_pc_exit()
happens; since function exit produced no coverage in the old UAPI,
__sanitizer_cov_trace_pc_exit() should do nothing for now.

This feature was added to LLVM in commit:
https://github.com/llvm/llvm-project/commit/dc5c6d008f487eea8f5d646011f9b3dca6caebd7

Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Signed-off-by: Jann Horn <ja...@google.com>
---

include/linux/kcov.h | 2 ++
kernel/kcov.c | 34 +++++++++++++++++++++++++++-------
lib/Kconfig.debug | 12 ++++++++++++
scripts/Makefile.kcov | 2 ++
tools/objtool/check.c | 2 ++
5 files changed, 45 insertions(+), 7 deletions(-)

diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index 895b761b2db1..cd79715db241 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -79,6 +79,8 @@ typedef unsigned long long kcov_u64;
#endif

void __sanitizer_cov_trace_pc(void);
+void __sanitizer_cov_trace_pc_entry(void);
+void __sanitizer_cov_trace_pc_exit(void);
void __sanitizer_cov_trace_cmp1(u8 arg1, u8 arg2);
void __sanitizer_cov_trace_cmp2(u16 arg1, u16 arg2);
void __sanitizer_cov_trace_cmp4(u32 arg1, u32 arg2);
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 35420f0ac524..5d9686c8b3ec 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -198,15 +198,10 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
return ip;
}

-/*
- * Entry point from instrumented code.
- * This is called once per basic-block/edge.
- */
-void notrace __sanitizer_cov_trace_pc(void)
+static __always_inline void notrace kcov_add_pc_record(unsigned long record)
{
struct task_struct *t;
unsigned long *area;
- unsigned long ip = canonicalize_ip(_RET_IP_);
unsigned long pos;

t = current;
@@ -226,11 +221,36 @@ void notrace __sanitizer_cov_trace_pc(void)
*/
WRITE_ONCE(area[0], pos);
barrier();
- area[pos] = ip;
+ area[pos] = record;
}
}
+
+/*
+ * Entry point from instrumented code.
+ * This is called once per basic-block/edge.
+ */
+void notrace __sanitizer_cov_trace_pc(void)
+{
+ kcov_add_pc_record(canonicalize_ip(_RET_IP_));
+}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);

+#ifdef CONFIG_KCOV_EXT_RECORDS
+void notrace __sanitizer_cov_trace_pc_entry(void)
+{
+ unsigned long record = canonicalize_ip(_RET_IP_);
+
+ /*
+ * This hook replaces __sanitizer_cov_trace_pc() for the function entry
+ * basic block; it should still emit a record even in classic kcov mode.
+ */
+ kcov_add_pc_record(record);
+}
+void notrace __sanitizer_cov_trace_pc_exit(void)
+{
+}
+#endif
+
#ifdef CONFIG_KCOV_ENABLE_COMPARISONS
static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
{
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a2f0d3e97889..6ddf58692b09 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2195,6 +2195,18 @@ config KCOV

For more details, see Documentation/dev-tools/kcov.rst.

+config KCOV_EXT_RECORDS
+ bool "Support extended KCOV records with function entry/exit records"
+ depends on KCOV
+ depends on 64BIT
+ depends on $(cc-option,-fsanitize-coverage=trace-pc-entry-exit)
+ help
+ Extended KCOV records allow distinguishing between multiple types of
+ records: Normal edge coverage, function entry, and function exit.
+
+ This will likely cause a small additional slowdown compared to normal
+ KCOV.
+
config KCOV_ENABLE_COMPARISONS
bool "Enable comparison operands collection by KCOV"
depends on KCOV
diff --git a/scripts/Makefile.kcov b/scripts/Makefile.kcov
index 78305a84ba9d..aa0be904268f 100644
--- a/scripts/Makefile.kcov
+++ b/scripts/Makefile.kcov
@@ -1,10 +1,12 @@
# SPDX-License-Identifier: GPL-2.0-only
kcov-flags-y += -fsanitize-coverage=trace-pc
+kcov-flags-$(CONFIG_KCOV_EXT_RECORDS) += -fsanitize-coverage=trace-pc-entry-exit
kcov-flags-$(CONFIG_KCOV_ENABLE_COMPARISONS) += -fsanitize-coverage=trace-cmp

kcov-rflags-y += -Cpasses=sancov-module
kcov-rflags-y += -Cllvm-args=-sanitizer-coverage-level=3
kcov-rflags-y += -Cllvm-args=-sanitizer-coverage-trace-pc
+kcov-rflags-$(CONFIG_KCOV_EXT_RECORDS) += -Cllvm-args=-sanitizer-coverage-trace-pc-entry-exit
kcov-rflags-$(CONFIG_KCOV_ENABLE_COMPARISONS) += -Cllvm-args=-sanitizer-coverage-trace-compares

export CFLAGS_KCOV := $(kcov-flags-y)
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index df04e6be2f66..d70cb640e2ec 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1220,6 +1220,8 @@ static const char *uaccess_safe_builtin[] = {
"write_comp_data",
"check_kcov_mode",
"__sanitizer_cov_trace_pc",
+ "__sanitizer_cov_trace_pc_entry",
+ "__sanitizer_cov_trace_pc_exit",
"__sanitizer_cov_trace_const_cmp1",
"__sanitizer_cov_trace_const_cmp2",
"__sanitizer_cov_trace_const_cmp4",

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

The following patch will need to check t->kcov_mode in different ways at
different check_kcov_mode() call sites. In preparation for that, move the
mode check up the call hierarchy.

Signed-off-by: Jann Horn <ja...@google.com>
---

kernel/kcov.c | 31 +++++++++++++++++--------------
1 file changed, 17 insertions(+), 14 deletions(-)

diff --git a/kernel/kcov.c b/kernel/kcov.c
index 5d9686c8b3ec..26baccaaefa9 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -167,10 +167,8 @@ static __always_inline bool in_softirq_really(void)
return in_serving_softirq() && !in_hardirq() && !in_nmi();
}

-static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_struct *t)
+static notrace bool check_kcov_context(struct task_struct *t)
{
- unsigned int mode;
-
/*
* We are interested in code coverage as a function of a syscall inputs,
* so we ignore code executed in interrupts, unless we are in a remote
@@ -178,7 +176,6 @@ static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_stru
*/
if (!in_task() && !(in_softirq_really() && t->kcov_softirq))
return false;
- mode = READ_ONCE(t->kcov_mode);
/*
* There is some code that runs in interrupts but for which
* in_interrupt() returns false (e.g. preempt_schedule_irq()).
@@ -187,7 +184,7 @@ static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_stru
* kcov_start().
*/
barrier();
- return mode == needed_mode;
+ return true;

}

static notrace unsigned long canonicalize_ip(unsigned long ip)

@@ -198,14 +195,12 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
return ip;
}

-static __always_inline void notrace kcov_add_pc_record(unsigned long record)
+static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, unsigned long record)
{
- struct task_struct *t;
unsigned long *area;
unsigned long pos;

- t = current;
- if (!check_kcov_mode(KCOV_MODE_TRACE_PC, t))
+ if (!check_kcov_context(t))
return;

area = t->kcov_area;
@@ -213,7 +208,7 @@ static __always_inline void notrace kcov_add_pc_record(unsigned long record)
pos = READ_ONCE(area[0]) + 1;
if (likely(pos < t->kcov_size)) {
/* Previously we write pc before updating pos. However, some
- * early interrupt code could bypass check_kcov_mode() check
+ * early interrupt code could bypass check_kcov_context() check
* and invoke __sanitizer_cov_trace_pc(). If such interrupt is
* raised between writing pc and updating pos, the pc could be
* overitten by the recursive __sanitizer_cov_trace_pc().
@@ -231,20 +226,28 @@ static __always_inline void notrace kcov_add_pc_record(unsigned long record)
*/
void notrace __sanitizer_cov_trace_pc(void)
{
- kcov_add_pc_record(canonicalize_ip(_RET_IP_));
+ struct task_struct *cur = current;
+
+ if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC)
+ return;
+ kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_));
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);

#ifdef CONFIG_KCOV_EXT_RECORDS
void notrace __sanitizer_cov_trace_pc_entry(void)
{
+ struct task_struct *cur = current;

unsigned long record = canonicalize_ip(_RET_IP_);

+ unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);

/*

* This hook replaces __sanitizer_cov_trace_pc() for the function entry

* basic block; it should still emit a record even in classic kcov mode.

*/
- kcov_add_pc_record(record);
+ if (kcov_mode != KCOV_MODE_TRACE_PC)
+ return;
+ kcov_add_pc_record(cur, record);
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{
@@ -259,7 +262,7 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
u64 count, start_index, end_pos, max_pos;

t = current;
- if (!check_kcov_mode(KCOV_MODE_TRACE_CMP, t))
+ if (READ_ONCE(t->kcov_mode) != KCOV_MODE_TRACE_CMP || !check_kcov_context(t))
return;

ip = canonicalize_ip(ip);
@@ -379,7 +382,7 @@ static void kcov_start(struct task_struct *t, struct kcov *kcov,
t->kcov_size = size;
t->kcov_area = area;
t->kcov_sequence = sequence;
- /* See comment in check_kcov_mode(). */
+ /* See comment in check_kcov_context(). */
barrier();
WRITE_ONCE(t->kcov_mode, mode);
}

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

This is the second half of CONFIG_KCOV_EXT_RECORDS.

Introduce a new KCOV mode KCOV_TRACE_PC_EXT which replaces the upper 8 bits
of recorded instruction pointers with metadata. For now, userspace can use
this metadata to distinguish three types of records:

- function entry
- function exit
- normal basic block inside the function

Internally, this new mode is represented as a variant of
KCOV_MODE_TRACE_PC, distinguished with the flag KCOV_EXT_FORMAT.
Store this flag as part of the mode in task_struct::kcov_mode and in
kcov::mode to avoid having to pass it around separately everywhere.

Signed-off-by: Jann Horn <ja...@google.com>
---

include/linux/kcov.h | 7 +++++++
include/uapi/linux/kcov.h | 12 ++++++++++++
kernel/kcov.c | 39 ++++++++++++++++++++++++++++++++++-----
3 files changed, 53 insertions(+), 5 deletions(-)

diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index cd79715db241..6c9f0373022f 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -23,8 +23,15 @@ enum kcov_mode {
KCOV_MODE_TRACE_CMP = 3,
};

+/*
+ * Modifier for KCOV_MODE_TRACE_PC to record function entry/exit marked with
+ * metadata bits.
+ */
+#define KCOV_EXT_FORMAT (1 << 29)
#define KCOV_IN_CTXSW (1 << 30)

+#define KCOV_MODE_TRACE_PC_EXT (KCOV_MODE_TRACE_PC | KCOV_EXT_FORMAT)
+
void kcov_task_init(struct task_struct *t);
void kcov_task_exit(struct task_struct *t);

diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index ed95dba9fa37..8d8a233bd61f 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -35,8 +35,20 @@ enum {
KCOV_TRACE_PC = 0,
/* Collecting comparison operands mode. */
KCOV_TRACE_CMP = 1,
+ /*
+ * Extended PC coverage collection mode.
+ * In this mode, the top byte of the PC is replaced with flag bits
+ * (KCOV_RECORDFLAG_*).
+ */
+ KCOV_TRACE_PC_EXT = 2,
};

+#define KCOV_RECORD_IP_MASK 0x00ffffffffffffff
+#define KCOV_RECORDFLAG_TYPEMASK 0xf000000000000000
+#define KCOV_RECORDFLAG_TYPE_NORMAL 0xf000000000000000
+#define KCOV_RECORDFLAG_TYPE_ENTRY 0x0000000000000000
+#define KCOV_RECORDFLAG_TYPE_EXIT 0x1000000000000000
+
/*
* The format for the types of collected comparisons.
*
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 26baccaaefa9..701ad69493bf 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -55,7 +55,12 @@ struct kcov {
refcount_t refcount;
/* The lock protects mode, size, area and t. */
spinlock_t lock;
- enum kcov_mode mode __guarded_by(&lock);
+ /*
+ * Mode, consists of:
+ * - enum kcov_mode
+ * - flag KCOV_EXT_FORMAT
+ */
+ unsigned int mode __guarded_by(&lock);
/* Size of arena (in long's). */
unsigned int size __guarded_by(&lock);
/* Coverage buffer shared with user space. */
@@ -228,8 +233,14 @@ void notrace __sanitizer_cov_trace_pc(void)
{

struct task_struct *cur = current;

- if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC)
+ if ((READ_ONCE(cur->kcov_mode) & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
return;
+ /*
+ * No bitops are needed here for setting the record type because
+ * KCOV_RECORDFLAG_TYPE_NORMAL has the high bits set.
+ * This relies on userspace not caring about the rest of the top byte
+ * for KCOV_RECORDFLAG_TYPE_NORMAL records.
+ */
kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_));
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
@@ -245,12 +256,28 @@ void notrace __sanitizer_cov_trace_pc_entry(void)

* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/

- if (kcov_mode != KCOV_MODE_TRACE_PC)
+ if ((kcov_mode & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
return;
+ if ((kcov_mode & KCOV_EXT_FORMAT) != 0)
+ record = (record & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_ENTRY;

kcov_add_pc_record(cur, record);
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{

+ struct task_struct *cur = current;

+ unsigned long record;
+
+ /*
+ * This hook is not called at the beginning of a basic block; the basic
+ * block from which the hook was invoked is already covered by a
+ * preceding hook call.
+ * So unlike __sanitizer_cov_trace_pc_entry(), this PC should only be
+ * reported in extended mode, where function exit events are recorded.
+ */
+ if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC_EXT)
+ return;
+ record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;
+ kcov_add_pc_record(cur, record);
}
#endif

@@ -373,7 +400,7 @@ EXPORT_SYMBOL(__sanitizer_cov_trace_switch);
#endif /* ifdef CONFIG_KCOV_ENABLE_COMPARISONS */

static void kcov_start(struct task_struct *t, struct kcov *kcov,

- unsigned int size, void *area, enum kcov_mode mode,
+ unsigned int size, void *area, unsigned int mode,
int sequence)
{
kcov_debug("t = %px, size = %u, area = %px\n", t, size, area);
@@ -590,6 +617,8 @@ static int kcov_get_mode(unsigned long arg)
#else
return -ENOTSUPP;
#endif
+ else if (arg == KCOV_TRACE_PC_EXT)
+ return IS_ENABLED(CONFIG_KCOV_EXT_RECORDS) ? KCOV_MODE_TRACE_PC_EXT : -ENOTSUPP;
else
return -EINVAL;
}
@@ -1098,7 +1127,7 @@ void kcov_remote_stop(void)
* and kcov_remote_stop(), hence the sequence check.
*/
if (sequence == kcov->sequence && kcov->remote)
- kcov_move_area(kcov->mode, kcov->area, kcov->size, area);
+ kcov_move_area(kcov->mode & ~KCOV_EXT_FORMAT, kcov->area, kcov->size, area);
spin_unlock(&kcov->lock);

spin_lock(&kcov_remote_lock);

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

In case kcov is re-enabled with a different call stack than the one it
was disabled with, emit events that summarize changes to the call stack
so that userspace can continue tracking the call stack across context
switches.

Signed-off-by: Jann Horn <ja...@google.com>
---

include/linux/kcov.h | 11 ++---------
include/uapi/linux/kcov.h | 2 ++
kernel/kcov.c | 48 +++++++++++++++++++++++++++++++++++++++++++++--
3 files changed, 50 insertions(+), 11 deletions(-)

diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index 6c9f0373022f..357f4de8790a 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -35,15 +35,8 @@ enum kcov_mode {

void kcov_task_init(struct task_struct *t);
void kcov_task_exit(struct task_struct *t);

-#define kcov_prepare_switch(t) \
-do { \
- (t)->kcov_mode |= KCOV_IN_CTXSW; \
-} while (0)
-
-#define kcov_finish_switch(t) \
-do { \
- (t)->kcov_mode &= ~KCOV_IN_CTXSW; \
-} while (0)
+void kcov_prepare_switch(struct task_struct *cur);
+void kcov_finish_switch(struct task_struct *cur);

/* See Documentation/dev-tools/kcov.rst for usage details. */
void kcov_remote_start(u64 handle);
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 8d8a233bd61f..75c582784055 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -48,6 +48,8 @@ enum {
#define KCOV_RECORDFLAG_TYPE_NORMAL 0xf000000000000000
#define KCOV_RECORDFLAG_TYPE_ENTRY 0x0000000000000000
#define KCOV_RECORDFLAG_TYPE_EXIT 0x1000000000000000
+/* Summarized entry/exit events that occurred in an untraced region. */
+#define KCOV_RECORDFLAG_TYPE_EESUM 0x2000000000000000

/*
* The format for the types of collected comparisons.

diff --git a/kernel/kcov.c b/kernel/kcov.c
index 701ad69493bf..712f0f744ec5 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -76,6 +76,8 @@ struct kcov {
* kcov_remote_stop(), see the comment there.
*/
int sequence;
+ int suppressed_stack_delta;
+ int suppressed_stack_mindelta;
};

struct kcov_remote_area {
@@ -256,8 +258,12 @@ void notrace __sanitizer_cov_trace_pc_entry(void)

* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/

- if ((kcov_mode & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
+ if ((kcov_mode & ~(KCOV_EXT_FORMAT|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC)
return;
+ if (kcov_mode & KCOV_IN_CTXSW) {
+ cur->kcov->suppressed_stack_delta++;
+ return;

+ }
if ((kcov_mode & KCOV_EXT_FORMAT) != 0)

record = (record & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_ENTRY;
kcov_add_pc_record(cur, record);

@@ -266,6 +272,7 @@ void notrace __sanitizer_cov_trace_pc_exit(void)

{
struct task_struct *cur = current;

unsigned long record;

+ unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);

/*

* This hook is not called at the beginning of a basic block; the basic

@@ -274,8 +281,16 @@ void notrace __sanitizer_cov_trace_pc_exit(void)

* So unlike __sanitizer_cov_trace_pc_entry(), this PC should only be

* reported in extended mode, where function exit events are recorded.

*/
- if (READ_ONCE(cur->kcov_mode) != KCOV_MODE_TRACE_PC_EXT)
+ if ((kcov_mode & ~KCOV_IN_CTXSW) != KCOV_MODE_TRACE_PC_EXT)
return;
+ if (kcov_mode & KCOV_IN_CTXSW) {
+ struct kcov *kcov = cur->kcov;
+
+ if (kcov->suppressed_stack_mindelta == kcov->suppressed_stack_delta)
+ kcov->suppressed_stack_mindelta--;
+ kcov->suppressed_stack_delta--;

+ return;
+ }
record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;

kcov_add_pc_record(cur, record);
}
@@ -399,6 +414,35 @@ void notrace __sanitizer_cov_trace_switch(kcov_u64 val, void *arg)

EXPORT_SYMBOL(__sanitizer_cov_trace_switch);
#endif /* ifdef CONFIG_KCOV_ENABLE_COMPARISONS */

+void kcov_prepare_switch(struct task_struct *cur)
+{
+#ifdef CONFIG_KCOV_EXT_RECORDS
+ struct kcov *kcov = cur->kcov;
+
+ if (kcov) {
+ kcov->suppressed_stack_mindelta = 0;
+ kcov->suppressed_stack_delta = 0;
+ }
+#endif
+ cur->kcov_mode |= KCOV_IN_CTXSW;
+}
+
+void kcov_finish_switch(struct task_struct *cur)
+{
+ struct kcov *kcov = cur->kcov;

+ unsigned long record;
+

+ cur->kcov_mode &= ~KCOV_IN_CTXSW;
+ if (!IS_ENABLED(CONFIG_KCOV_EXT_RECORDS))
+ return;
+ if ((cur->kcov_mode & KCOV_EXT_FORMAT) == 0)
+ return;
+ record = KCOV_RECORDFLAG_TYPE_EESUM |
+ (((u16)(s16)kcov->suppressed_stack_mindelta)<<16) |
+ (((u16)(s16)kcov->suppressed_stack_delta)<<16);
+ kcov_add_pc_record(cur, record);
+}
+

static void kcov_start(struct task_struct *t, struct kcov *kcov,

unsigned int size, void *area, unsigned int mode,
int sequence)

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

Refactor the "write"/"is_write" arguments of kasan_check_range() and
kasan_report() into "flags" arguments that can contain the flag
KASAN_TYPE_WRITE.
This prepares for a following patch that introduces a second flag.

This should hopefully not change the machine code in hotpaths - both before
and after this change, the argument is either 0 or 1 at the assembly level
depending on whether the access is a write.
Only kasan_report() should have to do a tiny bit of extra work to do a
bittest.

Signed-off-by: Jann Horn <ja...@google.com>
---

arch/arm64/kernel/traps.c | 2 +-
arch/arm64/mm/fault.c | 2 +-

include/linux/kasan.h | 6 ++++--
mm/kasan/common.c | 2 +-
mm/kasan/generic.c | 20 ++++++++++----------
mm/kasan/kasan.h | 6 +++---
mm/kasan/report.c | 3 ++-
mm/kasan/report_generic.c | 8 ++++----
mm/kasan/shadow.c | 24 ++++++++++++------------
mm/kasan/sw_tags.c | 20 ++++++++++----------
10 files changed, 48 insertions(+), 45 deletions(-)

diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..9f31fe3f1660 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -1069,7 +1069,7 @@ int kasan_brk_handler(struct pt_regs *regs, unsigned long esr)
void *addr = (void *)regs->regs[0];
u64 pc = regs->pc;

- kasan_report(addr, size, write, pc);
+ kasan_report(addr, size, write ? KASAN_TYPE_WRITE : 0, pc);

/*
* The instrumentation allows to control whether we can proceed after
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0b52557652be..221e39869ae6 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -365,7 +365,7 @@ static void report_tag_fault(unsigned long addr, unsigned long esr,
* find out access size.
*/
bool is_write = !!(esr & ESR_ELx_WNR);
- kasan_report((void *)addr, 0, is_write, regs->pc);
+ kasan_report((void *)addr, 0, is_write ? KASAN_TYPE_WRITE : 0, regs->pc);
}
#else
/* Tag faults aren't enabled without CONFIG_KASAN_HW_TAGS. */
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index bf233bde68c7..03c7ac79345d 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -33,6 +33,8 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;
#define KASAN_VMALLOC_PAGE_RANGE 0x1 /* Apply exsiting page range */
#define KASAN_VMALLOC_TLB_FLUSH 0x2 /* TLB flush */

+#define KASAN_TYPE_WRITE 0x1
+
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

#include <linux/pgtable.h>
@@ -526,11 +528,11 @@ static inline void *kasan_reset_tag(const void *addr)
* kasan_report - print a report about a bad memory access detected by KASAN
* @addr: address of the bad access
* @size: size of the bad access
- * @is_write: whether the bad access is a write or a read
+ * @flags: bitmask, can contain KASAN_TYPE_* flags
* @ip: instruction pointer for the accessibility check or the bad access itself
*/
bool kasan_report(const void *addr, size_t size,
- bool is_write, unsigned long ip);
+ unsigned int flags, unsigned long ip);

#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index b7d05c2a6d93..1ab77ac9719c 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -571,7 +571,7 @@ void __kasan_mempool_unpoison_object(void *ptr, size_t size, unsigned long ip)
bool __kasan_check_byte(const void *address, unsigned long ip)
{
if (!kasan_byte_accessible(address)) {
- kasan_report(address, 1, false, ip);
+ kasan_report(address, 1, 0, ip);
return false;
}
return true;
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 2b8e73f5f6a7..9efd6fbbb7c3 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -173,7 +173,7 @@ static __always_inline bool memory_is_poisoned(const void *addr, size_t size)
}

static __always_inline bool check_region_inline(const void *addr,
- size_t size, bool write,
+ size_t size, unsigned int flags,
unsigned long ret_ip)
{
if (!kasan_enabled())
@@ -183,21 +183,21 @@ static __always_inline bool check_region_inline(const void *addr,
return true;

if (unlikely(addr + size < addr))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);

if (unlikely(!addr_has_metadata(addr)))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);

if (likely(!memory_is_poisoned(addr, size)))
return true;

- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
}

-bool kasan_check_range(const void *addr, size_t size, bool write,
+bool kasan_check_range(const void *addr, size_t size, unsigned int flags,
unsigned long ret_ip)
{
- return check_region_inline(addr, size, write, ret_ip);
+ return check_region_inline(addr, size, flags, ret_ip);
}

bool kasan_byte_accessible(const void *addr)
@@ -252,7 +252,7 @@ EXPORT_SYMBOL(__asan_unregister_globals);
#define DEFINE_ASAN_LOAD_STORE(size) \
void __asan_load##size(void *addr) \
{ \
- check_region_inline(addr, size, false, _RET_IP_); \
+ check_region_inline(addr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_load##size); \
__alias(__asan_load##size) \
@@ -260,7 +260,7 @@ EXPORT_SYMBOL(__asan_unregister_globals);
EXPORT_SYMBOL(__asan_load##size##_noabort); \
void __asan_store##size(void *addr) \
{ \
- check_region_inline(addr, size, true, _RET_IP_); \
+ check_region_inline(addr, size, KASAN_TYPE_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_store##size); \
__alias(__asan_store##size) \
@@ -275,7 +275,7 @@ DEFINE_ASAN_LOAD_STORE(16);

void __asan_loadN(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, false, _RET_IP_);
+ kasan_check_range(addr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__asan_loadN);

@@ -285,7 +285,7 @@ EXPORT_SYMBOL(__asan_loadN_noabort);

void __asan_storeN(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, true, _RET_IP_);
+ kasan_check_range(addr, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__asan_storeN);

diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index fc9169a54766..c833bd44e3cc 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -339,11 +339,11 @@ static __always_inline bool addr_has_metadata(const void *addr)
* kasan_check_range - Check memory region, and report if invalid access.
* @addr: the accessed address
* @size: the accessed size
- * @write: true if access is a write access
+ * @flags: bitmask, can contain KASAN_TYPE_* flags
* @ret_ip: return address
* @return: true if access was valid, false if invalid
*/
-bool kasan_check_range(const void *addr, size_t size, bool write,
+bool kasan_check_range(const void *addr, size_t size, unsigned int flags,
unsigned long ret_ip);

#else /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
@@ -379,7 +379,7 @@ static inline void kasan_print_aux_stacks(struct kmem_cache *cache, const void *
#endif

bool kasan_report(const void *addr, size_t size,
- bool is_write, unsigned long ip);
+ unsigned int flags, unsigned long ip);
void kasan_report_invalid_free(void *object, unsigned long ip, enum kasan_report_type type);

struct slab *kasan_addr_to_slab(const void *addr);
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index e804b1e1f886..cfe00ebb98ec 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -568,13 +568,14 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty
* user_access_save/restore(): kasan_report_invalid_free() cannot be called
* from a UACCESS region, and kasan_report_async() is not used on x86.
*/
-bool kasan_report(const void *addr, size_t size, bool is_write,
+bool kasan_report(const void *addr, size_t size, unsigned int flags,
unsigned long ip)
{
bool ret = true;
unsigned long ua_flags = user_access_save();
unsigned long irq_flags;
struct kasan_report_info info;
+ bool is_write = (flags & KASAN_TYPE_WRITE);

if (unlikely(report_suppressed_sw()) || unlikely(!report_enabled())) {
ret = false;
diff --git a/mm/kasan/report_generic.c b/mm/kasan/report_generic.c
index f5b8e37b3805..445183e2f4d3 100644
--- a/mm/kasan/report_generic.c
+++ b/mm/kasan/report_generic.c
@@ -364,14 +364,14 @@ void kasan_print_address_stack_frame(const void *addr)
#define DEFINE_ASAN_REPORT_LOAD(size) \
void __asan_report_load##size##_noabort(void *addr) \
{ \
- kasan_report(addr, size, false, _RET_IP_); \
+ kasan_report(addr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_report_load##size##_noabort)

#define DEFINE_ASAN_REPORT_STORE(size) \
void __asan_report_store##size##_noabort(void *addr) \
{ \
- kasan_report(addr, size, true, _RET_IP_); \
+ kasan_report(addr, size, KASAN_TYPE_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__asan_report_store##size##_noabort)

@@ -388,12 +388,12 @@ DEFINE_ASAN_REPORT_STORE(16);

void __asan_report_load_n_noabort(void *addr, ssize_t size)
{
- kasan_report(addr, size, false, _RET_IP_);
+ kasan_report(addr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__asan_report_load_n_noabort);

void __asan_report_store_n_noabort(void *addr, ssize_t size)
{
- kasan_report(addr, size, true, _RET_IP_);
+ kasan_report(addr, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__asan_report_store_n_noabort);
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index d286e0a04543..a24f1225dd88 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -28,13 +28,13 @@

bool __kasan_check_read(const volatile void *p, unsigned int size)
{
- return kasan_check_range((void *)p, size, false, _RET_IP_);
+ return kasan_check_range((void *)p, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_read);

bool __kasan_check_write(const volatile void *p, unsigned int size)
{
- return kasan_check_range((void *)p, size, true, _RET_IP_);
+ return kasan_check_range((void *)p, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_write);

@@ -50,7 +50,7 @@ EXPORT_SYMBOL(__kasan_check_write);
#undef memset
void *memset(void *addr, int c, size_t len)
{
- if (!kasan_check_range(addr, len, true, _RET_IP_))
+ if (!kasan_check_range(addr, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;

return __memset(addr, c, len);
@@ -60,8 +60,8 @@ void *memset(void *addr, int c, size_t len)
#undef memmove
void *memmove(void *dest, const void *src, size_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;

return __memmove(dest, src, len);
@@ -71,8 +71,8 @@ void *memmove(void *dest, const void *src, size_t len)
#undef memcpy
void *memcpy(void *dest, const void *src, size_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;

return __memcpy(dest, src, len);
@@ -81,7 +81,7 @@ void *memcpy(void *dest, const void *src, size_t len)

void *__asan_memset(void *addr, int c, ssize_t len)
{
- if (!kasan_check_range(addr, len, true, _RET_IP_))
+ if (!kasan_check_range(addr, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;

return __memset(addr, c, len);
@@ -91,8 +91,8 @@ EXPORT_SYMBOL(__asan_memset);
#ifdef __HAVE_ARCH_MEMMOVE
void *__asan_memmove(void *dest, const void *src, ssize_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;

return __memmove(dest, src, len);
@@ -102,8 +102,8 @@ EXPORT_SYMBOL(__asan_memmove);

void *__asan_memcpy(void *dest, const void *src, ssize_t len)
{
- if (!kasan_check_range(src, len, false, _RET_IP_) ||
- !kasan_check_range(dest, len, true, _RET_IP_))
+ if (!kasan_check_range(src, len, 0, _RET_IP_) ||
+ !kasan_check_range(dest, len, KASAN_TYPE_WRITE, _RET_IP_))
return NULL;

return __memcpy(dest, src, len);
diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index c75741a74602..af77b642ede7 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -72,7 +72,7 @@ u8 kasan_random_tag(void)
return (u8)(state % (KASAN_TAG_MAX + 1));
}

-bool kasan_check_range(const void *addr, size_t size, bool write,
+bool kasan_check_range(const void *addr, size_t size, unsigned int flags,
unsigned long ret_ip)
{
u8 tag;
@@ -83,7 +83,7 @@ bool kasan_check_range(const void *addr, size_t size, bool write,
return true;

if (unlikely(addr + size < addr))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);

tag = get_tag((const void *)addr);

@@ -109,12 +109,12 @@ bool kasan_check_range(const void *addr, size_t size, bool write,

untagged_addr = kasan_reset_tag((const void *)addr);
if (unlikely(!addr_has_metadata(untagged_addr)))
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
shadow_first = kasan_mem_to_shadow(untagged_addr);
shadow_last = kasan_mem_to_shadow(untagged_addr + size - 1);
for (shadow = shadow_first; shadow <= shadow_last; shadow++) {
if (*shadow != tag) {
- return !kasan_report(addr, size, write, ret_ip);
+ return !kasan_report(addr, size, flags, ret_ip);
}
}

@@ -137,12 +137,12 @@ bool kasan_byte_accessible(const void *addr)
#define DEFINE_HWASAN_LOAD_STORE(size) \
void __hwasan_load##size##_noabort(void *addr) \
{ \
- kasan_check_range(addr, size, false, _RET_IP_); \
+ kasan_check_range(addr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__hwasan_load##size##_noabort); \
void __hwasan_store##size##_noabort(void *addr) \
{ \
- kasan_check_range(addr, size, true, _RET_IP_); \
+ kasan_check_range(addr, size, KASAN_TYPE_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__hwasan_store##size##_noabort)

@@ -154,13 +154,13 @@ DEFINE_HWASAN_LOAD_STORE(16);

void __hwasan_loadN_noabort(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, false, _RET_IP_);
+ kasan_check_range(addr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__hwasan_loadN_noabort);

void __hwasan_storeN_noabort(void *addr, ssize_t size)
{
- kasan_check_range(addr, size, true, _RET_IP_);
+ kasan_check_range(addr, size, KASAN_TYPE_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__hwasan_storeN_noabort);

@@ -173,6 +173,6 @@ EXPORT_SYMBOL(__hwasan_tag_memory);
void kasan_tag_mismatch(void *addr, unsigned long access_info,
unsigned long ret_ip)
{
- kasan_report(addr, 1 << (access_info & 0xf), access_info & 0x10,
- ret_ip);
+ kasan_report(addr, 1 << (access_info & 0xf),
+ (access_info & 0x10) ? KASAN_TYPE_WRITE : 0, ret_ip);
}

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

This commit only introduces tracing of memory accesses that are
instrumented at the source level with instrument_*(); a followup commit
will additionally provide data based on ASAN compiler instrumentation.

I am adding the instrumentation hook definitions directly in
include/linux/instrumented.h instead of adding separate headers; this
way the compiler won't have to read yet another header file for almost
every compilation unit.

To avoid instrumenting files that shouldn't be instrumented, reuse
KASAN's __SANITIZE_ADDRESS__.

Signed-off-by: Jann Horn <ja...@google.com>
---

include/linux/instrumented.h | 30 +++++++++++++++++
include/linux/kcov.h | 11 ++++++
include/uapi/linux/kcov.h | 24 +++++++++++++
kernel/kcov.c | 80 +++++++++++++++++++++++++++++++++++++++++---
lib/Kconfig.debug | 11 ++++++
5 files changed, 152 insertions(+), 4 deletions(-)

diff --git a/include/linux/instrumented.h b/include/linux/instrumented.h
index a1b4cf81adc2..940776dff616 100644
--- a/include/linux/instrumented.h
+++ b/include/linux/instrumented.h
@@ -13,6 +13,26 @@
#include <linux/kcsan-checks.h>
#include <linux/kmsan-checks.h>
#include <linux/types.h>
+#ifdef CONFIG_KCOV_MEMORY
+/* For build speed, only include this header in builds that actually need it. */
+#include <uapi/linux/kcov.h>
+#endif
+
+#ifdef CONFIG_KCOV_MEMORY
+void _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type);
+#else
+static __always_inline void _kcov_handle_memaccess(const volatile void *p,
+ size_t size, unsigned int type) {}
+/* Discard type argument to avoid depending on kcov header. */
+#define _kcov_handle_memaccess(p, size, type) _kcov_handle_memaccess((p), (size), 0)
+#endif
+
+#if defined(__SANITIZE_ADDRESS__) || !defined(CONFIG_KCOV_MEMORY)
+#define kcov_handle_memaccess _kcov_handle_memaccess
+#else
+static __always_inline void kcov_handle_memaccess(const volatile void *p,
+ size_t size, unsigned int type) {}
+#endif

/**
* instrument_read - instrument regular read access
@@ -24,6 +44,7 @@
*/
static __always_inline void instrument_read(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, 0);
kasan_check_read(v, size);
kcsan_check_read(v, size);
}
@@ -38,6 +59,7 @@ static __always_inline void instrument_read(const volatile void *v, size_t size)
*/
static __always_inline void instrument_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_WRITE);
kasan_check_write(v, size);
kcsan_check_write(v, size);
}
@@ -52,6 +74,7 @@ static __always_inline void instrument_write(const volatile void *v, size_t size
*/
static __always_inline void instrument_read_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_RMW);
kasan_check_write(v, size);
kcsan_check_read_write(v, size);
}
@@ -79,6 +102,7 @@ static __always_inline void instrument_atomic_check_alignment(const volatile voi
*/
static __always_inline void instrument_atomic_read(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_ATOMIC);
kasan_check_read(v, size);
kcsan_check_atomic_read(v, size);
instrument_atomic_check_alignment(v, size);
@@ -94,6 +118,7 @@ static __always_inline void instrument_atomic_read(const volatile void *v, size_
*/
static __always_inline void instrument_atomic_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_ATOMIC);
kasan_check_write(v, size);
kcsan_check_atomic_write(v, size);
instrument_atomic_check_alignment(v, size);
@@ -109,6 +134,7 @@ static __always_inline void instrument_atomic_write(const volatile void *v, size
*/
static __always_inline void instrument_atomic_read_write(const volatile void *v, size_t size)
{
+ kcov_handle_memaccess(v, size, MEMORY_ACCESS_RECORD_RMW|MEMORY_ACCESS_RECORD_ATOMIC);
kasan_check_write(v, size);
kcsan_check_atomic_read_write(v, size);
instrument_atomic_check_alignment(v, size);
@@ -126,6 +152,7 @@ static __always_inline void instrument_atomic_read_write(const volatile void *v,
static __always_inline void
instrument_copy_to_user(void __user *to, const void *from, unsigned long n)
{
+ kcov_handle_memaccess(from, n, 0);
kasan_check_read(from, n);
kcsan_check_read(from, n);
kmsan_copy_to_user(to, from, n, 0);
@@ -143,6 +170,7 @@ instrument_copy_to_user(void __user *to, const void *from, unsigned long n)
static __always_inline void
instrument_copy_from_user_before(const void *to, const void __user *from, unsigned long n)
{
+ kcov_handle_memaccess(to, n, MEMORY_ACCESS_RECORD_WRITE);
kasan_check_write(to, n);
kcsan_check_write(to, n);
}
@@ -176,6 +204,8 @@ instrument_copy_from_user_after(const void *to, const void __user *from,
static __always_inline void instrument_memcpy_before(void *to, const void *from,
unsigned long n)
{
+ kcov_handle_memaccess(from, n, 0);
+ kcov_handle_memaccess(to, n, MEMORY_ACCESS_RECORD_WRITE);
kasan_check_write(to, n);
kasan_check_read(from, n);
kcsan_check_write(to, n);
diff --git a/include/linux/kcov.h b/include/linux/kcov.h
index 357f4de8790a..e4b818df189e 100644
--- a/include/linux/kcov.h
+++ b/include/linux/kcov.h
@@ -23,6 +23,7 @@ enum kcov_mode {
KCOV_MODE_TRACE_CMP = 3,
};

+#define KCOV_ENABLE_MEMORY (1 << 28)
/*

* Modifier for KCOV_MODE_TRACE_PC to record function entry/exit marked with

* metadata bits.
@@ -31,6 +32,7 @@ enum kcov_mode {
#define KCOV_IN_CTXSW (1 << 30)

#define KCOV_MODE_TRACE_PC_EXT (KCOV_MODE_TRACE_PC | KCOV_EXT_FORMAT)
+#define KCOV_MODE_TRACE_PC_AND_MEM (KCOV_MODE_TRACE_PC_EXT | KCOV_ENABLE_MEMORY)

void kcov_task_init(struct task_struct *t);
void kcov_task_exit(struct task_struct *t);

@@ -109,4 +111,13 @@ static inline void kcov_remote_start_usb_softirq(u64 id) {}
static inline void kcov_remote_stop_softirq(void) {}

#endif /* CONFIG_KCOV */
+
+#ifdef CONFIG_KCOV_MEMORY
+void __kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type,
+ unsigned long ret_ip);
+#else /* CONFIG_KCOV_MEMORY */
+static inline void __kcov_handle_memaccess(const volatile void *p, size_t size,
+ unsigned int type, unsigned long ret_ip) {}
+#endif /* CONFIG_KCOV_MEMORY */
+
#endif /* _LINUX_KCOV_H */
diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 75c582784055..7d7147e7b427 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -22,6 +22,7 @@ struct kcov_remote_arg {
#define KCOV_ENABLE _IO('c', 100)
#define KCOV_DISABLE _IO('c', 101)
#define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg)
+#define KCOV_GET_MEMORY_RECORD_SIZE _IO('c', 103)

enum {
/*
@@ -41,6 +42,8 @@ enum {
* (KCOV_RECORDFLAG_*).
*/
KCOV_TRACE_PC_EXT = 2,
+ /* Extended PC coverage mode with tracing of memory accesses. */
+ KCOV_TRACE_MEMORY_ACCESS = 3,
};

#define KCOV_RECORD_IP_MASK 0x00ffffffffffffff
@@ -50,6 +53,7 @@ enum {
#define KCOV_RECORDFLAG_TYPE_EXIT 0x1000000000000000

/* Summarized entry/exit events that occurred in an untraced region. */

#define KCOV_RECORDFLAG_TYPE_EESUM 0x2000000000000000
+#define KCOV_RECORDFLAG_TYPE_MEMORY 0x3000000000000000

/*
* The format for the types of collected comparisons.

@@ -74,4 +78,24 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
return subsys | inst;
}

+/*
+ * Data format for memory access tracing mode.
+ * This is an extensible struct (it can be extended by appending elements);
+ * userspace can query the struct size used by the running kernel with
+ * KCOV_GET_MEMORY_ACCESS_RECORD_SIZE.
+ */
+#define MEMORY_ACCESS_RECORD_TYPE_MASK 0xf
+#define MEMORY_ACCESS_RECORD_TYPE_ACCESS 0x0
+/* flags for MEMORY_ACCESS_RECORD_TYPE_ACCESS */
+#define MEMORY_ACCESS_RECORD_WRITE 0x10
+#define MEMORY_ACCESS_RECORD_RMW 0x20
+#define MEMORY_ACCESS_RECORD_ATOMIC 0x40
+struct memory_access_record {
+ __aligned_u64 ip_address_and_kcov_flags;
+ __aligned_u64 data_address;
+ __u32 size;
+ __u32 flags; /* MEMORY_ACCESS_RECORD_* */
+ __aligned_u64 time;
+} __attribute__((aligned(8)));
+
#endif /* _LINUX_KCOV_IOCTLS_H */
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 712f0f744ec5..83e05aa61728 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -235,7 +235,8 @@ void notrace __sanitizer_cov_trace_pc(void)

{
struct task_struct *cur = current;

- if ((READ_ONCE(cur->kcov_mode) & ~KCOV_EXT_FORMAT) != KCOV_MODE_TRACE_PC)
+ if ((READ_ONCE(cur->kcov_mode) & ~(KCOV_ENABLE_MEMORY|KCOV_EXT_FORMAT))
+ != KCOV_MODE_TRACE_PC)
return;
/*

* No bitops are needed here for setting the record type because

@@ -258,7 +259,7 @@ void notrace __sanitizer_cov_trace_pc_entry(void)

* This hook replaces __sanitizer_cov_trace_pc() for the function entry
* basic block; it should still emit a record even in classic kcov mode.
*/

- if ((kcov_mode & ~(KCOV_EXT_FORMAT|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC)
+ if ((kcov_mode & ~(KCOV_ENABLE_MEMORY|KCOV_EXT_FORMAT|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC)
return;
if (kcov_mode & KCOV_IN_CTXSW) {
cur->kcov->suppressed_stack_delta++;
@@ -281,7 +282,7 @@ void notrace __sanitizer_cov_trace_pc_exit(void)

* So unlike __sanitizer_cov_trace_pc_entry(), this PC should only be
* reported in extended mode, where function exit events are recorded.
*/

- if ((kcov_mode & ~KCOV_IN_CTXSW) != KCOV_MODE_TRACE_PC_EXT)
+ if ((kcov_mode & ~(KCOV_ENABLE_MEMORY|KCOV_IN_CTXSW)) != KCOV_MODE_TRACE_PC_EXT)
return;
if (kcov_mode & KCOV_IN_CTXSW) {

struct kcov *kcov = cur->kcov;

@@ -663,6 +664,8 @@ static int kcov_get_mode(unsigned long arg)
#endif

else if (arg == KCOV_TRACE_PC_EXT)

return IS_ENABLED(CONFIG_KCOV_EXT_RECORDS) ? KCOV_MODE_TRACE_PC_EXT : -ENOTSUPP;

+ else if (arg == KCOV_TRACE_MEMORY_ACCESS)
+ return IS_ENABLED(CONFIG_KCOV_MEMORY) ? KCOV_MODE_TRACE_PC_AND_MEM : -ENOTSUPP;
else
return -EINVAL;
}
@@ -803,6 +806,10 @@ static int kcov_ioctl_locked(struct kcov *kcov, unsigned int cmd,
/* Put either in kcov_task_exit() or in KCOV_DISABLE. */
kcov_get(kcov);
return 0;
+ case KCOV_GET_MEMORY_RECORD_SIZE:
+ if (!IS_ENABLED(CONFIG_KCOV_MEMORY))
+ return -ENOTSUPP;
+ return sizeof(struct memory_access_record);
default:
return -ENOTTY;
}
@@ -1171,7 +1178,8 @@ void kcov_remote_stop(void)

* and kcov_remote_stop(), hence the sequence check.
*/
if (sequence == kcov->sequence && kcov->remote)

- kcov_move_area(kcov->mode & ~KCOV_EXT_FORMAT, kcov->area, kcov->size, area);
+ kcov_move_area(kcov->mode & ~(KCOV_ENABLE_MEMORY|KCOV_EXT_FORMAT),
+ kcov->area, kcov->size, area);
spin_unlock(&kcov->lock);

spin_lock(&kcov_remote_lock);
@@ -1194,6 +1202,70 @@ struct kcov_common_handle_id kcov_common_handle(void)
}
EXPORT_SYMBOL(kcov_common_handle);

+#ifdef CONFIG_KCOV_MEMORY
+static notrace bool kcov_get_memaccess_record(struct task_struct *t,
+ struct memory_access_record **recordp)
+{
+ u64 *area = (u64 *)t->kcov_area;
+ /* The buffer was allocated for t->kcov_size unsigned longs. */
+ u64 max_pos = t->kcov_size * sizeof(unsigned long);
+ u64 count = READ_ONCE(area[0]);
+ u64 start_pos = sizeof(unsigned long) + count * sizeof(unsigned long);
+ u64 end_pos = start_pos + sizeof(struct memory_access_record);
+
+ if (unlikely(end_pos > max_pos))
+ return false;
+
+ /* See comment in kcov_add_pc_record(). */
+ WRITE_ONCE(area[0], count + sizeof(struct memory_access_record)/sizeof(unsigned long));
+ barrier();
+ *recordp = (void *)area + start_pos;
+ return true;
+}
+
+/*
+ * Memory ordering doesn't matter a lot here because timestamps aren't
+ * collected atomically with memory accesses anyway.
+ * The important things are that the clock access has to be uaccess-safe,
+ * notrace, and have high granularity.
+ */
+static notrace __always_inline u64 kcov_get_time(void)
+{
+#ifdef CONFIG_X86
+ return rdtsc_ordered();
+#else
+ return 0;
+#endif
+}
+
+void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type,
+ unsigned long ret_ip)
+{
+ struct task_struct *t = current;
+ struct memory_access_record *record;
+ unsigned int kcov_mode = READ_ONCE(t->kcov_mode);
+
+ if (kcov_mode != KCOV_MODE_TRACE_PC_AND_MEM || !check_kcov_context(t))
+ return;
+ if (!kcov_get_memaccess_record(t, &record))
+ return;
+ *record = (struct memory_access_record) {
+ .ip_address_and_kcov_flags =
+ (ret_ip & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_MEMORY,
+ .data_address = (u64)p,
+ .size = size,
+ .flags = type,
+ .time = kcov_get_time()
+ };
+}
+
+void notrace _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type)
+{
+ __kcov_handle_memaccess(p, size, type, _RET_IP_);
+}
+EXPORT_SYMBOL(_kcov_handle_memaccess);
+#endif /* CONFIG_KCOV_MEMORY */
+
#ifdef CONFIG_KCOV_SELFTEST
static void __init selftest(void)
{
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 6ddf58692b09..5de427ccc42d 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2217,6 +2217,17 @@ config KCOV_ENABLE_COMPARISONS
These operands can be used by fuzzing engines to improve the quality
of fuzzing coverage.

+config KCOV_MEMORY
+ bool "Enable memory access trace collection by KCOV"
+ depends on KCOV
+ depends on KCOV_EXT_RECORDS
+ help
+ Provide a KCOV mode which records memory access operations and allows
+ userspace to inject execution delays to impose constraints on the
+ order in which multithreaded execution happens.
+
+ This is mainly useful for testing race condition bugs.
+
config KCOV_INSTRUMENT_ALL
bool "Instrument all code by default"
depends on KCOV

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

In CONFIG_KCOV_MEMORY builds, let KASAN provide information about memory
accesses to KCOV.
Since KCOV already receives information from instrument_*() directly,
filter out those accesses by introducing KASAN_TYPE_EXPLICIT.
Because the KCOV usecase requires seeing ~every memory access (including
accesses to globals and repeated accesses to the same memory location
within a basic block), gate this on CC_IS_CLANG and let it flip some hidden
LLVM flags that disable ASAN optimizations.

Signed-off-by: Jann Horn <ja...@google.com>
---

include/linux/kasan.h | 1 +
lib/Kconfig.debug | 2 ++
lib/Kconfig.kasan | 9 +++++++++
mm/kasan/generic.c | 15 +++++++++++++++
mm/kasan/shadow.c | 5 +++--
scripts/Makefile.kasan | 17 +++++++++++++++++
tools/objtool/check.c | 1 +
7 files changed, 48 insertions(+), 2 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 03c7ac79345d..4b915e0c51bc 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -34,6 +34,7 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;

#define KASAN_VMALLOC_TLB_FLUSH 0x2 /* TLB flush */

#define KASAN_TYPE_WRITE 0x1
+#define KASAN_TYPE_EXPLICIT 0x2

#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 5de427ccc42d..f763f0504f62 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2221,6 +2221,8 @@ config KCOV_MEMORY

bool "Enable memory access trace collection by KCOV"

depends on KCOV
depends on KCOV_EXT_RECORDS
+ depends on HAVE_KASAN_REPORT_EVERY_ACCESS
+ select KASAN_REPORT_EVERY_ACCESS
help

Provide a KCOV mode which records memory access operations and allows

userspace to inject execution delays to impose constraints on the

diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index a4bb610a7a6f..9a43f6269c5c 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -228,4 +228,13 @@ config KASAN_EXTRA_INFO
boot parameter, it will add 8 * stack_ring_size bytes of additional
memory consumption.

+# Is KASAN_REPORT_EVERY_ACCESS allowed?
+config HAVE_KASAN_REPORT_EVERY_ACCESS
+ def_bool y
+ depends on KASAN_OUTLINE
+ depends on CC_IS_CLANG
+
+config KASAN_REPORT_EVERY_ACCESS
+ bool
+
endif # KASAN
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 9efd6fbbb7c3..6cf88f569e64 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -32,6 +32,8 @@
#include <linux/types.h>
#include <linux/vmalloc.h>
#include <linux/bug.h>
+#include <linux/kcov.h>
+#include <uapi/linux/kcov.h>

#include "kasan.h"
#include "../slab.h"
@@ -182,6 +184,19 @@ static __always_inline bool check_region_inline(const void *addr,
if (unlikely(size == 0))
return true;

+ /*
+ * Do not route information about an access to KCOV if we got called
+ * through the instrument_*() path - KCOV can get those accesses
+ * directly from instrument_*(), and get a bit more metadata about the
+ * access that way.
+ */
+ if (likely((flags & KASAN_TYPE_EXPLICIT) == 0)) {
+ unsigned int kcov_flags =
+ (flags & KASAN_TYPE_WRITE) ? MEMORY_ACCESS_RECORD_WRITE : 0;
+
+ __kcov_handle_memaccess(addr, size, kcov_flags, ret_ip);
+ }
+

if (unlikely(addr + size < addr))

return !kasan_report(addr, size, flags, ret_ip);

diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index a24f1225dd88..84f8567d4f2c 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -28,13 +28,14 @@

bool __kasan_check_read(const volatile void *p, unsigned int size)
{

- return kasan_check_range((void *)p, size, 0, _RET_IP_);
+ return kasan_check_range((void *)p, size, KASAN_TYPE_EXPLICIT, _RET_IP_);

}
EXPORT_SYMBOL(__kasan_check_read);

bool __kasan_check_write(const volatile void *p, unsigned int size)
{

- return kasan_check_range((void *)p, size, KASAN_TYPE_WRITE, _RET_IP_);

+ return kasan_check_range((void *)p, size,

+ KASAN_TYPE_WRITE|KASAN_TYPE_EXPLICIT, _RET_IP_);
}
EXPORT_SYMBOL(__kasan_check_write);

diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan
index 91504e81247a..82e88c5fb9bc 100644
--- a/scripts/Makefile.kasan
+++ b/scripts/Makefile.kasan
@@ -60,6 +60,23 @@ kasan_params += asan-instrumentation-with-call-threshold=$(call_threshold) \
asan-instrument-allocas=1 \
asan-globals=1

+# When we piggyback tracing of memory accesses for race condition testing on top
+# of KASAN, we want the compiler to report memory accesses even when KASAN can
+# prove that no UAF/OOB can occur; in particular, these optimizations must be
+# inhibited:
+#
+# - suppression of ASAN hook calls for global variables
+# - merging of multiple accesses in a basic block into a single ASAN hook call
+#
+# For now, known stack variable accesses are still ignored as a performance
+# tradeoff, though that will probably make a small number of races (where
+# another task concurrently accesses stuff on our stack) invisible to the
+# instrumentation. (Stack access instrumentation is gated on
+# asan-use-stack-safety and asan-skip-promotable-allocas.)
+ifdef CONFIG_KASAN_REPORT_EVERY_ACCESS
+kasan_params += asan-opt-globals=0 asan-opt-same-temp=0
+endif # CONFIG_KASAN_REPORT_EVERY_ACCESS
+
# Instrument memcpy/memset/memmove calls by using instrumented __asan_mem*()
# instead. With compilers that don't support this option, compiler-inserted
# memintrinsics won't be checked by KASAN on GENERIC_ENTRY architectures.
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index d70cb640e2ec..08ebfe1f3fac 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1219,6 +1219,7 @@ static const char *uaccess_safe_builtin[] = {
/* KCOV */
"write_comp_data",
"check_kcov_mode",
+ "__kcov_handle_memaccess",
"__sanitizer_cov_trace_pc",
"__sanitizer_cov_trace_pc_entry",
"__sanitizer_cov_trace_pc_exit",

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

To help with using CONFIG_KCOV_MEMORY for detecting use-after-free issues,
log when memory (SLUB objects or page allocations) is being freed.

This should happen after KASAN has already marked the memory as freed;
this will become important if we allow driving delay injection off this
in the future.

Signed-off-by: Jann Horn <ja...@google.com>
---

include/uapi/linux/kcov.h | 1 +
mm/kasan/common.c | 2 ++
mm/page_alloc.c | 3 +++
3 files changed, 6 insertions(+)

diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 7d7147e7b427..76822d1c119a 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -90,6 +90,7 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)
#define MEMORY_ACCESS_RECORD_WRITE 0x10
#define MEMORY_ACCESS_RECORD_RMW 0x20
#define MEMORY_ACCESS_RECORD_ATOMIC 0x40
+#define MEMORY_ACCESS_RECORD_FREE 0x80
struct memory_access_record {
__aligned_u64 ip_address_and_kcov_flags;
__aligned_u64 data_address;
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 1ab77ac9719c..3a648eaad024 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -283,6 +283,8 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, bool init,
return false;

poison_slab_object(cache, object, init);
+ _kcov_handle_memaccess(object, cache->object_size,
+ MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_FREE);

if (no_quarantine)
return false;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 083cbcb5bdde..910d14925b84 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1446,6 +1446,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
if (init)
clear_highpages_kasan_tagged(page, 1 << order);

+ _kcov_handle_memaccess(page_address(page), (1 << order)*PAGE_SIZE,
+ MEMORY_ACCESS_RECORD_WRITE|MEMORY_ACCESS_RECORD_FREE);
+
/*
* arch_free_page() can make the page's contents inaccessible. s390
* does this. So nothing which can access the page's contents should

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

It is helpful to know which code location a function was called from for:

- attributing calls to source locations in the callee
- attributing calls to inlined functions

For this purpose, make function entry records bigger, and record the caller
instruction address in them.

Signed-off-by: Jann Horn <ja...@google.com>
---

kernel/kcov.c | 26 ++++++++++++++++++--------
lib/Kconfig.debug | 2 ++
2 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/kernel/kcov.c b/kernel/kcov.c
index 83e05aa61728..88aedaf41a9e 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -202,7 +202,8 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
return ip;
}

-static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, unsigned long record)
+static __always_inline notrace
+void kcov_add_pc_record(struct task_struct *t, unsigned long record, bool hasext, unsigned long ext)
{

unsigned long *area;
unsigned long pos;

@@ -213,7 +214,7 @@ static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, un
area = t->kcov_area;
/* The first 64-bit word is the number of subsequent PCs. */
pos = READ_ONCE(area[0]) + 1;
- if (likely(pos < t->kcov_size)) {
+ if (likely(pos + (hasext?1:0) < t->kcov_size)) {

/* Previously we write pc before updating pos. However, some

* early interrupt code could bypass check_kcov_context() check
* and invoke __sanitizer_cov_trace_pc(). If such interrupt is

@@ -221,9 +222,11 @@ static __always_inline void notrace kcov_add_pc_record(struct task_struct *t, un

* overitten by the recursive __sanitizer_cov_trace_pc().

* Update pos before writing pc to avoid such interleaving.
*/
- WRITE_ONCE(area[0], pos);
+ WRITE_ONCE(area[0], pos + (hasext?1:0));
barrier();
area[pos] = record;
+ if (hasext)
+ area[pos+1] = ext;
}
}

@@ -244,7 +247,7 @@ void notrace __sanitizer_cov_trace_pc(void)

* This relies on userspace not caring about the rest of the top byte

* for KCOV_RECORDFLAG_TYPE_NORMAL records.
*/
- kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_));
+ kcov_add_pc_record(cur, canonicalize_ip(_RET_IP_), false, 0);
}
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);

@@ -254,6 +257,7 @@ void notrace __sanitizer_cov_trace_pc_entry(void)

struct task_struct *cur = current;

unsigned long record = canonicalize_ip(_RET_IP_);

unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);

+ bool ext_format;

/*

* This hook replaces __sanitizer_cov_trace_pc() for the function entry

@@ -265,9 +269,15 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
cur->kcov->suppressed_stack_delta++;
return;
}
- if ((kcov_mode & KCOV_EXT_FORMAT) != 0)
+ ext_format = (kcov_mode & KCOV_EXT_FORMAT) != 0;
+ if (ext_format)

record = (record & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_ENTRY;

- kcov_add_pc_record(cur, record);
+ /*
+ * __builtin_return_address(1) is safe because this function is only
+ * called from C functions, which are compiled with frame pointers
+ * enabled
+ */
+ kcov_add_pc_record(cur, record, ext_format, (unsigned long)__builtin_return_address(1));
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{
@@ -293,7 +303,7 @@ void notrace __sanitizer_cov_trace_pc_exit(void)
return;

}
record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;

- kcov_add_pc_record(cur, record);
+ kcov_add_pc_record(cur, record, false, 0);
}
#endif

@@ -441,7 +451,7 @@ void kcov_finish_switch(struct task_struct *cur)
record = KCOV_RECORDFLAG_TYPE_EESUM |
(((u16)(s16)kcov->suppressed_stack_mindelta)<<16) |
(((u16)(s16)kcov->suppressed_stack_delta)<<16);
- kcov_add_pc_record(cur, record);
+ kcov_add_pc_record(cur, record, false, 0);

}

static void kcov_start(struct task_struct *t, struct kcov *kcov,

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index f763f0504f62..55c786a373b5 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2200,6 +2200,8 @@ config KCOV_EXT_RECORDS
depends on KCOV
depends on 64BIT
depends on $(cc-option,-fsanitize-coverage=trace-pc-entry-exit)
+ select ARCH_WANT_FRAME_POINTERS
+ select FRAME_POINTER
help

Extended KCOV records allow distinguishing between multiple types of

records: Normal edge coverage, function entry, and function exit.

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

For manual analysis of traces, log the old value at the memory location as
part of the memory_access_record.

This is best-effort; in particular:

- the value may not be recorded if it has an unusual size
- the recorded value may not match the value observed by the instrumented
memory operation in cases where the value is modified concurrently

Signed-off-by: Jann Horn <ja...@google.com>
---

include/uapi/linux/kcov.h | 2 ++
kernel/kcov.c | 20 ++++++++++++++++++++
2 files changed, 22 insertions(+)

diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 76822d1c119a..235f73e11d59 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -91,12 +91,14 @@ static inline __u64 kcov_remote_handle(__u64 subsys, __u64 inst)

#define MEMORY_ACCESS_RECORD_RMW 0x20
#define MEMORY_ACCESS_RECORD_ATOMIC 0x40

#define MEMORY_ACCESS_RECORD_FREE 0x80
+#define MEMORY_ACCESS_RECORD_VALUE 0x100 /* value field is valid */

struct memory_access_record {
__aligned_u64 ip_address_and_kcov_flags;
__aligned_u64 data_address;

__u32 size;

__u32 flags; /* MEMORY_ACCESS_RECORD_* */

__aligned_u64 time;
+ __aligned_u64 value;
} __attribute__((aligned(8)));

#endif /* _LINUX_KCOV_IOCTLS_H */
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 88aedaf41a9e..ef405940a2cb 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -1267,6 +1267,26 @@ void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsign
.flags = type,
.time = kcov_get_time()
};
+
+ switch (size) {
+ case 1:
+ __get_kernel_nofault((u8 *)&record->value, p, u8, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ case 2:
+ __get_kernel_nofault((u16 *)&record->value, p, u16, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ case 4:
+ __get_kernel_nofault((u32 *)&record->value, p, u32, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ case 8:
+ __get_kernel_nofault((u64 *)&record->value, p, u64, handle_fault);
+ record->flags |= MEMORY_ACCESS_RECORD_VALUE;
+ break;
+ }
+handle_fault:;

}

void notrace _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type)

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

Introduce KCOV-based delay injection, which is intended for
deterministically testing race condition bugs.

Preceding patches allow userspace to record a trace of function entry/exit
events and memory access events in a multi-threaded test case.

After userspace identifies kernel memory accesses that could be part of a
race condition, userspace can use KCOV_SET_DI on the participating threads
to instruct KCOV to ensure that pairs of memory accesses execute in the
desired order.

A kcov_di_stack with type DI_STACK_WAIT instructs KCOV to spin-wait for
another kcov_di_stack (normally installed on another thread) with type
DI_STACK_WAKE and with the same flagidx.

Signed-off-by: Jann Horn <ja...@google.com>
---

include/uapi/linux/kcov.h | 45 ++++++
kernel/kcov.c | 380 +++++++++++++++++++++++++++++++++++++++++++++-
tools/objtool/check.c | 1 +
3 files changed, 422 insertions(+), 4 deletions(-)

diff --git a/include/uapi/linux/kcov.h b/include/uapi/linux/kcov.h
index 235f73e11d59..88bd11d7d108 100644
--- a/include/uapi/linux/kcov.h
+++ b/include/uapi/linux/kcov.h
@@ -3,6 +3,7 @@
#define _LINUX_KCOV_IOCTLS_H

#include <linux/types.h>
+#include <linux/ioctl.h>

/*
* Argument for KCOV_REMOTE_ENABLE ioctl, see Documentation/dev-tools/kcov.rst
@@ -23,6 +24,10 @@ struct kcov_remote_arg {

#define KCOV_DISABLE _IO('c', 101)
#define KCOV_REMOTE_ENABLE _IOW('c', 102, struct kcov_remote_arg)

#define KCOV_GET_MEMORY_RECORD_SIZE _IO('c', 103)

+#define KCOV_SET_DI _IOW('c', 104, struct kcov_set_di_arg)
+#define KCOV_RESET_DI_FLAGS _IO('c', 105)
+#define KCOV_WAKE_DI_FLAG _IO('c', 106)
+#define KCOV_SPINWAIT_DI_FLAG _IO('c', 107)

enum {
/*
@@ -54,6 +59,11 @@ enum {

/* Summarized entry/exit events that occurred in an untraced region. */
#define KCOV_RECORDFLAG_TYPE_EESUM 0x2000000000000000

#define KCOV_RECORDFLAG_TYPE_MEMORY 0x3000000000000000
+/* these two record types have a flag index in the low bits */
+#define KCOV_RECORDFLAG_TYPE_WAIT 0x4000000000000000
+#define KCOV_RECORDFLAG_TYPE_WAKE 0x5000000000000000
+/* set in KCOV_RECORDFLAG_TYPE_WAIT record to mark that the wait timed out */
+#define KCOV_WAIT_TIMEOUT 0x0010000000000000

/*
* The format for the types of collected comparisons.

@@ -101,4 +111,39 @@ struct memory_access_record {
__aligned_u64 value;
} __attribute__((aligned(8)));

+
+/*
+ * Delay Injection API
+ */
+struct kcov_di_stack_elem {
+ __aligned_u64 ip;
+ __aligned_u64 parent_idx;
+};
+enum di_stack_type {
+ DI_STACK_WAIT = 0,
+ DI_STACK_WAKE_PRE,
+ DI_STACK_WAKE_POST
+};
+struct kcov_di_stack {
+ __aligned_u64 elems;
+ __u32 num_elems;
+ enum di_stack_type type;
+ __u32 flagidx;
+};
+struct kcov_set_di_arg {
+ /*
+ * Pointer to array of struct kcov_di_stack.
+ * The array consists of function entry instruction addresses, with a
+ * memory access instruction address at the end.
+ * These must be addresses as reported in KCOV_RECORDFLAG_TYPE_ENTRY and
+ * KCOV_RECORDFLAG_TYPE_MEMORY events (so they are not the addresses
+ * where functions begin).
+ */
+ __aligned_u64 stacks;
+ __u32 num_stacks;
+
+ int sync_bits_fd;
+ __aligned_u64 spin_limit;
+};
+

#endif /* _LINUX_KCOV_IOCTLS_H */
diff --git a/kernel/kcov.c b/kernel/kcov.c

index ef405940a2cb..318ae2c89891 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -32,6 +32,22 @@
/* Number of 64-bit words written per one comparison: */
#define KCOV_WORDS_PER_CMP 4

+#define NUM_SYNC_BITS 64
+
+struct di_stack_elem {
+ unsigned long ip;
+ unsigned long parent_idx;
+ unsigned long cur_parent_idx;
+};
+
+struct di_stack {
+ struct di_stack_elem *elems;
+ unsigned int num_elems;
+ enum di_stack_type type;
+ unsigned int flagidx;
+ unsigned int nomatch_depth;
+};
+
/*
* kcov descriptor (one per opened debugfs file).
* State transitions of the descriptor:
@@ -78,6 +94,18 @@ struct kcov {
int sequence;
int suppressed_stack_delta;
int suppressed_stack_mindelta;
+
+ /* delay injection */
+ struct {
+ DECLARE_BITMAP(sync_bits, NUM_SYNC_BITS);
+ struct di_stack *match_stacks;
+ unsigned int num_match_stacks;
+ u64 spin_limit;
+ unsigned int stack_used;
+ unsigned int shared_nomatch_depth;
+ unsigned int pending_sync_bit;
+ struct kcov *syncbits_owner;
+ } di;
};

struct kcov_remote_area {
@@ -252,12 +280,53 @@ void notrace __sanitizer_cov_trace_pc(void)
EXPORT_SYMBOL(__sanitizer_cov_trace_pc);

#ifdef CONFIG_KCOV_EXT_RECORDS
+static void notrace kcov_di_enter_slowpath(struct kcov *kcov, unsigned long ip)
+{
+ unsigned int i;
+ bool no_matches = true;
+
+ for (int need_increments = 0; need_increments < 2; need_increments++) {
+ for (i = 0; i < kcov->di.num_match_stacks; i++) {
+ struct di_stack *dis = &kcov->di.match_stacks[i];
+ struct di_stack_elem *next_elem;
+
+ if (dis->nomatch_depth || kcov->di.stack_used >= dis->num_elems-1) {
+no_match:
+ if (need_increments)
+ dis->nomatch_depth++;
+ continue;
+ }
+ next_elem = &dis->elems[kcov->di.stack_used];
+ if (next_elem->ip != ip)
+ goto no_match;
+ if (need_increments == 0)
+ next_elem->cur_parent_idx++;
+ if (next_elem->parent_idx != next_elem->cur_parent_idx-1)
+ goto no_match;
+
+ /* going a step down in the di_stack */
+ no_matches = false;
+ next_elem[1].cur_parent_idx = 0;
+ }
+
+ if (likely(need_increments == 0 && no_matches)) {
+ kcov->di.shared_nomatch_depth++;
+ return;
+ }
+ /* do second pass and increment individual nomatch counters */
+ }
+
+ kcov->di.stack_used++;
+}
+

void notrace __sanitizer_cov_trace_pc_entry(void)
{
struct task_struct *cur = current;

- unsigned long record = canonicalize_ip(_RET_IP_);
+ unsigned long ip = canonicalize_ip(_RET_IP_);
+ unsigned long record = ip;

unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);

bool ext_format;
+ struct kcov *kcov;

/*
* This hook replaces __sanitizer_cov_trace_pc() for the function entry

@@ -267,7 +336,7 @@ void notrace __sanitizer_cov_trace_pc_entry(void)

return;
if (kcov_mode & KCOV_IN_CTXSW) {
cur->kcov->suppressed_stack_delta++;

- return;
+ goto handle_distack;

}
ext_format = (kcov_mode & KCOV_EXT_FORMAT) != 0;

if (ext_format)
@@ -278,12 +347,25 @@ void notrace __sanitizer_cov_trace_pc_entry(void)
* enabled
*/

kcov_add_pc_record(cur, record, ext_format, (unsigned long)__builtin_return_address(1));

+
+handle_distack:
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ kcov = cur->kcov;
+ if (unlikely(kcov->di.num_match_stacks)) {
+ if (likely(kcov->di.shared_nomatch_depth > 0)) {
+ kcov->di.shared_nomatch_depth++;
+ } else {
+ kcov_di_enter_slowpath(kcov, ip);
+ }
+ }
+ }
}
void notrace __sanitizer_cov_trace_pc_exit(void)
{

struct task_struct *cur = current;

unsigned long record;

unsigned int kcov_mode = READ_ONCE(cur->kcov_mode);

+ struct kcov *kcov;

/*
* This hook is not called at the beginning of a basic block; the basic

@@ -300,10 +382,31 @@ void notrace __sanitizer_cov_trace_pc_exit(void)

if (kcov->suppressed_stack_mindelta == kcov->suppressed_stack_delta)

kcov->suppressed_stack_mindelta--;
kcov->suppressed_stack_delta--;
- return;
+ goto handle_distack;

}
record = (canonicalize_ip(_RET_IP_) & KCOV_RECORD_IP_MASK) | KCOV_RECORDFLAG_TYPE_EXIT;

kcov_add_pc_record(cur, record, false, 0);
+
+handle_distack:
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ kcov = cur->kcov;
+ if (unlikely(kcov->di.num_match_stacks)) {
+ if (likely(kcov->di.shared_nomatch_depth > 0)) {
+ kcov->di.shared_nomatch_depth--;
+ } else {
+ unsigned int i;
+
+ if (kcov->di.stack_used)
+ kcov->di.stack_used--;
+ for (i = 0; i < kcov->di.num_match_stacks; i++) {
+ struct di_stack *dis = &kcov->di.match_stacks[i];
+
+ if (dis->nomatch_depth > 0)
+ dis->nomatch_depth--;
+ }
+ }
+ }
+ }
}
#endif

@@ -458,6 +561,17 @@ static void kcov_start(struct task_struct *t, struct kcov *kcov,

unsigned int size, void *area, unsigned int mode,
int sequence)

{
+ int i;
+
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ /* delay injection */
+ kcov->di.stack_used = 0;
+ kcov->di.shared_nomatch_depth = 0;
+ kcov->di.pending_sync_bit = UINT_MAX;
+ for (i = 0; i < kcov->di.num_match_stacks; i++)
+ kcov->di.match_stacks[i].elems[0].cur_parent_idx = 0;
+ }
+

kcov_debug("t = %px, size = %u, area = %px\n", t, size, area);

t->kcov = kcov;
/* Cache in task struct for performance. */
@@ -547,6 +661,15 @@ static void kcov_get(struct kcov *kcov)
refcount_inc(&kcov->refcount);
}

+static void free_di_stacks(struct di_stack *di_stacks, unsigned int num_stacks)
+{
+ unsigned int i;
+
+ for (i = 0; i < num_stacks; i++)
+ kfree(di_stacks[i].elems);
+ kfree(di_stacks);
+}
+
static void kcov_put(struct kcov *kcov)
{
if (refcount_dec_and_test(&kcov->refcount)) {
@@ -555,6 +678,11 @@ static void kcov_put(struct kcov *kcov)
kcov_remote_reset(kcov);
vfree(kcov->area);
);
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY)) {
+ free_di_stacks(kcov->di.match_stacks, kcov->di.num_match_stacks);
+ if (kcov->di.syncbits_owner && kcov->di.syncbits_owner != kcov)
+ kcov_put(kcov->di.syncbits_owner);
+ }
kfree(kcov);
}
}
@@ -825,9 +953,180 @@ static int kcov_ioctl_locked(struct kcov *kcov, unsigned int cmd,
}
}

+static const struct file_operations kcov_fops;
+
+static int kcov_set_delay_injection(struct kcov *kcov, unsigned long arg_uaddr)
+{
+ struct kcov_set_di_arg arg;
+ struct di_stack *di_stacks;
+ int i, j;
+ int ret;
+ unsigned long flags;
+ struct file *syncbits_owner_file;
+ struct kcov *syncbits_owner;
+

+ if (!IS_ENABLED(CONFIG_KCOV_MEMORY))
+ return -ENOTSUPP;

+ if (copy_from_user(&arg, (void __user *)arg_uaddr, sizeof(arg)))
+ return -EFAULT;
+ if (arg.num_stacks > 128)
+ return -ERANGE;
+
+ /*
+ * This feature *intentionally* allows forcing the kernel to spinloop
+ * for a long time, including in contexts in which that would normally
+ * be a terrible idea.
+ * To prevent the user from causing a persistent system hang with this,
+ * cap the number of spinloop iterations.
+ */
+ if (arg.spin_limit > 10000000000)
+ return -ERANGE;
+
+ di_stacks = kmalloc_array(arg.num_stacks, sizeof(struct di_stack), GFP_KERNEL|__GFP_ZERO);
+ if (!di_stacks)
+ return -ENOMEM;
+
+ if (arg.sync_bits_fd != -1) {
+ syncbits_owner_file = fget(arg.sync_bits_fd);
+ if (!syncbits_owner_file) {
+ ret = -EBADF;
+ goto out_freestacks;
+ }
+ if (syncbits_owner_file->f_op != &kcov_fops ||
+ syncbits_owner_file->private_data == kcov) {
+ ret = -EBADF;
+ fput(syncbits_owner_file);
+ goto out_freestacks;
+ }
+ syncbits_owner = syncbits_owner_file->private_data;
+ kcov_get(syncbits_owner);
+ fput(syncbits_owner_file);
+
+ /*
+ * Ensure that the syncbits_owner does not, and can never,
+ * point to yet another KCOV instance.
+ */
+ spin_lock_irqsave(&syncbits_owner->lock, flags);
+ if (syncbits_owner->di.syncbits_owner &&
+ syncbits_owner->di.syncbits_owner != syncbits_owner) {
+ spin_unlock_irqrestore(&syncbits_owner->lock, flags);
+ ret = -ELOOP;
+ goto out_put_syncbits_owner;
+ }
+ if (!syncbits_owner->di.syncbits_owner)
+ syncbits_owner->di.syncbits_owner = syncbits_owner;
+ spin_unlock_irqrestore(&syncbits_owner->lock, flags);
+ } else {
+ syncbits_owner = kcov;
+ kcov_get(syncbits_owner);
+ }
+
+ for (i = 0; i < arg.num_stacks; i++) {
+ struct kcov_di_stack __user *user_stackp =
+ ((struct kcov_di_stack __user *)u64_to_user_ptr(arg.stacks)) + i;
+ struct kcov_di_stack u_di_stack;
+
+ if (copy_from_user(&u_di_stack, user_stackp, sizeof(struct kcov_di_stack))) {
+ ret = -EFAULT;
+ goto out_put_syncbits_owner;
+ }
+ if (u_di_stack.num_elems < 2 || u_di_stack.num_elems > 32 ||
+ u_di_stack.flagidx >= NUM_SYNC_BITS) {
+ ret = -ERANGE;
+ goto out_put_syncbits_owner;
+ }
+ if (u_di_stack.type != DI_STACK_WAIT && u_di_stack.type != DI_STACK_WAKE_PRE &&
+ u_di_stack.type != DI_STACK_WAKE_POST) {
+ ret = -EINVAL;
+ goto out_put_syncbits_owner;
+ }
+ di_stacks[i] = (struct di_stack) {
+ .elems = kmalloc_array(u_di_stack.num_elems, sizeof(struct di_stack_elem),
+ GFP_KERNEL),
+ .num_elems = u_di_stack.num_elems,
+ .type = u_di_stack.type,
+ .flagidx = u_di_stack.flagidx
+ };
+ if (!di_stacks[i].elems) {
+ ret = -ENOMEM;
+ goto out_put_syncbits_owner;
+ }
+ for (j = 0; j < u_di_stack.num_elems; j++) {
+ struct kcov_di_stack_elem __user *user_elemp =
+ ((struct kcov_di_stack_elem __user *)u_di_stack.elems) + j;
+ struct kcov_di_stack_elem user_elem;
+
+ if (copy_from_user(&user_elem, user_elemp, sizeof(user_elem))) {
+ ret = -EFAULT;
+ goto out_put_syncbits_owner;
+ }
+ di_stacks[i].elems[j] = (struct di_stack_elem) {
+ .ip = user_elem.ip,
+ .parent_idx = user_elem.parent_idx
+ };
+ }
+ }
+
+ spin_lock_irqsave(&kcov->lock, flags);
+ if (kcov->t) {
+ ret = -EBUSY;
+ } else if (kcov->di.syncbits_owner && kcov->di.syncbits_owner != syncbits_owner) {
+ ret = -EBADFD;
+ } else {
+ /* load config */
+ free_di_stacks(kcov->di.match_stacks, kcov->di.num_match_stacks);
+ kcov->di.match_stacks = di_stacks;
+ kcov->di.num_match_stacks = arg.num_stacks;
+ kcov->di.spin_limit = arg.spin_limit;
+ if (!kcov->di.syncbits_owner) {
+ /* Avoid reference loop. */
+ if (syncbits_owner != kcov)
+ kcov_get(syncbits_owner);
+ kcov->di.syncbits_owner = syncbits_owner;
+ }
+
+ ret = 0;
+ }
+ spin_unlock_irqrestore(&kcov->lock, flags);
+
+out_put_syncbits_owner:
+ kcov_put(syncbits_owner);
+out_freestacks:
+ if (ret)
+ free_di_stacks(di_stacks, arg.num_stacks);
+ return ret;
+}
+
+static int notrace __kcov_spin_wait(struct kcov *kcov, unsigned int flagidx)
+{
+ while (!test_bit(flagidx, kcov->di.syncbits_owner->di.sync_bits)) {
+ u64 spin_limit = READ_ONCE(kcov->di.spin_limit);
+
+ if (spin_limit == 0) /* spin timeout */
+ return -ETIMEDOUT;
+ WRITE_ONCE(kcov->di.spin_limit, spin_limit - 1);
+ cpu_relax();
+ }
+ return 0;
+}
+
+/*
+ * Look up kcov->syncbits_owner in a way that is safe is @kcov is not active on
+ * the current task.
+ */
+static struct kcov *get_syncbits_owner(struct kcov *kcov)
+{
+ guard(spinlock_irqsave)(&kcov->lock);
+
+ if (!kcov->di.syncbits_owner)
+ return NULL;
+ kcov_get(kcov->di.syncbits_owner);
+ return kcov->di.syncbits_owner;
+}
+
static long kcov_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
{
- struct kcov *kcov;
+ struct kcov *kcov, *syncbits_owner;
int res;
struct kcov_remote_arg *remote_arg = NULL;
unsigned int remote_num_handles;
@@ -862,6 +1161,29 @@ static long kcov_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
kcov->mode = KCOV_MODE_INIT;
spin_unlock_irqrestore(&kcov->lock, flags);
return 0;
+ case KCOV_SET_DI:
+ return kcov_set_delay_injection(kcov, arg);
+ case KCOV_RESET_DI_FLAGS:
+ case KCOV_WAKE_DI_FLAG:
+ case KCOV_SPINWAIT_DI_FLAG:

+ if (!IS_ENABLED(CONFIG_KCOV_MEMORY))
+ return -ENOTSUPP;

+ if (arg >= NUM_SYNC_BITS)
+ return -EINVAL;
+ syncbits_owner = get_syncbits_owner(kcov);
+ if (!syncbits_owner)
+ return -EINVAL;
+ if (cmd == KCOV_RESET_DI_FLAGS) {
+ bitmap_clear(syncbits_owner->di.sync_bits, 0, NUM_SYNC_BITS);
+ res = 0;
+ } else if (cmd == KCOV_WAKE_DI_FLAG) {
+ set_bit(arg, syncbits_owner->di.sync_bits);
+ res = 0;
+ } else {
+ res = __kcov_spin_wait(syncbits_owner, arg);
+ }
+ kcov_put(syncbits_owner);
+ return res;
case KCOV_REMOTE_ENABLE:
if (get_user(remote_num_handles, (unsigned __user *)(arg +
offsetof(struct kcov_remote_arg, num_handles))))
@@ -1254,9 +1576,56 @@ void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsign

struct task_struct *t = current;

struct memory_access_record *record;

unsigned int kcov_mode = READ_ONCE(t->kcov_mode);

+ struct kcov *kcov;
+ int di_wake_idx = -1;

if (kcov_mode != KCOV_MODE_TRACE_PC_AND_MEM || !check_kcov_context(t))

return;
+
+ kcov = t->kcov;
+ if (IS_ENABLED(CONFIG_KCOV_MEMORY) && unlikely(kcov->di.num_match_stacks)) {
+ if (unlikely(kcov->di.pending_sync_bit != UINT_MAX)) {
+ set_bit(kcov->di.pending_sync_bit, kcov->di.syncbits_owner->di.sync_bits);
+ kcov->di.pending_sync_bit = UINT_MAX;
+ }
+
+ if (unlikely(kcov->di.shared_nomatch_depth == 0)) {
+ /* similar to kcov_di_enter_slowpath */
+ unsigned int i;
+
+ for (i = 0; i < kcov->di.num_match_stacks; i++) {
+ struct di_stack *dis = &kcov->di.match_stacks[i];
+ struct di_stack_elem *elem;
+
+ if (dis->nomatch_depth || kcov->di.stack_used != dis->num_elems-1)
+ continue;
+ elem = &dis->elems[kcov->di.stack_used];
+ if (elem->ip != ret_ip)
+ continue;
+ if (elem->parent_idx != elem->cur_parent_idx++)
+ continue;
+ if (dis->type == DI_STACK_WAIT) {
+ unsigned long wait_record = KCOV_RECORDFLAG_TYPE_WAIT;
+
+ wait_record |= dis->flagidx;
+ if (__kcov_spin_wait(kcov->di.syncbits_owner, dis->flagidx))
+ wait_record |= KCOV_WAIT_TIMEOUT;
+ kcov_add_pc_record(t, wait_record, false, 0);
+ } else if (dis->type == DI_STACK_WAKE_PRE) {
+ kcov_add_pc_record(t,
+ KCOV_RECORDFLAG_TYPE_WAKE | dis->flagidx,
+ false, 0);
+ set_bit(dis->flagidx,
+ kcov->di.syncbits_owner->di.sync_bits);
+ } else {
+ /* DI_STACK_WAKE_POST */
+ di_wake_idx = dis->flagidx;
+ kcov->di.pending_sync_bit = dis->flagidx;
+ }
+ }
+ }
+ }
+
if (!kcov_get_memaccess_record(t, &record))
return;
*record = (struct memory_access_record) {
@@ -1287,6 +1656,9 @@ void notrace __kcov_handle_memaccess(const volatile void *p, size_t size, unsign
break;
}
handle_fault:;
+
+ if (unlikely(di_wake_idx != -1))
+ kcov_add_pc_record(t, KCOV_RECORDFLAG_TYPE_WAKE | di_wake_idx, false, 0);

}

void notrace _kcov_handle_memaccess(const volatile void *p, size_t size, unsigned int type)

diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 08ebfe1f3fac..30d748249b1c 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -1222,6 +1222,7 @@ static const char *uaccess_safe_builtin[] = {

"__kcov_handle_memaccess",
"__sanitizer_cov_trace_pc",
"__sanitizer_cov_trace_pc_entry",

+ "kcov_di_enter_slowpath",

"__sanitizer_cov_trace_pc_exit",
"__sanitizer_cov_trace_const_cmp1",
"__sanitizer_cov_trace_const_cmp2",

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

Document the new KCOV features CONFIG_KCOV_EXT_RECORDS and
CONFIG_KCOV_MEMORY.

Signed-off-by: Jann Horn <ja...@google.com>
---

Documentation/dev-tools/kcov.rst | 70 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)

diff --git a/Documentation/dev-tools/kcov.rst b/Documentation/dev-tools/kcov.rst
index 1a739290c8ec..41eef656ed90 100644
--- a/Documentation/dev-tools/kcov.rst
+++ b/Documentation/dev-tools/kcov.rst
@@ -383,3 +383,73 @@ local tasks spawned by the process and the global task that handles USB bus #1:
perror("close"), exit(1);
return 0;
}
+
+Extended trace format
+---------------------
+
+If the kernel is built with ``CONFIG_KCOV_EXT_RECORDS=y`` (which requires LLVM
+>=23.1.0), the ``KCOV_TRACE_PC_EXT`` mode can be used instead of
+``KCOV_TRACE_PC``.
+
+``KCOV_TRACE_PC_EXT`` uses the top byte of recorded PCs to store a record type.
+The function entry block is recorded with type ``KCOV_RECORDFLAG_TYPE_ENTRY``,
+and an additional record with ``KCOV_RECORDFLAG_TYPE_EXIT`` is generated on
+function exit.
+
+``KCOV_RECORDFLAG_TYPE_ENTRY`` records are immediately followed by the PC from
+which the call occurred.
+
+After code sections which have to temporarily stop emitting KCOV trace events,
+a ``KCOV_RECORDFLAG_TYPE_EESUM`` record summarizes the entry/exit events that
+happened.
+
+Together, these record types allow keeping track of the current stack trace.
+
+Memory access tracing
+---------------------
+
+If the kernel is built with ``CONFIG_KCOV_MEMORY=y`` (which depends on
+``CONFIG_KCOV_EXT_RECORDS=y``), the ``KCOV_TRACE_MEMORY_ACCESS`` mode can be
+used to produce a trace similar to ``KCOV_TRACE_PC_EXT``, but with additional
+``KCOV_RECORDFLAG_TYPE_MEMORY`` records that are emitted for every memory
+access.
+
+In such a trace, when a record with type ``KCOV_RECORDFLAG_TYPE_MEMORY`` is
+encountered, the trace element is a ``struct memory_access_record`` with a
+size returned by the ioctl ``KCOV_GET_MEMORY_RECORD_SIZE``.
+
+Delay injection
+---------------
+
+If the kernel is built with ``CONFIG_KCOV_MEMORY=y``, userspace can configure
+soft ordering constraints (like "this load on thread A should happen before that
+write happens on thread B") through the ioctl ``KCOV_SET_DI``, with an argument
+pointing to a ``struct kcov_set_di_arg``.
+The kernel will attempt to fulfill these ordering constraints by spin-waiting,
+with a configurable timeout ``spin_limit`` after which the kernel gives up on
+forcing the specified ordering.
+
+For each thread, userspace supplies an array of ``struct kcov_di_stack``
+elements, each of which describes an action to take at a specific call stack
+ending at an instrumented memory access.
+An action is one of:
+
+ - ``DI_STACK_WAKE_PRE``: "set synchronization bit N before this memory access"
+ - ``DI_STACK_WAKE_POST``: "set synchronization bit N after this memory access"
+ - ``DI_STACK_WAIT``: "spin-wait for synchronization bit N"
+
+These are normally paired between two threads: One thread sets synchronization
+bit N after the access at call stack A, another thread spin-waits for
+synchronization bit N before the access at call stack B, and this establishes an
+A-happens-before-B ordering.
+
+Since this involves multiple threads (and therefore multiple KCOV instances),
+the member ``sync_bits_fd`` in ``struct kcov_set_di_arg`` informs the kernel
+which KCOV instance holds the shared synchronization bits (where -1 means the
+current instance).
+
+Userspace can also directly interact with these synchronization bits using:
+
+ - ``KCOV_RESET_DI_FLAGS`` for zeroing all bits
+ - ``KCOV_WAKE_DI_FLAG`` for setting a specific bit
+ - ``KCOV_SPINWAIT_DI_FLAG`` for spin-waiting on a specific bit

--
2.55.0.979.g7e5102b832-goog

[Jann Horn]

On Tue, Sep 8, 2026 at 6:55 PM Jann Horn <ja...@google.com> wrote:
> To help with using CONFIG_KCOV_MEMORY for detecting use-after-free issues,
> log when memory (SLUB objects or page allocations) is being freed.
>
> This should happen after KASAN has already marked the memory as freed;
> this will become important if we allow driving delay injection off this
> in the future.

Oops, I should include the page allocator and SLUB people on this one
patch in the series, since I'm adding a hook in their code.