[syzbot] [bpf?] KCSAN: data-race in bpf_percpu_array_update / bpf_percpu_array_update (2)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 727dbda16b83 Merge tag 'hardening-v6.6-rc1' of git://git.k..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=136f39dfa80000
kernel config: https://syzkaller.appspot.com/x/.config?x=dea9c2ce3f646a25
dashboard link: https://syzkaller.appspot.com/bug?extid=97522333291430dd277f
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/9923a023ab11/disk-727dbda1.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/650dbc695d77/vmlinux-727dbda1.xz
kernel image: https://storage.googleapis.com/syzbot-assets/361da71276bf/bzImage-727dbda1.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+975223...@syzkaller.appspotmail.com

==================================================================
BUG: KCSAN: data-race in bpf_percpu_array_update / bpf_percpu_array_update

write to 0xffffe8fffe7425d8 of 8 bytes by task 8257 on cpu 1:
bpf_long_memcpy include/linux/bpf.h:428 [inline]
bpf_obj_memcpy include/linux/bpf.h:441 [inline]
copy_map_value_long include/linux/bpf.h:464 [inline]
bpf_percpu_array_update+0x3bb/0x500 kernel/bpf/arraymap.c:380
bpf_map_update_value+0x190/0x370 kernel/bpf/syscall.c:175
generic_map_update_batch+0x3ae/0x4f0 kernel/bpf/syscall.c:1749
bpf_map_do_batch+0x2df/0x3d0 kernel/bpf/syscall.c:4648
__sys_bpf+0x28a/0x780
__do_sys_bpf kernel/bpf/syscall.c:5241 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5239 [inline]
__x64_sys_bpf+0x43/0x50 kernel/bpf/syscall.c:5239
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd

write to 0xffffe8fffe7425d8 of 8 bytes by task 8268 on cpu 0:
bpf_long_memcpy include/linux/bpf.h:428 [inline]
bpf_obj_memcpy include/linux/bpf.h:441 [inline]
copy_map_value_long include/linux/bpf.h:464 [inline]
bpf_percpu_array_update+0x3bb/0x500 kernel/bpf/arraymap.c:380
bpf_map_update_value+0x190/0x370 kernel/bpf/syscall.c:175
generic_map_update_batch+0x3ae/0x4f0 kernel/bpf/syscall.c:1749
bpf_map_do_batch+0x2df/0x3d0 kernel/bpf/syscall.c:4648
__sys_bpf+0x28a/0x780
__do_sys_bpf kernel/bpf/syscall.c:5241 [inline]
__se_sys_bpf kernel/bpf/syscall.c:5239 [inline]
__x64_sys_bpf+0x43/0x50 kernel/bpf/syscall.c:5239
do_syscall_x64 arch/x86/entry/common.c:50 [inline]
do_syscall_64+0x41/0xc0 arch/x86/entry/common.c:80
entry_SYSCALL_64_after_hwframe+0x63/0xcd

value changed: 0x0000000000000000 -> 0xfffffff000002788

Reported by Kernel Concurrency Sanitizer on:
CPU: 0 PID: 8268 Comm: syz-executor.4 Not tainted 6.5.0-syzkaller-00453-g727dbda16b83 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
==================================================================


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

[Yonghong Song]

This case is with two tasks doing bpf_map batch update together for the
same map and key.

> write to 0xffffe8fffe7425d8 of 8 bytes by task 8257 on cpu 1:

> write to 0xffffe8fffe7425d8 of 8 bytes by task 8268 on cpu 0:

So concurrency is introduced by user applications.
In my opinion, this probably not an issue from kernel perspective.

[Marco Elver]

Perhaps not, but I recall there being a discussion about making KCSAN
aware of memory accesses done by BPF programs (memcpy being a tiny
subset of those). Not sure if the above data race qualifies as
something we might want to still detect, i.e. a kernel dev testing
their kernel might be interested in such a report.

Regardless, in this case we should teach syzkaller to ignore KCSAN
data races that originate from bpf user operations whatever the
origin.

[Daniel Borkmann]

I presume KCSAN could be silenced here via READ_ONCE/WRITE_ONCE conversion?

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index f58895830ada..32c4a37045f2 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -424,8 +424,11 @@ static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)
long *ldst = dst;

size /= sizeof(long);
- while (size--)
- *ldst++ = *lsrc++;
+ while (size--) {
+ WRITE_ONCE(*ldst, READ_ONCE(*lsrc));
+ ldst++;
+ lsrc++;
+ }
}

/* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */

[Marco Elver]

In this case, maybe data_race(*ldst++ = *lsrc++) would be more
appropriate and efficient. Unlikely that READ_ONCE()/WRITE_ONCE() here
helps make this any safer, i.e. the memcpy is still not atomic and if
it's a bug on the user's side, it'll corrupt data either way.

[Daniel Borkmann]

On 8/29/23 10:36 PM, Marco Elver wrote:
[...]

> In this case, maybe data_race(*ldst++ = *lsrc++) would be more
> appropriate and efficient. Unlikely that READ_ONCE()/WRITE_ONCE() here
> helps make this any safer, i.e. the memcpy is still not atomic and if
> it's a bug on the user's side, it'll corrupt data either way.

I wasn't aware of data_race(), thanks for the suggestion! Just flushed out
a small patch.

Thanks,
Daniel