Re: [PATCH 0/5 v16] KASan for Arm

[Nathan Chancellor]

On Mon, Oct 19, 2020 at 10:41:35AM +0200, Linus Walleij wrote:
> This is the 16th and final (knock on wood) version of
> KASan for ARM32.
>
> Changes since v15:
>
> - Things now work on all boards we have tested on including
> Broadcom and i.MX6Q.
>
> - Folded in a fix from Ard to PAGE_ALIGN() the end of
> mappings making everything work on all Broadcom board.
>
> - Folded in a fix from Ahmad Fatoum making things work
> with fortify on i.MX6Q.
>
> - Testing and testing and testing on build servers.
>
> - We are good to go.
>
> I will now put this in Russell's patch tracker for v5.11.
>
> There is a git branch you can pull in:
> https://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-integrator.git/log/?h=kasan
>
> This branch includes Ard's two patches already in Russell's
> patch tracker.
>
>
> Abbott Liu (1):
> ARM: Define the virtual space of KASan's shadow region
>
> Andrey Ryabinin (3):
> ARM: Disable KASan instrumentation for some code
> ARM: Replace string mem* functions for KASan
> ARM: Enable KASan for ARM
>
> Linus Walleij (1):
> ARM: Initialize the mapping of KASan shadow memory
>
> Documentation/arm/memory.rst | 5 +
> Documentation/dev-tools/kasan.rst | 4 +-
> .../features/debug/KASAN/arch-support.txt | 2 +-
> arch/arm/Kconfig | 10 +
> arch/arm/boot/compressed/Makefile | 1 +
> arch/arm/boot/compressed/string.c | 19 ++
> arch/arm/include/asm/kasan.h | 33 ++
> arch/arm/include/asm/kasan_def.h | 81 +++++
> arch/arm/include/asm/memory.h | 5 +
> arch/arm/include/asm/pgalloc.h | 8 +-
> arch/arm/include/asm/string.h | 26 ++
> arch/arm/include/asm/thread_info.h | 8 +
> arch/arm/include/asm/uaccess-asm.h | 2 +-
> arch/arm/kernel/entry-armv.S | 3 +-
> arch/arm/kernel/entry-common.S | 9 +-
> arch/arm/kernel/head-common.S | 7 +-
> arch/arm/kernel/setup.c | 2 +
> arch/arm/kernel/unwind.c | 6 +-
> arch/arm/lib/memcpy.S | 3 +
> arch/arm/lib/memmove.S | 5 +-
> arch/arm/lib/memset.S | 3 +
> arch/arm/mm/Makefile | 5 +
> arch/arm/mm/kasan_init.c | 292 ++++++++++++++++++
> arch/arm/mm/mmu.c | 18 ++
> arch/arm/mm/pgd.c | 16 +-
> arch/arm/vdso/Makefile | 2 +
> 26 files changed, 561 insertions(+), 14 deletions(-)
> create mode 100644 arch/arm/include/asm/kasan.h
> create mode 100644 arch/arm/include/asm/kasan_def.h
> create mode 100644 arch/arm/mm/kasan_init.c
>
> --
> 2.26.2
>

After this series was applied and available in -next, ARCH=arm LLVM=1
allyesconfig builds started failing:

$ make -skj"$(nproc)" ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- KCONFIG_ALLCONFIG=<(echo CONFIG_CPU_BIG_ENDIAN=n) LLVM=1 distclean allyesconfig vmlinux
...
ld.lld: error: section: .exit.data is not contiguous with other relro sections
...

$ git bisect ld: [3f267ec60b922eff2a5c90d532357a39f155b730] Add linux-next specific files for 20201029
# good: [23859ae44402f4d935b9ee548135dd1e65e2cbf4] Merge tag 'trace-v5.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
git bisect start '3f267ec60b922eff2a5c90d532357a39f155b730' '23859ae44402f4d935b9ee548135dd1e65e2cbf4'
# bad: [bfa70a4ea4bfa6f87b58cf8b90b88297389c92b7] Merge remote-tracking branch 'mtd/mtd/next' into master
git bisect bad bfa70a4ea4bfa6f87b58cf8b90b88297389c92b7
# bad: [37a292dcf77532547f335ed5063d9169031c9b08] Merge remote-tracking branch 'sunxi/sunxi/for-next' into master
git bisect bad 37a292dcf77532547f335ed5063d9169031c9b08
# good: [e6d922c77db276a16f0b7933c2a9951dc9c0052c] Merge remote-tracking branch 'drm-misc-fixes/for-linux-next-fixes' into master
git bisect good e6d922c77db276a16f0b7933c2a9951dc9c0052c
# bad: [cbe49fbb8f6c8d29bc1d9a5a9a742ef2c2eb6320] Merge remote-tracking branch 'mvebu/for-next' into master
git bisect bad cbe49fbb8f6c8d29bc1d9a5a9a742ef2c2eb6320
# bad: [d0e12484e7e1ede73c538744cdbe9439f7335d01] Merge remote-tracking branch 'arm-soc/for-next' into master
git bisect bad d0e12484e7e1ede73c538744cdbe9439f7335d01
# good: [24a23387c15f34bad2485a9e1c3b7ac6f0fb35a6] Merge branch 'asm-generic-cleanup' into asm-generic
git bisect good 24a23387c15f34bad2485a9e1c3b7ac6f0fb35a6
# good: [3a8eb4d3421a2ca0f95ac3b1a8f012940d4f0d52] Merge remote-tracking branch 'kbuild/for-next' into master
git bisect good 3a8eb4d3421a2ca0f95ac3b1a8f012940d4f0d52
# bad: [20f96e606509ee5084690179afe1810b95617a92] Merge branches 'fixes' and 'misc' into for-next
git bisect bad 20f96e606509ee5084690179afe1810b95617a92
# good: [d6d51a96c7d63b7450860a3037f2d62388286a52] ARM: 9014/2: Replace string mem* functions for KASan
git bisect good d6d51a96c7d63b7450860a3037f2d62388286a52
# good: [5615f69bc2097452ecc954f5264d784e158d6801] ARM: 9016/2: Initialize the mapping of KASan shadow memory
git bisect good 5615f69bc2097452ecc954f5264d784e158d6801
# bad: [fc2933c133744305236793025b00c2f7d258b687] ARM: 9020/1: mm: use correct section size macro to describe the FDT virtual address
git bisect bad fc2933c133744305236793025b00c2f7d258b687
# bad: [421015713b306e47af95d4d61cdfbd96d462e4cb] ARM: 9017/2: Enable KASan for ARM
git bisect bad 421015713b306e47af95d4d61cdfbd96d462e4cb
# first bad commit: [421015713b306e47af95d4d61cdfbd96d462e4cb] ARM: 9017/2: Enable KASan for ARMog

An allyesconfig kernel compiled with clang does not link properly with
ld.bfd without a workaround [1], which I do not have time to apply and
test now but can later if it is relevant. I have not done any triage on
this yet either but I wanted to get the report out in case there is
anything obvious.

[1]: https://github.com/ClangBuiltLinux/linux/issues/325

Cheers,
Nathan

[Nick Desaulniers]

relro? smells like:
https://lore.kernel.org/lkml/20201016175339.242...@google.com/T/#u

--
Thanks,
~Nick Desaulniers

[Nathan Chancellor]

Huh, did not even realize that the error messages were the same, my bad!

This issue is simple enough to produce by just adding CONFIG_KASAN=y to
multi_v7_defconfig. I tried adding '-z norelro' to LDFLAGS_vmlinux in
arch/arm/Makefile and it fixes the build error but the resulting kernel
does not boot in QEMU.

$ make -skj"$(nproc)" ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 distclean defconfig

$ scripts/config -e KASAN

$ make -skj"$(nproc)" ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 olddefconfig zImage

$ ~/cbl/github/boot-utils/boot-qemu.sh -a arm32_v7 -k .
/home/nathan/cbl/github/boot-utils/images/arm/rootfs.cpio.zst: 3176448 bytes
+ timeout --foreground 3m unbuffer qemu-system-arm -machine virt -no-reboot -append 'console=ttyAMA0 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/arm/rootfs.cpio -kernel /home/nathan/src/linux-next/arch/arm/boot/zImage -m 512m -nodefaults -serial mon:stdio
+ RET=124
+ set +x

'-z norelro' boots fine without KASAN so I assume there is something up
specifically with KASAN and LLVM for ARM.

Cheers,
Nathan

[Ard Biesheuvel]

On Fri, 30 Oct 2020 at 02:32, Nathan Chancellor

How long did you wait for it to boot? Booting a KASAN kernel under
QEMU emulation is going to be *very* slow.

[Nathan Chancellor]

The timeout command above is 3m and there was no output from the kernel
in that amount of time. What kind of time should I be reasonably
expecting?

Cheers,
Nathan

[Ard Biesheuvel]

On Fri, 30 Oct 2020 at 08:57, Nathan Chancellor

3 minutes is probably long enough.

Do you see any output with 'earlycon=pl011,0x9000000' added to the
kernel command line?

[Nathan Chancellor]

Yes.

$ ~/cbl/github/boot-utils/boot-qemu.sh -a arm32_v7 -k .
/home/nathan/cbl/github/boot-utils/images/arm/rootfs.cpio.zst: 3176448 bytes

+ timeout --foreground 3m unbuffer qemu-system-arm -machine virt -no-reboot -append 'console=ttyAMA0 earlycon=pl011,0x9000000 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/arm/rootfs.cpio -kernel /home/nathan/src/linux-next/arch/arm/boot/zImage -m 512m -nodefaults -serial mon:stdio
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.10.0-rc1-next-20201029-dirty (nathan@ubuntu-m3-large-x86) (ClangBuiltLinux clang version 12.0.0 (https://github.com/llvm/llvm-project 1df8d7b4f23da304061bf30b617132f8ba9ab80a), LLD 12.0.0 (https://github.com/llvm/llvm-project 1df8d7b4f23da304061bf30b617132f8ba9ab80a)) #1 SMP Thu Oct 29 18:26:00 MST 2020
[ 0.000000] CPU: ARMv7 Processor [412fc0f1] revision 1 (ARMv7), cr=10c5387d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[ 0.000000] OF: fdt: Machine model: linux,dummy-virt
[ 0.000000] earlycon: pl11 at MMIO 0x09000000 (options '')
[ 0.000000] printk: bootconsole [pl11] enabled
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] efi: UEFI not found.
[ 0.000000] cma: Reserved 64 MiB at 0x5c000000
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000040000000-0x000000005fffffff]
[ 0.000000] Normal empty
[ 0.000000] HighMem empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000040000000-0x000000005fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000005fffffff]
[ 0.000000] kasan: Mapping kernel virtual memory block: c0000000-e0000000 at shadow: b7000000-bb000000
[ 0.000000] kasan: Mapping kernel virtual memory block: bf000000-c0000000 at shadow: b6e00000-b7000000
[ 0.000000] kasan: Kernel address sanitizer initialized
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv0.2 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: Trusted OS migration not required
[ 0.000000] percpu: Embedded 20 pages/cpu s49676 r8192 d24052 u81920
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 130048
[ 0.000000] Kernel command line: console=ttyAMA0 earlycon=pl011,0x9000000
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 329852K/524288K available (21504K kernel code, 8544K rwdata, 14676K rodata, 2048K init, 4843K bss, 128900K reserved, 65536K cma-reserved, 0K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=1.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] GICv2m: range[mem 0x08020000-0x08020fff], SPI[80:143]
[ 0.000000] random: get_random_bytes called from start_kernel+0x208/0x3d0 with crng_init=0
[ 0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[ 0.000156] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[ 0.000565] Switching to timer-based delay loop, resolution 16ns
[ 0.006124] Console: colour dummy device 80x30
[ 0.007840] Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=625000)
[ 0.011577] pid_max: default: 32768 minimum: 301
[ 0.014889] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.015189] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.043926] CPU: Testing write buffer coherency: ok
[ 0.045820] CPU0: Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable
[ 0.057716] /cpus/cpu@0 missing clock-frequency property
[ 0.058105] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.070417] Setting up static identity map for 0x40300000 - 0x403000ac
[ 0.075105] rcu: Hierarchical SRCU implementation.
[ 0.084843] EFI services will not be available.
[ 0.087124] smp: Bringing up secondary CPUs ...
[ 0.087384] smp: Brought up 1 node, 1 CPU
[ 0.087617] SMP: Total of 1 processors activated (125.00 BogoMIPS).
[ 0.087878] CPU: All CPU(s) started in SVC mode.
[ 0.105778] devtmpfs: initialized
[ 0.283349] VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
[ 0.310896] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.311679] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.312469] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.312803] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.313125] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.313445] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.313761] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.314082] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.314400] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.314718] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.315031] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.315338] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.315661] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.315979] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.316296] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.316608] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.316923] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.317234] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.317549] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.317860] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.318177] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.318493] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.318809] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.319133] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.319457] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.319774] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.320091] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.320593] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.320909] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.321220] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.321532] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.321844] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.322157] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.322475] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.322788] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.323098] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.323424] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.323738] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.324050] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.324358] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.324673] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.324988] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.325303] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.325624] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.325941] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.326255] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.326573] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.326883] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.327193] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.327510] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.327826] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.328138] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.328457] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.328767] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.329081] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.329400] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.329711] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.330021] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.330326] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.330780] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.331104] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.331417] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.331733] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.332054] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.332381] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.332696] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.333001] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.333320] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.333648] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.333964] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.334272] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.334580] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.334894] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.335212] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.335528] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.335846] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.336162] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.336482] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.336804] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.337124] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.337443] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.337761] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.338087] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.338414] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.338756] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.339074] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.339396] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.339737] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.340061] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.340379] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.340695] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.341161] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.341504] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.341843] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.342151] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.342492] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.342809] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.343129] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.343445] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.343761] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.344079] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.344410] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.344755] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.345066] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.345383] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.345719] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.346031] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.346350] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.346664] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.346976] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.347289] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.347642] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.347952] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.348275] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.348618] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.348930] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.349245] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.349563] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.349872] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.350191] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.350508] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.350832] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.351280] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.351632] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.351940] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.352257] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.352604] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.352918] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.353228] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.353543] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.353855] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.354170] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.354503] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.354833] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.355144] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.355477] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.355811] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.356133] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.356439] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.356757] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.357075] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.357381] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.357726] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.358037] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.358345] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.358663] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.358979] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.359287] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.359599] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.359911] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.360230] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.360549] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.360860] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.361313] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.361642] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.361953] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.362262] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.362578] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.362888] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.363205] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.363520] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.363821] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.364139] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.364445] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.364759] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.365070] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.365384] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.365695] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.366003] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.366306] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.366611] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.366924] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.367248] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.367569] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.367876] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.368197] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.368515] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.368829] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.369137] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.369454] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.369772] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.370079] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.370393] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.370709] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.371023] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.371490] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.371808] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.372122] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.372435] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 0.372767] futex hash table entries: 256 (order: 2, 16384 bytes, linear)

Then there is no output after that.

Cheers,
Nathan

[Ard Biesheuvel]

On Fri, 30 Oct 2020 at 09:04, Nathan Chancellor

Ouch. This looks like futex_init() is being called over and over
again, which is a core_initcall.

This will need some dissecting of the binary, but we need to figure
out what is going on in the initcall dispatch code.

If you build with symbols, you should be able to run it in the
debugger. Just set a breakpoint on futex_init() and step through the
code until you can observe how/why it is being called a second time.

[Nathan Chancellor]

I have no idea how it is happening but futex_init keeps getting called
from within itself, if I am reading gdb right?

(gdb) hbreak futex_init
Hardware assisted breakpoint 1 at 0xc2727fac: file kernel/futex.c, line 4028.
(gdb) c
Continuing.

Breakpoint 1, futex_init () at kernel/futex.c:4028
4028 {
(gdb) n
4035 futex_hashsize = roundup_pow_of_two(256 * num_possible_cpus());
(gdb)
4038 futex_queues = alloc_large_system_hash("futex", sizeof(*futex_queues),
(gdb)
4035 futex_hashsize = roundup_pow_of_two(256 * num_possible_cpus());
(gdb)
4038 futex_queues = alloc_large_system_hash("futex", sizeof(*futex_queues),
(gdb)
4043 futex_hashsize = 1UL << futex_shift;
(gdb)
4045 futex_detect_cmpxchg();
(gdb)

Breakpoint 1, futex_init () at kernel/futex.c:4028
4028 {
(gdb)
4035 futex_hashsize = roundup_pow_of_two(256 * num_possible_cpus());
(gdb)
4038 futex_queues = alloc_large_system_hash("futex", sizeof(*futex_queues),
(gdb)
4035 futex_hashsize = roundup_pow_of_two(256 * num_possible_cpus());
(gdb)
4038 futex_queues = alloc_large_system_hash("futex", sizeof(*futex_queues),
(gdb)
4043 futex_hashsize = 1UL << futex_shift;
(gdb)
4045 futex_detect_cmpxchg();
(gdb)

If I step into futex_detect_cmpxchg(), I see us jump into vector_dabt
then back out futex_init.

(gdb) hbreak futex_init
Hardware assisted breakpoint 1 at 0xc2727fac: file kernel/futex.c, line 4028.
(gdb) hbreak futex_detect_cmpxchg
Hardware assisted breakpoint 2 at 0xc27280c8: file kernel/futex.c, line 4008.
(gdb) c
Continuing.

Breakpoint 1, futex_init () at kernel/futex.c:4028
4028 {
(gdb) c
Continuing.

Breakpoint 2, futex_detect_cmpxchg () at kernel/futex.c:4008
4008 {
(gdb) n
4022 if (cmpxchg_futex_value_locked(&curval, NULL, 0, 0) == -EFAULT)
(gdb)
vector_dabt () at arch/arm/kernel/entry-armv.S:1099
1099 vector_stub dabt, ABT_MODE, 8
(gdb)
__dabt_svc () at arch/arm/kernel/entry-armv.S:194
194 svc_entry uaccess=0
(gdb)
__dabt_svc () at arch/arm/kernel/entry-armv.S:195
195 mov r2, sp
(gdb)
196 dabt_helper
(gdb)
198 svc_exit r5 @ return from exception
(gdb)
0xc0458040 in ?? ()
(gdb)
Cannot find bounds of current function
(gdb) finish
Run till exit from #0 0xc0458040 in ?? ()

Breakpoint 1, futex_init () at kernel/futex.c:4028
4028 {
(gdb)

I am going to be honest, I do not really understand what is going on
here but I am more than happy to provide whatever information is needed
to further explore this.

Cheers,
Nathan

[Arnd Bergmann]

I can't explain it, but I'd point out that futex_detect_cmpxchg() has caused
problems in the past, with multiple patches for it proposed in the past
and none of them merged. One of the patches I had sent for it was:

commit 8232a8ffc332fa6a50296a51c4d85200a747256c
Author: Arnd Bergmann <ar...@arndb.de>
Date: Mon Mar 4 17:33:00 2019 +0100

[SUBMITTED 20190307] futex: mark futex_detect_cmpxchg() as 'noinline'

On 32-bit ARM, I got a link failure in futex_init() when building
with clang in some random configurations:

kernel/futex.o:(.text.fixup+0x5c): relocation truncated to fit:
R_ARM_JUMP24 against `.init.text'

As far as I can tell, the problem is that a branch is over 16MB
apart in those configurations, but only if it branches back to
the init text.

Marking the futex_detect_cmpxchg() function as noinline and
not __init avoids the problem for me.

Signed-off-by: Arnd Bergmann <ar...@arndb.de>

diff --git a/kernel/futex.c b/kernel/futex.c
index e646661f6282..6fbbed45f51b 100644
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -4061,7 +4061,7 @@ SYSCALL_DEFINE6(futex_time32, u32 __user *,
uaddr, int, op, u32, val,
}
#endif /* CONFIG_COMPAT_32BIT_TIME */

-static void __init futex_detect_cmpxchg(void)
+static noinline void futex_detect_cmpxchg(void)
{
#ifndef CONFIG_HAVE_FUTEX_CMPXCHG
u32 curval;

I forgot what the problem was, but you might try that patch, or find
the previous discussions in the archive.

Arnd

[Nathan Chancellor]

As it turns out, that is the patch that fixes the ld.bfd issue that I
mentioned earlier in this thread and it fixes this issue as well.

To summarize, with the below diff, I can successfully boot
multi_v7_defconfig + CONFIG_KASAN=y when LLVM=1 is used (and I see KASAN
get initialized properly).

Cheers,
Nathan

diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index 4d76eab2b22d..3c0a64cefe52 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -10,7 +10,7 @@
#
# Copyright (C) 1995-2001 by Russell King

-LDFLAGS_vmlinux := --no-undefined -X --pic-veneer
+LDFLAGS_vmlinux := --no-undefined -X --pic-veneer -z norelro
ifeq ($(CONFIG_CPU_ENDIAN_BE8),y)
LDFLAGS_vmlinux += --be8
KBUILD_LDFLAGS_MODULE += --be8
diff --git a/kernel/futex.c b/kernel/futex.c
index be68ac0d49ad..226bb20d175f 100644
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -4004,7 +4004,7 @@ SYSCALL_DEFINE6(futex_time32, u32 __user *, uaddr, int, op, u32, val,

[Fāng-ruì Sòng]

On Fri, Oct 30, 2020 at 2:09 AM Nathan Chancellor

> --
> You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clang-built-li...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20201030090945.GA3635064%40ubuntu-m3-large-x86.

About "ARM: Replace string mem* functions for KASan", it added .weak
memcpy to arch/arm/lib/memcpy.S
However, the convention is to change .weak + ENTRY(...) to WEAK(...)

In particular, clang 12 will reject `.weak memcpy; .global memcpy`
(they had different semantics in GNU as and LLVM integrated assembler
for a long time AND binutils maintainer agreed that .weak and .global
are strange and should not be used - though binutils will not make the
change to cause disruption)

Please see https://github.com/ClangBuiltLinux/linux/issues/1190

[Nick Desaulniers]

On Wed, Nov 4, 2020 at 4:30 PM Fāng-ruì Sòng <mas...@google.com> wrote:
>
> About "ARM: Replace string mem* functions for KASan", it added .weak
> memcpy to arch/arm/lib/memcpy.S
> However, the convention is to change .weak + ENTRY(...) to WEAK(...)
>
> In particular, clang 12 will reject `.weak memcpy; .global memcpy`
> (they had different semantics in GNU as and LLVM integrated assembler
> for a long time AND binutils maintainer agreed that .weak and .global
> are strange and should not be used - though binutils will not make the
> change to cause disruption)
>
> Please see https://github.com/ClangBuiltLinux/linux/issues/1190

It's straightforward for us to fixup; Fangrui has sent patches for
x86, perf, and 64b arm already which have been accepted. I forget
though what is the convention for "Fixes:" tags for patches in -next?
The SHAs change when pulled, right?
--
Thanks,
~Nick Desaulniers

[Ard Biesheuvel]

No, only when rebasing. It depends on the maintainer how likely that
is to happen for changes that are in -next, but usually the commit IDs
are stable once pulled into -next.

[Mike Rapoport]

With a small exception of mm tree ;-)

--
Sincerely yours,
Mike.

[Nick Desaulniers]

On Thu, Oct 29, 2020 at 6:32 PM Nathan Chancellor

<natecha...@gmail.com> wrote:
>
> On Thu, Oct 29, 2020 at 05:38:22PM -0700, Nick Desaulniers wrote:
> > On Thu, Oct 29, 2020 at 5:29 PM Nathan Chancellor
> > <natecha...@gmail.com> wrote:
> > >
> > > After this series was applied and available in -next, ARCH=arm LLVM=1
> > > allyesconfig builds started failing:
> > >
> > > $ make -skj"$(nproc)" ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- KCONFIG_ALLCONFIG=<(echo CONFIG_CPU_BIG_ENDIAN=n) LLVM=1 distclean allyesconfig vmlinux
> > > ...
> > > ld.lld: error: section: .exit.data is not contiguous with other relro sections
> > > ...
> >

> > relro? smells like:
> > https://lore.kernel.org/lkml/20201016175339.242...@google.com/T/#u
> >
>
> Huh, did not even realize that the error messages were the same, my bad!
>
> This issue is simple enough to produce by just adding CONFIG_KASAN=y to
> multi_v7_defconfig. I tried adding '-z norelro' to LDFLAGS_vmlinux in
> arch/arm/Makefile and it fixes the build error but the resulting kernel
> does not boot in QEMU.

> '-z norelro' boots fine without KASAN so I assume there is something up
> specifically with KASAN and LLVM for ARM.

Regardless, we still want `-z norelro`. Can you please send a patch if
you still have the diff locally?
--
Thanks,
~Nick Desaulniers

[Nathan Chancellor]

When linking a multi_v7_defconfig + CONFIG_KASAN=y kernel with
LD=ld.lld, the following error occurs:

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 zImage

ld.lld: error: section: .exit.data is not contiguous with other relro sections

LLD defaults to '-z relro', which is unneeded for the kernel because
program headers are not used nor is there any position independent code
generation or linking for ARM. Add '-z norelro' to LDFLAGS_vmlinux to
avoid this error.

Link: https://github.com/ClangBuiltLinux/linux/issues/1189
Signed-off-by: Nathan Chancellor <natecha...@gmail.com>
---
arch/arm/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index 4d76eab2b22d..3c0a64cefe52 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -10,7 +10,7 @@
#
# Copyright (C) 1995-2001 by Russell King

-LDFLAGS_vmlinux := --no-undefined -X --pic-veneer
+LDFLAGS_vmlinux := --no-undefined -X --pic-veneer -z norelro
ifeq ($(CONFIG_CPU_ENDIAN_BE8),y)
LDFLAGS_vmlinux += --be8
KBUILD_LDFLAGS_MODULE += --be8

base-commit: a0796429c6abecf8afaeb65b1db286af1fb579d1
--
2.29.2

[Nick Desaulniers]

On Mon, Nov 9, 2020 at 5:56 PM Nathan Chancellor
<natecha...@gmail.com> wrote:
>
> When linking a multi_v7_defconfig + CONFIG_KASAN=y kernel with
> LD=ld.lld, the following error occurs:
>
> $ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 zImage
> ld.lld: error: section: .exit.data is not contiguous with other relro sections
>
> LLD defaults to '-z relro', which is unneeded for the kernel because
> program headers are not used nor is there any position independent code
> generation or linking for ARM. Add '-z norelro' to LDFLAGS_vmlinux to
> avoid this error.
>
> Link: https://github.com/ClangBuiltLinux/linux/issues/1189
> Signed-off-by: Nathan Chancellor <natecha...@gmail.com>

Reviewed-by: Nick Desaulniers <ndesau...@google.com>
Tested-by: Nick Desaulniers <ndesau...@google.com>

Thanks for the patch!
See also: https://lore.kernel.org/linux-arm-kernel/20201016175339.242...@google.com/
where we just did the same thing for aarch64.

> ---
> arch/arm/Makefile | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/arm/Makefile b/arch/arm/Makefile
> index 4d76eab2b22d..3c0a64cefe52 100644
> --- a/arch/arm/Makefile
> +++ b/arch/arm/Makefile
> @@ -10,7 +10,7 @@
> #
> # Copyright (C) 1995-2001 by Russell King
>
> -LDFLAGS_vmlinux := --no-undefined -X --pic-veneer
> +LDFLAGS_vmlinux := --no-undefined -X --pic-veneer -z norelro
> ifeq ($(CONFIG_CPU_ENDIAN_BE8),y)
> LDFLAGS_vmlinux += --be8
> KBUILD_LDFLAGS_MODULE += --be8
>
> base-commit: a0796429c6abecf8afaeb65b1db286af1fb579d1
> --
> 2.29.2

--
Thanks,
~Nick Desaulniers

[Nick Desaulniers]

On Mon, Nov 9, 2020 at 6:05 PM Nick Desaulniers <ndesau...@google.com> wrote:
>
> On Mon, Nov 9, 2020 at 5:56 PM Nathan Chancellor
> <natecha...@gmail.com> wrote:
> >
> > When linking a multi_v7_defconfig + CONFIG_KASAN=y kernel with
> > LD=ld.lld, the following error occurs:
> >
> > $ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- LLVM=1 zImage
> > ld.lld: error: section: .exit.data is not contiguous with other relro sections
> >
> > LLD defaults to '-z relro', which is unneeded for the kernel because
> > program headers are not used nor is there any position independent code
> > generation or linking for ARM. Add '-z norelro' to LDFLAGS_vmlinux to
> > avoid this error.
> >
> > Link: https://github.com/ClangBuiltLinux/linux/issues/1189
> > Signed-off-by: Nathan Chancellor <natecha...@gmail.com>
>
> Reviewed-by: Nick Desaulniers <ndesau...@google.com>
> Tested-by: Nick Desaulniers <ndesau...@google.com>
>
> Thanks for the patch!
> See also: https://lore.kernel.org/linux-arm-kernel/20201016175339.242...@google.com/
> where we just did the same thing for aarch64.

I was thinking more about this last night. If we're going to be
playing whack-a-mole for each architecture with this, would it be
worthwhile for us to raise this up to the top level Makefile?
--
Thanks,
~Nick Desaulniers

[Nathan Chancellor]

Sure, I can send a patch along tomorrow that adds '-z norelro' to
LDFLAGS_vmlinux in the top level Makefile.

Cheers,
Nathan

[Nick Desaulniers]

Does this need to get submitted to
https://www.armlinux.org.uk/developer/patches/section.php?section=0
since `-z norelro` will produce warnings in ld.bfd for architectures
that don't support that (the results of the thread on adding it
globally)?

On Wed, Nov 11, 2020 at 6:52 PM Nathan Chancellor

--
Thanks,
~Nick Desaulniers