Re: KCSAN Support on ARM64 Kernel

Dmitry Vyukov

unread,

Oct 14, 2019, 4:40:55 AM10/14/19

to sgr...@codeaurora.org, Marco Elver, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern, Mark Rutland

On Mon, Oct 14, 2019 at 7:11 AM <sgr...@codeaurora.org> wrote:
>
> Hi Dmitry,
>
> I am from Qualcomm Linux Security Team, just going through KCSAN and found that there was a thread for arm64 support (https://lkml.org/lkml/2019/9/20/804).
>
> Can you please tell me if KCSAN is supported on ARM64 now? Can I just rebase the KCSAN branch on top of our let’s say android mainline kernel, enable the config and run syzkaller on that for finding race conditions?
>
> It would be very helpful if you reply, we want to setup this for finding issues on our proprietary modules that are not part of kernel mainline.
>
> Regards,
>
> Sachin Grover

+more people re KCSAN on ARM64

Marco Elver

unread,

Oct 14, 2019, 5:09:53 AM10/14/19

to Dmitry Vyukov, sgr...@codeaurora.org, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern, Mark Rutland

KCSAN does not yet have ARM64 support. Once it's upstream, I would
expect that Mark's patches (from repo linked in LKML thread) will just
cleanly apply to enable ARM64 support.

Thanks,
-- Marco

sgr...@codeaurora.org

unread,

Oct 14, 2019, 5:30:06 AM10/14/19

to Marco Elver, Dmitry Vyukov, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern, Mark Rutland

Hi Marco,

When can we expect upstream of KCSAN on kernel mainline. Any timeline?

Regards,
Sachin Grover

Marco Elver

unread,

Oct 14, 2019, 5:32:09 AM10/14/19

to sgr...@codeaurora.org, Dmitry Vyukov, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern, Mark Rutland

Hi Sachin,

My plan was to send patches upstream within the month.

Thanks,
-- Marco

Mark Rutland

unread,

Oct 14, 2019, 6:19:42 AM10/14/19

to Marco Elver, Dmitry Vyukov, sgr...@codeaurora.org, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern

Once the core kcsan bits are ready, I'll rebase the arm64 patch atop.
I'm expecting some things to change as part of review, so it'd be great
to see that posted ASAP.

For arm64 I'm not expecting major changes (other than those necessary to
handle the arm64 atomic rework that went in to v5.4-rc1)

FWIW, I was able to run Syzkaller atop of my arm64/kcsan branch, but
it's very noisy as it has none of the core fixes.

Thanks,
Mark.

sgr...@codeaurora.org

unread,

Dec 2, 2019, 12:07:28 AM12/2/19

to Mark Rutland, Marco Elver, Dmitry Vyukov, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern

Hi All,

Is there any update in Arm64 support of KCSAN.

Regards,
Sachin Grover

-----Original Message-----
From: Mark Rutland <mark.r...@arm.com>
Sent: Monday, 14 October, 2019 3:50 PM
To: Marco Elver <el...@google.com>
Cc: Dmitry Vyukov <dvy...@google.com>; sgr...@codeaurora.org; kasan-dev <kasa...@googlegroups.com>; LKML <linux-...@vger.kernel.org>; Paul E. McKenney <pau...@linux.ibm.com>; Will Deacon <willd...@google.com>; Andrea Parri <parri....@gmail.com>; Alan Stern <st...@rowland.harvard.edu>
Subject: Re: KCSAN Support on ARM64 Kernel

Marco Elver

unread,

Dec 2, 2019, 4:39:10 AM12/2/19

to sgr...@codeaurora.org, Mark Rutland, Dmitry Vyukov, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern

We're in the process of upstreaming KCSAN, which will simplify a lot
of things including Arm64 support. So far KCSAN is not yet in
mainline, but I assume when that happens (if things go well, very
soon) it should be trivial to add Arm64 support based on Mark's
prototype.

Thanks,
-- Marco

On Mon, 2 Dec 2019 at 06:07, <sgr...@codeaurora.org> wrote:
>
> Hi All,
>
> Is there any update in Arm64 support of KCSAN.
>
> Regards,
> Sachin Grover
>
> -----Original Message-----
> From: Mark Rutland <mark.r...@arm.com>
> Sent: Monday, 14 October, 2019 3:50 PM
> To: Marco Elver <el...@google.com>
> Cc: Dmitry Vyukov <dvy...@google.com>; sgr...@codeaurora.org; kasan-dev <kasa...@googlegroups.com>; LKML <linux-...@vger.kernel.org>; Paul E. McKenney <pau...@linux.ibm.com>; Will Deacon <willd...@google.com>; Andrea Parri <parri....@gmail.com>; Alan Stern <st...@rowland.harvard.edu>
> Subject: Re: KCSAN Support on ARM64 Kernel
>

Mukesh Ojha

unread,

Dec 12, 2019, 11:22:33 AM12/12/19

to Mark Rutland, Marco Elver, Dmitry Vyukov, sgr...@codeaurora.org, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern

Hi Mark,

Are the below patches enough for kcsan to be working on arm64 ?
I am not sure about the one you are mentioning about "atomic rework patches which went in 5.4 rc1" .

Thanks.
Mukesh

2019-10-03	arm64, kcsan: enable KCSAN for arm64arm64/kcsan	Mark Rutland	5	-1/+5

2019-09-24	locking/atomics, kcsan: Add KCSAN instrumentation	Marco Elver	2	-2/+199
2019-09-24	asm-generic, kcsan: Add KCSAN instrumentation for bitops	Marco Elver	1	-0/+18
2019-09-24	seqlock, kcsan: Add annotations for KCSAN	Marco Elver	1	-5/+42
2019-09-24	build, kcsan: Add KCSAN build exceptions	Marco Elver	3	-0/+17
2019-09-24	objtool, kcsan: Add KCSAN runtime functions to whitelist	Marco Elver	1	-0/+17
2019-09-24	kcsan: Add Kernel Concurrency Sanitizer infrastructure	Marco Elver

Mark Rutland

unread,

Dec 12, 2019, 12:27:11 PM12/12/19

to Mukesh Ojha, Marco Elver, Dmitry Vyukov, sgr...@codeaurora.org, kasan-dev, LKML, Paul E. McKenney, Will Deacon, Andrea Parri, Alan Stern

Hi Mukesh,

In future *please* reply in plaintext rather than HTML.

On Thu, Dec 12, 2019 at 04:22:30PM +0000, Mukesh Ojha wrote:
> On 10/14/2019 3:49 PM, Mark Rutland wrote:

> > Once the core kcsan bits are ready, I'll rebase the arm64 patch atop.
> > I'm expecting some things to change as part of review, so it'd be great
> > to see that posted ASAP.
> >
> > For arm64 I'm not expecting major changes (other than those necessary to
> > handle the arm64 atomic rework that went in to v5.4-rc1)
>

> Hi Mark,
>
> Are the below patches enough for kcsan to be working on arm64 ?

That depends on what branch you're using as a base. My arm64/kcsan
branch worked for me as-is, but as I mentioned that was /very/ noisy.
Both the kcsan code and the arm64 code have moved on since then, and I
have no idea how well that would backport.

I had a quick go at porting my arm64 patch atop the kcsan branch in
Paul's tree, and that doesn't get as far as producing earlycon output,
so more work will be necessary to investigate and debug that.

I hope to look at that, but I don't think I'll have the chance to do so
before the end of next week.

> I am not sure about the one you are mentioning about "atomic rework patches
> which went in 5.4 rc1" .

Marco Elver

unread,

Jun 15, 2020, 8:30:33 AM6/15/20

to sgr...@codeaurora.org, Dmitry Vyukov, kasan-dev, LKML, Paul E. McKenney, Andrea Parri, Alan Stern, Mark Rutland, Will Deacon

On Mon, 14 Oct 2019 at 11:31, Marco Elver <el...@google.com> wrote:
> My plan was to send patches upstream within the month.

[...]

> On Mon, 14 Oct 2019 at 11:30, <sgr...@codeaurora.org> wrote:

[...]

> > When can we expect upstream of KCSAN on kernel mainline. Any timeline?

[...]

> > > > Can you please tell me if KCSAN is supported on ARM64 now? Can I just rebase the KCSAN branch on top of our let’s say android mainline kernel, enable the config and run syzkaller on that for finding race conditions?

[...]

> > KCSAN does not yet have ARM64 support. Once it's upstream, I would expect that Mark's patches (from repo linked in LKML thread) will just cleanly apply to enable ARM64 support.

Just FYI, KCSAN is in mainline now. I believe porting it to other
architectures has also become much simpler due to its reworked
ONCE/atomic support relying on proper compiler instrumentation instead
of other tricks.

Thanks,
-- Marco

Marco Elver

unread,

Jul 10, 2020, 9:41:16 AM7/10/20

to sgr...@codeaurora.org, Mark Rutland, Will Deacon, Dmitry Vyukov, kasan-dev

[+Cc mailing list and other folks]

Hi Sachin,

On Fri, 10 Jul 2020 at 15:09, <sgr...@codeaurora.org> wrote:
> Are these all the KCSAN changes:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/kernel/kcsan
>
> And the same are applicable for arm64?

No, those aren't all KCSAN changes, those are only the core changes.
There are other changes, but unless they were in arch/, they will
apply to arm64 of course.

The the full list of changes up to the point KCSAN was merged can be
obtained with

git log locking-urgent-2020-06-11..locking-kcsan-2020-06-11

where both tags are on -tip
[https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git]. Note,
in case you're trying to backport this to an older kernel, I don't
recommend it because of all the ONCE changes that happened before the
merge. If you want to try and backport, we could dig out an older
pre-ONCE-rework version. Another reason I wouldn't recommend a
backport for now is because of all the unaddressed data races, and
KCSAN generally just throwing all kinds of (potentially already fixed
in mainline) reports at you.

On mainline, you could try to just cherry-pick Mark's patch from a few
months ago to enable one of the earlier KCSAN versions on arm64:
https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?h=arm64/kcsan&id=ae1d089527027ce710e464105a73eb0db27d7875

If you want to try this, I'd recommend also validating KCSAN works
using the kcsan-test module in -next:
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/kernel/kcsan/kcsan-test.c?id=1fe84fd4a4027a17d511a832f89ab14107650ba4

Thanks,
-- Marco

> -----Original Message-----
> From: Marco Elver <el...@google.com>
> Sent: Monday, 15 June, 2020 6:00 PM
> To: sgr...@codeaurora.org
> Cc: Dmitry Vyukov <dvy...@google.com>; kasan-dev <kasa...@googlegroups.com>; LKML <linux-...@vger.kernel.org>; Paul E. McKenney <pau...@linux.ibm.com>; Andrea Parri <parri....@gmail.com>; Alan Stern <st...@rowland.harvard.edu>; Mark Rutland <mark.r...@arm.com>; Will Deacon <wi...@kernel.org>
> Subject: Re: KCSAN Support on ARM64 Kernel
>

Mark Rutland

unread,

Jul 10, 2020, 9:57:57 AM7/10/20

to Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev

On Fri, Jul 10, 2020 at 03:41:03PM +0200, Marco Elver wrote:
> [+Cc mailing list and other folks]
>
> Hi Sachin,

Hi all,

> On Fri, 10 Jul 2020 at 15:09, <sgr...@codeaurora.org> wrote:
> > Are these all the KCSAN changes:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/kernel/kcsan
> >
> > And the same are applicable for arm64?
>
> No, those aren't all KCSAN changes, those are only the core changes.
> There are other changes, but unless they were in arch/, they will
> apply to arm64 of course.
>
> The the full list of changes up to the point KCSAN was merged can be
> obtained with
>
> git log locking-urgent-2020-06-11..locking-kcsan-2020-06-11
>
> where both tags are on -tip
> [https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git]. Note,
> in case you're trying to backport this to an older kernel, I don't
> recommend it because of all the ONCE changes that happened before the
> merge. If you want to try and backport, we could dig out an older
> pre-ONCE-rework version. Another reason I wouldn't recommend a
> backport for now is because of all the unaddressed data races, and
> KCSAN generally just throwing all kinds of (potentially already fixed
> in mainline) reports at you.
>
> On mainline, you could try to just cherry-pick Mark's patch from a few
> months ago to enable one of the earlier KCSAN versions on arm64:
> https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?h=arm64/kcsan&id=ae1d089527027ce710e464105a73eb0db27d7875

As a heads-up, since KCSAN now requires clang 11, I was waiting for the
release before sending the arm64 patch. I'd wanted to stress the result
locally with my arm64 Syzkaller instsance etc before sending it out, and
didn't fancy doing that from a locally-built clang on an arbitrary
commit.

If you think there'sa a sufficiently stable clang commit to test from,
I'm happy to give that a go.

Thanks,
Mark.

Marco Elver

unread,

Jul 10, 2020, 11:12:14 AM7/10/20

to Mark Rutland, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev

Thanks, Mark. LLVM/Clang is usually quite stable even the pre-release
(famous last words ;-)). We've been using LLVM commit
ca2dcbd030eadbf0aa9b660efe864ff08af6e18b
(https://github.com/llvm/llvm-project/commit/ca2dcbd030eadbf0aa9b660efe864ff08af6e18b).

There's also https://github.com/ClangBuiltLinux/tc-build, but I think
the version that one's pointing to is slightly older (May) and doesn't
yet have all the commits we want for KCSAN.

Thanks,
-- Marco

Mark Rutland

unread,

Jul 10, 2020, 1:53:10 PM7/10/20

to Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev

I built that locally, and rebased my arm64 enablement patches, but it
looks like there's a dodgy interaction with BTI, as the majority of
files produce a build-time warning:

| CC arch/arm64/kernel/psci.o
| warning: some functions compiled with BTI and some compiled without BTI
| warning: not setting BTI in feature flags

Regardless of whether the kernel has BTI and BTI_KERNEL selected it
doesn't produce any console output, but that may be something I need to
fix up and I haven't tried to debug it yet.

For now I've pushed out my rebased (and currently broken) patch to my
arm64/kcsan-new branch:

git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git arm64/kcsan-new

... with a note as to the brokenness.

Thanks,
Mark.

Marco Elver

unread,

Jul 13, 2020, 5:44:10 AM7/13/20

to Mark Rutland, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, clang-built-linux

[+Cc clang-built-linux]

Seems it's not KCSAN specific:
https://lore.kernel.org/linux-arm-kernel/20200507143332.GB1422@willie-the-truck/
and https://lore.kernel.org/lkml/202006191840.qO8NnNsK%25...@intel.com/

Thanks,
-- Marco

Mark Rutland

unread,

Jul 13, 2020, 7:27:26 AM7/13/20

to Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, clang-built-linux

Ah, so KCSAN tickles this because it happens to have generated
functions, judging by the propsoed fix:

https://reviews.llvm.org/D75181

... and practically speaking that means KCSAN isn't going to be usable
on arm64 until that's in. I'm not keen on making it mutually exclusive
with BTI as we've had to do for GCOV.

Regardless I'll try to dig into why it's failing to boot without BTI,
but it might be a few days before I have the free time.

Mark.

Mark Rutland

unread,

Jul 27, 2020, 1:58:59 PM7/27/20

to Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev

On Fri, Jul 10, 2020 at 06:53:09PM +0100, Mark Rutland wrote:
> On Fri, Jul 10, 2020 at 05:12:02PM +0200, Marco Elver wrote:
> > On Fri, 10 Jul 2020 at 15:57, Mark Rutland <mark.r...@arm.com> wrote:
> > > As a heads-up, since KCSAN now requires clang 11, I was waiting for the
> > > release before sending the arm64 patch. I'd wanted to stress the result
> > > locally with my arm64 Syzkaller instsance etc before sending it out, and
> > > didn't fancy doing that from a locally-built clang on an arbitrary
> > > commit.
> > >
> > > If you think there'sa a sufficiently stable clang commit to test from,
> > > I'm happy to give that a go.
> >
> > Thanks, Mark. LLVM/Clang is usually quite stable even the pre-release
> > (famous last words ;-)). We've been using LLVM commit
> > ca2dcbd030eadbf0aa9b660efe864ff08af6e18b
> > (https://github.com/llvm/llvm-project/commit/ca2dcbd030eadbf0aa9b660efe864ff08af6e18b).

> Regardless of whether the kernel has BTI and BTI_KERNEL selected it
> doesn't produce any console output, but that may be something I need to
> fix up and I haven't tried to debug it yet.

I had the chance to dig into this, and the issue was that some
instrumented code runs before we set up the per-cpu offset for the boot
CPU, and this ended up causing a recursive fault.

I have a preparatory patch to address that by changing the way we set up
the offset.

> For now I've pushed out my rebased (and currently broken) patch to my
> arm64/kcsan-new branch:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git arm64/kcsan-new

I've pushed out an updated branch with the preparatory patch, rebased
atop today's arm64 for-next/core branch. Note that due to the BTI issue
with generated functions this is still broken, and I won't be sending
this for review until that's fixed in clang.

Mark.

Marco Elver

unread,

Jul 27, 2020, 2:18:58 PM7/27/20

to Mark Rutland, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney

Great, thank you! Let's see which one comes first: BTI getting fixed
with Clang; or mainlining GCC support [1] and having GCC 11 released.
:-)

[1] https://lore.kernel.org/lkml/20200714173252.GA32057@paulmck-ThinkPad-P72/

Thanks,
-- Marco

sgr...@codeaurora.org

unread,

Sep 22, 2020, 1:03:38 AM9/22/20

to Marco Elver, Mark Rutland, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney

Hi Mark/Other Maintainers,

Is there any update on KCSAN for arm64 now?

Thanks,
Sachin

-----Original Message-----
From: Marco Elver <el...@google.com>
Sent: Monday, 27 July, 2020 11:49 PM
To: Mark Rutland <mark.r...@arm.com>
Cc: sgr...@codeaurora.org; Will Deacon <wi...@kernel.org>; Dmitry Vyukov <dvy...@google.com>; kasan-dev <kasa...@googlegroups.com>; Paul E. McKenney <pau...@kernel.org>
Subject: Re: KCSAN Support on ARM64 Kernel

Mark Rutland

unread,

Sep 23, 2020, 7:47:49 AM9/23/20

to sgr...@codeaurora.org, Marco Elver, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney

Hi,

On Tue, Sep 22, 2020 at 10:32:02AM +0530, sgr...@codeaurora.org wrote:
> Hi Mark/Other Maintainers,
>
> Is there any update on KCSAN for arm64 now?

Sorry for the delay on this -- I'm still working on this, but there are
a few issues. I've pushed out a WIP patchset to my arm64/kcsan branch:

git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git arm64/kcsan

If you'd like to give that a try, you might get by for now by disabling
LSE atomics in Kconfig -- see below for details.

The main issues are:

* Current builds of clang miscompile generated functions when BTI is
enabled, leading to build-time warnings (and potentially runtime
issues). I was hoping this was going to be fixed soon (and was
originally going to wait for the clang 11 release), but this seems to
be a larger structural issue with LLVM that we will have to workaround
for the timebeing.

This needs some Makefile/Kconfig work to forbid the combination of BTI
with any feature relying on compiler-generated functions, until clang
handles this correctly.

* KCSAN currently instruments some functions which are not safe to
instrument (e.g. code used during code patching, exception entry),
leading to crashes and hangs for common configurations (e.g. with LSE
atomics). This has also highlisted some existing issues in this area
(e.g. with other instrumentation).

I'm auditing and reworking code to address this, but I don't have a
good enough patch series yet. I intend to post that prework after rc1,
and hopefully the necessary bits are small enough that KCSAN can
follow in the same merge window.

Thanks,
Mark.

Marco Elver

unread,

Mar 1, 2021, 8:09:56 AM3/1/21

to Mark Rutland, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney

It's 2021, and I'd like to check if we have all the pieces in place
for KCSAN support on arm64. While it might not be terribly urgent
right now, I think we have all the blockers resolved.

On Wed, 23 Sept 2020 at 13:47, Mark Rutland <mark.r...@arm.com> wrote:
[...]

> The main issues are:
>
> * Current builds of clang miscompile generated functions when BTI is
> enabled, leading to build-time warnings (and potentially runtime
> issues). I was hoping this was going to be fixed soon (and was
> originally going to wait for the clang 11 release), but this seems to
> be a larger structural issue with LLVM that we will have to workaround
> for the timebeing.
>
> This needs some Makefile/Kconfig work to forbid the combination of BTI
> with any feature relying on compiler-generated functions, until clang
> handles this correctly.

I think https://reviews.llvm.org/D85649 fixed the BTI issue with
Clang. Or was there something else missing?

> * KCSAN currently instruments some functions which are not safe to
> instrument (e.g. code used during code patching, exception entry),
> leading to crashes and hangs for common configurations (e.g. with LSE
> atomics). This has also highlisted some existing issues in this area
> (e.g. with other instrumentation).
>
> I'm auditing and reworking code to address this, but I don't have a
> good enough patch series yet. I intend to post that prework after rc1,
> and hopefully the necessary bits are small enough that KCSAN can
> follow in the same merge window.

[...]

> > -----Original Message-----
> > From: Marco Elver <el...@google.com>
[...]
> > Let's see which one comes first: BTI getting fixed with Clang; or mainlining GCC support [1] and having GCC 11 released.

If Clang still has issues, KCSAN works with GCC 11, which will be
released this year.

Mark, was there anything else blocking?

Thanks,
-- Marco

Mark Rutland

unread,

Mar 2, 2021, 5:28:40 AM3/2/21

to Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney

On Mon, Mar 01, 2021 at 02:09:43PM +0100, Marco Elver wrote:
> It's 2021, and I'd like to check if we have all the pieces in place
> for KCSAN support on arm64. While it might not be terribly urgent
> right now, I think we have all the blockers resolved.
>
> On Wed, 23 Sept 2020 at 13:47, Mark Rutland <mark.r...@arm.com> wrote:
> [...]
> > The main issues are:
> >
> > * Current builds of clang miscompile generated functions when BTI is
> > enabled, leading to build-time warnings (and potentially runtime
> > issues). I was hoping this was going to be fixed soon (and was
> > originally going to wait for the clang 11 release), but this seems to
> > be a larger structural issue with LLVM that we will have to workaround
> > for the timebeing.
> >
> > This needs some Makefile/Kconfig work to forbid the combination of BTI
> > with any feature relying on compiler-generated functions, until clang
> > handles this correctly.
>
> I think https://reviews.llvm.org/D85649 fixed the BTI issue with
> Clang. Or was there something else missing?

I *think* so, but I haven't had a chance to go test with a recent clang
build. I see there's now as 11.1.0 build out on llvm.org, so I can try
to give that a spin in a bit, if no-one else does.

> > * KCSAN currently instruments some functions which are not safe to
> > instrument (e.g. code used during code patching, exception entry),
> > leading to crashes and hangs for common configurations (e.g. with LSE
> > atomics). This has also highlisted some existing issues in this area
> > (e.g. with other instrumentation).
> >
> > I'm auditing and reworking code to address this, but I don't have a
> > good enough patch series yet. I intend to post that prework after rc1,
> > and hopefully the necessary bits are small enough that KCSAN can
> > follow in the same merge window.

On this part, I know we still need to do a couple of things:

* Deal with instrumentation of early boot code. We need to set the
per-cpu offset earlier, and might also need to mark more of this as
noinstr.

I'll go respin the per-cpu offset patch in a moment as that's trivial.

* Prevent instrumentation of the patching/alternatives code, which I saw
blow up when instrumented. For KCSAN we can probably survive with a
simple refactoring and marking a few things as noinstr, but there's a
more general unsoundness problem here since the patching code calls
code whihc can be instrumented or patched (e.g. bitops, cache
maintenance, common ID register accessors), and making this watertight
will require some more invasive rework that I hadn't quite figured
out.

* I have a vague recollection that there was some problem with atomics,
and that in some cases we'd need to use arch_atomic() rather than
atomic(), but I can't remember whether that was to do with the
patching code or elsewhere.

> [...]
> > > -----Original Message-----
> > > From: Marco Elver <el...@google.com>
> [...]
> > > Let's see which one comes first: BTI getting fixed with Clang; or mainlining GCC support [1] and having GCC 11 released.
>
> If Clang still has issues, KCSAN works with GCC 11, which will be
> released this year.
>
> Mark, was there anything else blocking?

I think it's just the bits above, but I haven't had the chance to look
at this actively for a short while, so there might be more issues that
have cropped up since I last looked.

Thanks,
Mark.

Marco Elver

unread,

Mar 2, 2021, 5:46:48 AM3/2/21

to Mark Rutland, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney

Ok, that sounds like it's a bit more complicated then.

> * I have a vague recollection that there was some problem with atomics,
> and that in some cases we'd need to use arch_atomic() rather than
> atomic(), but I can't remember whether that was to do with the
> patching code or elsewhere.

I think this was inside noinstr functions, because ifdef ARCH_ATOMIC,
atomic ops are instrumented via atomic-instrumented.h.

> > [...]
> > > > -----Original Message-----
> > > > From: Marco Elver <el...@google.com>
> > [...]
> > > > Let's see which one comes first: BTI getting fixed with Clang; or mainlining GCC support [1] and having GCC 11 released.
> >
> > If Clang still has issues, KCSAN works with GCC 11, which will be
> > released this year.
> >
> > Mark, was there anything else blocking?
>
> I think it's just the bits above, but I haven't had the chance to look
> at this actively for a short while, so there might be more issues that
> have cropped up since I last looked.

Thanks for the summary above, we'll be patient as there's no rush. I
thought that the patches you had would hopefully "just work" with a
few minor additions, but it seems that was wishful thinking. :-)

Thanks,
-- Marco

Mark Rutland

unread,

Mar 2, 2021, 7:26:59 AM3/2/21

to Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney, Nick Desaulniers, Daniel Kiss

[Adding Nick and Daniel]

On Mon, Mar 01, 2021 at 02:09:43PM +0100, Marco Elver wrote:

> It's 2021, and I'd like to check if we have all the pieces in place
> for KCSAN support on arm64. While it might not be terribly urgent
> right now, I think we have all the blockers resolved.
>
> On Wed, 23 Sept 2020 at 13:47, Mark Rutland <mark.r...@arm.com> wrote:
> [...]
> > The main issues are:
> >
> > * Current builds of clang miscompile generated functions when BTI is
> > enabled, leading to build-time warnings (and potentially runtime
> > issues). I was hoping this was going to be fixed soon (and was
> > originally going to wait for the clang 11 release), but this seems to
> > be a larger structural issue with LLVM that we will have to workaround
> > for the timebeing.
> >
> > This needs some Makefile/Kconfig work to forbid the combination of BTI
> > with any feature relying on compiler-generated functions, until clang
> > handles this correctly.
>
> I think https://reviews.llvm.org/D85649 fixed the BTI issue with
> Clang. Or was there something else missing?

I just had a go with the clang+llvm 11.0.1 binary release, and it looks
like there's still some brokenness. Building v5.12-rc1 with defconfig +
CONFIG_KCSAN I get a stream of warnings of the form:

| warning: some functions compiled with BTI and some compiled without BTI
| warning: not setting BTI in feature flags

I took a look at arch/arm64/kernel/setup.o with objdump, and while
almost all functions begin with a PACIASP (which can act like a BTI),
there's a generated constructor function with neither a BTI nor a
PACIASP:

| 000000000000010c <tsan.module_ctor>:
| 10c: 14000000 b 0 <__tsan_init>

... IIUC this is a case that D85649 intended to fix, but missed? I
assume that D85649 is part of 11.0.1?

The resulting kernel does link, but won't boot (due to the Linux
structural issues I mentioned previously).

Thanks,
Mark.

Nick Desaulniers

unread,

Mar 2, 2021, 12:09:31 PM3/2/21

to Mark Rutland, Marco Elver, sgr...@codeaurora.org, Will Deacon, Dmitry Vyukov, kasan-dev, Paul E. McKenney, Daniel Kiss

$ git branch -a --contains a88c722e6 | grep release
remotes/origin/release/12.x
Looks like it landed in clang-12 (branched, but not released yet).

>
> The resulting kernel does link, but won't boot (due to the Linux
> structural issues I mentioned previously).
>
> Thanks,
> Mark.

--
Thanks,
~Nick Desaulniers

Reply all

Reply to author

Forward