Tagged pointers, MTE and Pointer Authentication Code

[John Dallman]

Preface: I am not producing an Android app. I work for a software component business, creating shared libraries, compiled from C and C++ code, for use in third-party customers' apps. I test my libraries in a command-line test harness, which I run in the ADB shell. I am only producing software for 64-bit ARM, because none of the customers want 32-bit code. 

I'm looking to "harden" my Android native code libraries in the future, and there are several different systems that use upper bits of 64-bit pointers. I'm a bit confused by them, but this is what I think they do:

Tagged Pointers (https://source.android.com/docs/security/test/tagged-pointers) put a tag in the top byte of 64-bit pointers from memory allocations on Android 11 onwards. This tag is checked when memory is freed. This is mostly to catch apps that incorrectly store data in the top byte of pointers, in preparation for ARM Memory Tagging Extensions (MTE) becoming available in ARM 8.5, which uses the same bits of the pointer.

Pointer Authentication (PA) appeared at ARM 8.3 and puts a short cryptographic signature in high bits of pointers. Qualcomm's document on their implementation (https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/pointer-auth-v7.pdf) implies that it uses all the bits that aren't part of the virtual address, less one which is presumably for indicating that PA is in use. 

This seems to indicate that PA and TP/MTE can't be used together. Is that correct? If so, which is the preferred choice?

Thanks, 

John

[Dan Albert]

+Elliott Hughes and +Florian Mayer know the details better, but I can answer in broad strokes:

You probably want hwasan rather than MTE. MTE is the best choice for hardening your code in the field. It's lightweight enough that it's reasonable to ship with it on for some use cases, but it won't catch every bug every time. It's a mitigation rather than a bug finding tool. I don't actually know how middleware would approach that. I think you can build your code with it and it'd still be up to the app that consumes your components whether it's active or not? Or maybe you'd need to build two variants if you wanted to allow app developers to choose that (enh probably knows).

HWAsan, OTOH, is the one you want for testing. MTE is probabilistic so it will not detect memory errors deterministically. HWAsan will. It's a bit of a hassle to set up right now in that you need to flash a special system image (those are available for pixel devices, idk about other OEMs). That's actually changing with Android U and NDK r26 though: you'll be able to run hwasan binaries on a regular Android image (and it should work on all LP64 SKUs rather than just pixels).

There's a third thing that I know nothing about and won't fit your needs, but I'm mentioning it here for anyone else lurking :) https://developer.android.com/ndk/guides/gwp-asan

Our docs linked above apparently do absolutely nothing to help you figure out which tool you ought to use (if you're approaching this with zero prior knowledge it's not even clear that they're alternatives). I'll start drafting a fix for that.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqgm0f3nGNJ42pEXqobf1DLcQ1FcSfcMU-sTAkTx4PX5mtg%40mail.gmail.com.

[enh]

yeah, practically speaking:

1. if you care about security (as you seem to), enable PAC and BTI
today, and ship binaries with PAC and BTI enabled.
2. test locally with HWASan.

BTI (which you don't mention, but which is usually mentioned in the
same breath as PAC) uses instructions that are in the hint space on
earlier processors (and thus no-ops) to ensure that you can't jump to
arbitrary instructions --- a branch instruction has to land on one of
these hints. (and there are multiple different ones, to narrow it down
further.)

there are two ways to use PAC --- there is a variant where there's a
new instruction that does a check-and-return-if-valid, or there's the
variant we use where there's a check-and-abort (that's in the hint
space again, so a no-op on cpus that are too old) followed by a
regular return.

(if you/your customers are in control of _all_ the code in the
process, there's also shadow stack. we don't recommend that more
widely because "none of the code in my process ever clobbers x18"
isn't an easy thing for most people to check.)

(danalbert already ninja'ed me with what i was going to say about
hwasan, so i'll just say "what he said".)

MTE code, on the other hand, requires MTE hardware. so that's a very
specialist niche right now.

but to answer your _explicit_ question, "no, PAC and MTE aren't
incompatible ... you just get fewer bits when they have to share". but
for now, just enable PAC.

(and if you _do_ happen to be in the aforementioned niche where you're
working with a hardware vendor who has MTE hardware, let us know
because i suspect our MTE folks would love to chat :-) )

[Florian Mayer]

If you want to run on currently available phones, HWASan would be the way to go. To slightly correct Dan: HWASan is also probabilistic for some bugs (use-after-free after the pointer had been malloced and freed again inbetween, and non-linear buffer overflows), but with a higher chance than MTE (255 / 256 rather than 15 / 16).

For running HWASan binaries on your phone, you could use Android 14 Beta 3 or newer, and compile them with -fsanitize=hwasan (and Android target triple). Then you should be able to run them as normal. For the best experience I would recommend using the NDK 26 Beta.  Add -Wl,-dynamic-linker=/system/bin/bootstrap/linker_hwasan64 to your LDFLAGS as well, this is only needed until the final NDK 26 is released.

[John Dallman]

Dan Albert wrote:

> HWAsan, OTOH, is the one you want for testing. MTE is probabilistic so it will not detect memory 

> errors deterministically. HWAsan will. It's a bit of a hassle to set up right now in that you need to 

> flash a special system image (those are available for pixel devices, idk about other OEMs). That's 

> actually changing with Android U and NDK r26 though: you'll be able to run hwasan binaries on a 

> regular Android image (and it should work on all LP64 SKUs rather than just pixels).

OK, I'll look at that when I move up to testing on Android 14 or later. 

enh wrote: 

> 2. test locally with HWASan.

As above. 

> 1. if you care about security (as you seem to), enable PAC and BTI today, and ship binaries 

> with PAC and BTI enabled.
>

> BTI (which you don't mention, but which is usually mentioned in the
> same breath as PAC) uses instructions that are in the hint space on
> earlier processors (and thus no-ops) to ensure that you can't jump to
> arbitrary instructions --- a branch instruction has to land on one of
> these hints. (and there are multiple different ones, to narrow it down
> further.)

I've been quite cautious about PAC. It needs Linux kernel 5.0 or later to handle PAC traps, and I'm supporting Android 9 onwards, so I can definitely be running on a 4.0 kernel. I'm not clear what happens if a PAC trap goes off on a device where the kernel doesn't support it, and I don't want to find out from customers. 

What is the compiler option for BTI? Does it require ARMv8.3, or something later? Come to that, what are the compiler options for the two kinds of PAC? 

> there are two ways to use PAC --- there is a variant where there's a
> new instruction that does a check-and-return-if-valid, or there's the
> variant we use where there's a check-and-abort (that's in the hint
> space again, so a no-op on cpus that are too old) followed by a
> regular return.

> but to answer your _explicit_ question, "no, PAC and MTE aren't
> incompatible ... you just get fewer bits when they have to share". but
> for now, just enable PAC.
> 
> (and if you _do_ happen to be in the aforementioned niche where you're
> working with a hardware vendor who has MTE hardware, let us know
> because i suspect our MTE folks would love to chat :-) )

Afraid not, 

My testing hardware is a bit unusual. I have a lot of testing, because we test the Android version of the modeler as thoroughly as the Windows and Linux versions. It's helpful to be able to run it all on one device and that needs Cortex-X level performance. However, if you run a 'phone with a Cortex-X series core flat out for most of a day, you find that the power input over USB 3.x can't keep the battery charged. The device runs out of power and stops. I've tried an Asus device with two USB ports, but it would not charge on its high-capacity port and talk to the host computer on the low-capacity port at the same time, I also tried a OnePlus device with wireless charging, and the charging stopped as soon as I connected the device to the host computer. Both seemed to share their i/o hardware between several uses to save a few cents.

   

I'm currently using hardware development kit devices sold by Qualcomm <https://developer.qualcomm.com/hardware/snapdragon-8-gen-1-hdk>. They don't have batteries, and their mains power adapters will run them indefinitely at full speed. They also can't suffer bulged batteries.

Florian Meyer wrote: 

> For running HWASan binaries on your phone, you could use Android 14 Beta 3 or newer, and compile them with 

> fsanitize=hwasan (and Android target triple). Then you should be able to run them as normal. For the best 

> experience I would recommend using the NDK 26 Beta.  

> Add -Wl,-dynamic-linker=/system/bin/bootstrap/linker_hwasan64 to your LDFLAGS as well, this is only needed 

> until the final NDK 26 is released.

Thanks! 

John

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAFVaGhvsntPSUbd9ZwEwdFTVO%2B7eDcmuFHm3iaWvyX9oJDvtRw%40mail.gmail.com.

[enh]

On Wed, Aug 2, 2023 at 2:03 AM John Dallman <jgdats...@gmail.com> wrote:
>
> Dan Albert wrote:
>
> > HWAsan, OTOH, is the one you want for testing. MTE is probabilistic so it will not detect memory
> > errors deterministically. HWAsan will. It's a bit of a hassle to set up right now in that you need to
> > flash a special system image (those are available for pixel devices, idk about other OEMs). That's
> > actually changing with Android U and NDK r26 though: you'll be able to run hwasan binaries on a
> > regular Android image (and it should work on all LP64 SKUs rather than just pixels).
>
> OK, I'll look at that when I move up to testing on Android 14 or later.
>
> enh wrote:
>
> > 2. test locally with HWASan.
>
> As above.
>
> > 1. if you care about security (as you seem to), enable PAC and BTI today, and ship binaries
> > with PAC and BTI enabled.
> >
> > BTI (which you don't mention, but which is usually mentioned in the
> > same breath as PAC) uses instructions that are in the hint space on
> > earlier processors (and thus no-ops) to ensure that you can't jump to
> > arbitrary instructions --- a branch instruction has to land on one of
> > these hints. (and there are multiple different ones, to narrow it down
> > further.)
>
> I've been quite cautious about PAC. It needs Linux kernel 5.0 or later to handle PAC traps, and I'm supporting Android 9 onwards, so I can definitely be running on a 4.0 kernel. I'm not clear what happens if a PAC trap goes off on a device where the kernel doesn't support it, and I don't want to find out from customers.

it won't "go off" in the first place, because it won't be enabled by
the OS if you have too old a cpu and/or kernel.

> What is the compiler option for BTI? Does it require ARMv8.3, or something later? Come to that, what are the compiler options for the two kinds of PAC?

-mbranch-protection=standard gets you PAC and BTI.

(the decision of whether to use the hint-space PAC instructions or the
side-effect PAC instructions is implicit, based on your target cpu. so
unless you're telling the compiler "don't just assume the NDK
baseline, you can assume higher than that", it'll use the hint-space
instructions.)

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqg%3DkaEwer2JHTxBt63PL3-rz6opoqy5ECJRJoMRO3WoORQ%40mail.gmail.com.

[Dan Albert]

The new doc as promised: https://developer.android.com/ndk/guides/memory-debug. lmk if there's anything that needs to be improved :)

(haven't caught up on the rest of the thread yet, so I'll be back tomorrow if there's anything I need to answer, but I didn't want to keep you waiting for the doc until I'd had time to catch up)

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAJgzZorvHaaco0g_tJDmfg_9Cbe6qEnBy8XdwvfkfoiD0uk4bg%40mail.gmail.com.

[Florian Mayer]

On Wed, 2 Aug 2023 at 02:03, John Dallman <jgdats...@gmail.com> wrote:

Afraid not, 

My testing hardware is a bit unusual. I have a lot of testing, because we test the Android version of the modeler as thoroughly as the Windows and Linux versions. It's helpful to be able to run it all on one device and that needs Cortex-X level performance. However, if you run a 'phone with a Cortex-X series core flat out for most of a day, you find that the power input over USB 3.x can't keep the battery charged. The device runs out of power and stops. I've tried an Asus device with two USB ports, but it would not charge on its high-capacity port and talk to the host computer on the low-capacity port at the same time, I also tried a OnePlus device with wireless charging, and the charging stopped as soon as I connected the device to the host computer. Both seemed to share their i/o hardware between several uses to save a few cents.

   

I'm currently using hardware development kit devices sold by Qualcomm <https://developer.qualcomm.com/hardware/snapdragon-8-gen-1-hdk>. They don't have batteries, and their mains power adapters will run them indefinitely at full speed. They also can't suffer bulged batteries.

If you have ARM64 machines (sadly not virtual machines, because they do not support nested virtualization) you might also be interested in using cuttlefish to run virtual devices. That should have many of the same advantages as your Qualcomm dev kit, but with more general purpose hardware.

[John Dallman]

> If you have ARM64 machines (sadly not virtual machines, because they do not support nested 

> virtualization) you might also be interested in using cuttlefish to run virtual devices. That should 

> have many of the same advantages as your Qualcomm dev kit, but with more general purpose 

> hardware.

I do have some ARM64 Linux machines, but they're rather slower than the Qualcomm development kit, and already pretty busy with my ARM Linux builds and tests. 

(If anyone reading along has not heard of Cuttlefish, here's the introduction page https://source.android.com/docs/setup/create/cuttlefish) 

John

--

You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAJAyTCC_SOA_9T%3DpdzgTQgK0t8iKYHCtmFDaaw%2B2LAx%3Dn-isJw%40mail.gmail.com.

[John Dallman]

> > I've been quite cautious about PAC. It needs Linux kernel 5.0 or later to handle 

> > PAC traps, and I'm supporting Android 9 onwards, so I can definitely be running 

> > on a 4.0 kernel. I'm not clear what happens if a PAC trap goes off on a device 

> > where the kernel doesn't support it, and I don't want to find out from customers.

> it won't "go off" in the first place, because it won't be enabled by
> the OS if you have too old a cpu and/or kernel.

Aha! That would be worth adding to the documentation, unless it's already there and I missed it. 

> > What is the compiler option for BTI? Does it require ARMv8.3, or something 

> later? Come to that, what are the compiler options for the two kinds of PAC?
> 
> -mbranch-protection=standard gets you PAC and BTI.

Good-oh. 

> (the decision of whether to use the hint-space PAC instructions or the
> side-effect PAC instructions is implicit, based on your target cpu. so
> unless you're telling the compiler "don't just assume the NDK
> baseline, you can assume higher than that", it'll use the hint-space
> instructions.)

Got it. 

So, to sum up, I can turn on PAC and BTI in my NDK23 builds with -mbranch-protection=standard. I'm using the default instruction set, so the extra instructions will be in hint space. 

My Qualcomm test devices are ARMv9 and running a 5.10 kernel, so I can set off PAC/BTI traps and see what happens. PAC traps default to disabled. The kernel won't turn them on unless (a) it can handle them and (b) it is running on hardware that supports them. Is all that correct? 

I need to be able to set off PAC and BTI traps deliberately, for testing. 

To get off a PAC trap, an easy way seems to be to take a pointer to a function, check that it has some non-zero high bits, zero those bits, and call the function via that pointer. 

To set off a BTI trap, I take a pointer to a function, increment it by one instruction, and call the function via that pointer. The program counter counts bytes, so to move it on by one instruction, I need to add 4. 

Does that make sense?       

Thanks very much,

John

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAJgzZorvHaaco0g_tJDmfg_9Cbe6qEnBy8XdwvfkfoiD0uk4bg%40mail.gmail.com.

[Dan Albert]

> it won't "go off" in the first place, because it won't be enabled by
> the OS if you have too old a cpu and/or kernel.

Aha! That would be worth adding to the documentation, unless it's already there and I missed it. 

It's in both sets of docs. The new doc:

 

Pros:
Can be enabled in all builds without causing problems on older devices

Just sent a patch to add "or kernels"

The old doc:

Android uses PAC/BTI instructions that do nothing on older processors that don't support the new instructions. Only ARMv9 devices will have the PAC/BTI protection, but you can run the same code on ARMv8 devices too: no need for multiple variants of your library.

If either was non-obvious, lmk and I'll fix it :) 

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqg%3DWUW5hH1%3DMr4ezOYyJ8dv%3DimViKYJtSkUEZnyV0CEccQ%40mail.gmail.com.

[Dan Albert]

s/old doc/detailed doc/ s/new doc/overview doc/

[enh]

On Mon, Aug 7, 2023 at 4:20 AM John Dallman <jgdats...@gmail.com> wrote:
>
> > > I've been quite cautious about PAC. It needs Linux kernel 5.0 or later to handle
> > > PAC traps, and I'm supporting Android 9 onwards, so I can definitely be running
> > > on a 4.0 kernel. I'm not clear what happens if a PAC trap goes off on a device
> > > where the kernel doesn't support it, and I don't want to find out from customers.
>
> > it won't "go off" in the first place, because it won't be enabled by
> > the OS if you have too old a cpu and/or kernel.
>
> Aha! That would be worth adding to the documentation, unless it's already there and I missed it.
>
>
> > > What is the compiler option for BTI? Does it require ARMv8.3, or something
> > later? Come to that, what are the compiler options for the two kinds of PAC?
> >
> > -mbranch-protection=standard gets you PAC and BTI.
>
> Good-oh.
>
> > (the decision of whether to use the hint-space PAC instructions or the
> > side-effect PAC instructions is implicit, based on your target cpu. so
> > unless you're telling the compiler "don't just assume the NDK
> > baseline, you can assume higher than that", it'll use the hint-space
> > instructions.)
>
> Got it.
>
> So, to sum up, I can turn on PAC and BTI in my NDK23 builds with -mbranch-protection=standard. I'm using the default instruction set, so the extra instructions will be in hint space.
>
> My Qualcomm test devices are ARMv9 and running a 5.10 kernel, so I can set off PAC/BTI traps and see what happens. PAC traps default to disabled. The kernel won't turn them on unless (a) it can handle them and (b) it is running on hardware that supports them. Is all that correct?

correct, though you'll also need a version of Android that's enabling
PAC/BTI. (i think it may be on as far back as S, but definitely T.)

> I need to be able to set off PAC and BTI traps deliberately, for testing.
>
> To get off a PAC trap, an easy way seems to be to take a pointer to a function, check that it has some non-zero high bits, zero those bits, and call the function via that pointer.

no, because it's a _return_ address you're trying to mess up. (BTI is
the one for calls.) i'm not sure there's an easy way that isn't
assembler?

> To set off a BTI trap, I take a pointer to a function, increment it by one instruction, and call the function via that pointer. The program counter counts bytes, so to move it on by one instruction, I need to add 4.

yeah, that should work. (though obviously that's likely to crash for
_other_ reasons too!)

i should fix system/core/debuggerd/crasher to add pac and bti crashes.
i was hoping to get round to that today, but ran out of time. i'll
follow up when that's done if you want some copy & paste assembler...

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqg%3DWUW5hH1%3DMr4ezOYyJ8dv%3DimViKYJtSkUEZnyV0CEccQ%40mail.gmail.com.

[John Dallman]

> correct, though you'll also need a version of Android that's enabling PAC/BTI. 

> (i think it may be on as far back as S, but definitely T.)

Ah, that may be a problem. The latest Android I have at present is Android 12, which is "S". I'll try it and see. 

> no, because it's a _return_ address you're trying to mess up. (BTI is
> the one for calls.) i'm not sure there's an easy way that isn't
> assembler?

Not that I can see, although I'll keep looking. 

> > To set off a BTI trap, I take a pointer to a function, increment it by one instruction, 

> > and call the function via that pointer. The program counter counts bytes, so to 

> > move it on by one instruction, I need to add 4.
> yeah, that should work. (though obviously that's likely to crash for
> _other_ reasons too!)

Absolutely! I'll put some assembler no-ops at the start of the target function, though they may well not help.

> i should fix system/core/debuggerd/crasher to add pac and bti crashes.
> i was hoping to get round to that today, but ran out of time. i'll
> follow up when that's done if you want some copy & paste assembler...

That would be very welcome!

Thanks very much. 

John

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAJgzZoreBQvu4L%3DmU0XPxDjArkZ2s0GU87o1YPmV3sUteEv71w%40mail.gmail.com.

[John Dallman]

In the detailed documentation, spelling out the conditions where it will be activated:

• ARMv9 hardware or later. 
• Kernel 5.0 or later.
• Android T (or is it S) or later. 

and explaining that unless all these conditions are met the traps won't get activated will be reassuring. Developers get very jumpy about setting off traps that they can't test for themselves on customer devices. Saying you were doing it for improved security is a great way to get fired. 

Thanks,

John

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAFVaGhu2QiGtpDgFVxrnTRfu8KOgayR1bQdbM-4mMXrJgYBbqQ%40mail.gmail.com.

[enh]

On Tue, Aug 8, 2023 at 2:24 AM John Dallman <jgdats...@gmail.com> wrote:
>
> > correct, though you'll also need a version of Android that's enabling PAC/BTI.
> > (i think it may be on as far back as S, but definitely T.)
>
> Ah, that may be a problem. The latest Android I have at present is Android 12, which is "S". I'll try it and see.

looks like BTI was in S:
https://android-review.googlesource.com/c/platform/bionic/+/1242754?tab=change-view-tab-header-android_wimcl

so PAC should be too.

> > no, because it's a _return_ address you're trying to mess up. (BTI is
> > the one for calls.) i'm not sure there's an easy way that isn't
> > assembler?
>
> Not that I can see, although I'll keep looking.

fwiw, i didn't even try and went straight to assembler:
https://android-review.googlesource.com/c/platform/system/core/+/2699774

note the note.gnu.property section that you'll need. i've sent a patch
upstream to toybox so readelf on the device will be able to decode the
PAC/BTI bits in a binary, but you can check on the host (`llvm-readelf
-aW my-binary` and look for PAC and BTI near the end).

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqgn4HBSnOvXVaEux%3DEMPEXG5exZiv13e88W_C1An8XQnaQ%40mail.gmail.com.

[John Dallman]

Splendid, thanks very much.

John

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAJgzZopvPHLbTJjX489CjaCUFJP9eU6FTz3qLX4DLp6zeSDxLw%40mail.gmail.com.

[John Dallman]

On Wed, Aug 9, 2023 at 12:18 AM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

fwiw, i didn't even try and went straight to assembler:
https://android-review.googlesource.com/c/platform/system/core/+/2699774

I presume this is under the Apache 2.0 license? Is there a way I can prove that? I'm working in closed-source proprietary software, and the lawyers are naturally twitchy about copyleft licenses. 

 

Thanks,

John

[enh]

On Thu, Sep 28, 2023 at 8:45 AM John Dallman <jgdats...@gmail.com> wrote:
>
> On Wed, Aug 9, 2023 at 12:18 AM 'enh' via android-ndk <andro...@googlegroups.com> wrote:
>>
>> fwiw, i didn't even try and went straight to assembler:
>> https://android-review.googlesource.com/c/platform/system/core/+/2699774
>
>
> I presume this is under the Apache 2.0 license? Is there a way I can prove that?

yeah. the canonical truth is always found by looking in the
corresponding Android.bp file. in this case:
```
package {
default_applicable_licenses: ["Android-Apache-2.0"],
}
```
(there's also a MODULE_LICENSE_* file in the directory; in this case
MODULE_LICENSE_APACHE2, but i expect those to disappear over time.
having the _build system_ understand licensing makes automated
checking much easier and more reliable.)

> I'm working in closed-source proprietary software, and the lawyers are naturally twitchy about copyleft licenses.

normally each file also includes a copyright header. it's just a
historical accident that these .S files don't, and i'll fix that. (i'd
love for us to move to SPDX headers one of these years, but i'm not
holding my breath!)

https://android-review.googlesource.com/c/platform/system/core/+/2767705

> Thanks,
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqg%3D24a5JgeM58rN1FDYidWmqUEYCk9bvRZKkXUSfntWcbw%40mail.gmail.com.

[John Dallman]

On Thu, Sep 28, 2023 at 6:42 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

 

normally each file also includes a copyright header. it's just a
historical accident that these .S files don't, and i'll fix that. (i'd
love for us to move to SPDX headers one of these years, but i'm not
holding my breath!)

https://android-review.googlesource.com/c/platform/system/core/+/2767705

[John Dallman]

On Thu, Sep 28, 2023 at 6:42 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

>> fwiw, i didn't even try and went straight to assembler:
>> https://android-review.googlesource.com/c/platform/system/core/+/2699774

https://android-review.googlesource.com/c/platform/system/core/+/2767705

 

I have this code compiling and running in a basic test program. 

On Android 9 on a Snapdragon 845, both the functions hang, as they are intended to.

On Android 12, on a Snapdragon 8 Gen 1, the PAC test produces "Illegal instruction" which is presumably what's intended. 

On the same hardware, the BTI test hangs. This is not what's meant to happen, and I'm trying to figure out what's missing. 

The device is a "Snapdragon® 8 Gen 1 Mobile Hardware Development Kit", whose web page is at https://developer.qualcomm.com/hardware/snapdragon-8-gen-1-hdk.

That's supposed to be ARMv9.0-A, which should have BTI if Wikipedia is correct. So the hardware is provisionally OK, and maybe the OS is bad? I don't have access to the sources for this device (or any others), but it claims to have:

$ uname -a
Linux localhost 5.10.81-android12-9-g6db10b8c584d #1 SMP PREEMPT Thu Mar 10 06:09:55 UTC 2022 aarch64

grep ' bti' /proc/cpuinfo gets me hits on all eight cores. 

If you can point me to the documentation for the note.gnu.property section the source file sets up, I can check if that's correct. If that was missing the BTI flag, that could presumably produce this problem? 

Thanks,

John

[enh]

it's there at the end of .S file:
https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/crasher/arm64/crashglue.S;l=101

look for that in your final ELF file. llvm-readelf lets you dump the
notes, and will decode them. if you're using AOSP ToT, i've added
PAC/BTI detection to toybox readelf on the device too
https://github.com/landley/toybox/commit/e25d0e03aca284f04875e4710ca43c7997263b57.

you can also use strace to check for PROT_BTI in the mmap() calls (but
you'll need the AOSP ToT one where i've added support for that bit
https://android-review.googlesource.com/c/platform/external/strace/+/2704457).

> Thanks,
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqg%3DtfSCC1oTWn%2BE0vw6maS6J4HwYoVSpXUw4JCg-0PhVvA%40mail.gmail.com.

[John Dallman]

On Wed, Oct 4, 2023 at 4:47 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

> If you can point me to the documentation for the note.gnu.property section the source file sets up, I can check if that's correct. If that was missing the BTI flag, that could presumably produce this problem?

it's there at the end of .S file:
https://cs.android.com/android/platform/superproject/main/+/main:system/core/debuggerd/crasher/arm64/crashglue.S;l=101

look for that in your final ELF file. llvm-readelf lets you dump the notes, and will decode them. 

I'm learning things here ... the .o file that clang produces from the .S file has a .note.gnu.property section:

> llvm-readelf --notes aarch64_crash_pac_bti.o
Displaying notes found in: .note.gnu.property
  Owner                Data size        Description
  GNU                  0x00000010       NT_GNU_PROPERTY_TYPE_0 (property note)
    Properties:    aarch64 feature: BTI, PAC

So that part was OK. 

After some experiments, I found that if the .c file with main() in it was not compiled -mbranch-protection=standard, the .note.gnu.property section does not appear in the final executable and BTI does not trap. If I compile with -mbranch-protection=standard, the .o file with main() now has that .note.gnu.property section, and so does my final executable. The thing that was surprising, and confused me for a while, is that PAC does trap without that note section being present. 

Thanks very much, I think I'm good now, 

John

[Elliott Hughes]

(sending again via the web interface since google groups rejected my reply to the list as spam /facepalm)

yeah, PAC is "self contained" in the sense that either a function is
compiled with PAC or it isn't --- neither callers nor callees need to
know, so it's fine to mix PAC and non-PAC code. ("fine" albeit
"probably unintended"!)

BTI, on the other hand, is more asymmetric --- if you've linked
together code built with and without BTI, you can't tell the cpu to
trap because you don't know that you don't have BTI-enabled code
calling non-BTI-enabled code (and thus landing on something other than
a BTI instruction).

 

Thanks very much, I think I'm good now, 

excellent!

 

John

[John Dallman]

yeah, PAC is "self contained" in the sense that either a function is
compiled with PAC or it isn't --- neither callers nor callees need to
know, so it's fine to mix PAC and non-PAC code. ("fine" albeit
"probably unintended"!)

BTI, on the other hand, is more asymmetric --- if you've linked
together code built with and without BTI, you can't tell the cpu to
trap because you don't know that you don't have BTI-enabled code
calling non-BTI-enabled code (and thus landing on something other than
a BTI instruction).

Ah, another of the things that make my job interesting. I work for a software component business, producing shared libraries (and equivalents on other platforms) that are licensed out to ISVs who use them in their apps. The libraries are used in-process, they aren't servers of any kind, just huge blocks of math-and-data-handling subroutines. 

If I compile with "-mbranch-protection=standard" on a device that supports PAC, my libraries will detect PAC errors even if the customer's app does not compile for it. But my libraries won't detect BTI errors unless the customer's app was compiled for it. Compiling only for PAC, or only for BTI, doesn't seem to improve the situation much. Let me see what the size and speed overheads are like!

Best,

John

[Dan Albert]

The libc++ that ships with the NDK is in the same situation. We build it with both so that anyone downstream that wants BTI coverage can have it.

--

You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqgkumXEFLTULF%3DbNd-Dx72PoXC9W7F%3Dhr-UxBgUMVzcSPg%40mail.gmail.com.

[enh]

well, you said _shared_ libraries above (rather than _static_), so
you're actually in luck... the PROT_BTI granularity is per shared
library (or the executable itself).

if you think about it, this makes sense:
1. calls _within_ the library are by definition fine because you're
required to have built all your objects with BTI.
2. calls _out_ of the library don't matter (because there's nothing
special about call sites).
3. calls _in_ to the library are -- providing you have your ELF note
as "proof" that you're BTI-clean -- fine because all your valid
destinations have a branch target instruction.

so if you care about this kind of thing, the only real reason i can
think of not to enable this (and the reason why we haven't flipped the
ndk default yet) is that you might reasonably question whether the
[small] code size increase is valuable to enough [of your] users to
warrant it. and that's a question for you and your customers to
discuss (probably involving checking your Play dashboard to see what
devices your users are actually using).

but you needn't worry about being the first mover --- Chrome cares
enough about security that they already shipped
(https://community.arm.com/arm-community-blogs/b/operating-systems-blog/posts/control-flow-integrity)
even though the majority of devices still aren't new enough to
benefit; the value of better protection for those devices that _can_
have it is worth it to them.

> Best,

>
> John
>
> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqgkumXEFLTULF%3DbNd-Dx72PoXC9W7F%3Dhr-UxBgUMVzcSPg%40mail.gmail.com.

[John Dallman]

On Tue, Oct 10, 2023 at 9:56 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

so if you care about this kind of thing, the only real reason i cann

think of not to enable this (and the reason why we haven't flipped the
ndk default yet) is that you might reasonably question whether the
[small] code size increase is valuable to enough [of your] users to
warrant it. and that's a question for you and your customers to
discuss (probably involving checking your Play dashboard to see what
devices your users are actually using).

Well, I have it implemented and have done enough testing to get an idea of the effects.

It increases the size of the text segment by about 0.7%, in a 60MB shared library. 

It slows down my performance benchmarks by 0.4% on a Snapdragon 845 running Android 9, where it's all no-ops, or 1.6% on a Snapdragon 8 Gen 1 with Android 12, where I can set off PAC and BTI traps at will. I presume that the slowdown on the 845 is just no-op time, and the larger slowdown when the traps are active is mostly due to the hash computations for PAC. Those computations can presumably be partly hidden by out-of-order execution, but not entirely. 

In my routine testing, done in every overnight build, which takes about 18 hours on that Snapdragon 8 Gen 1, I have no regressions attributable to this change, which is good. I have a final level of testing to run, which will take about a week on that device; I wanted to get the performance effects before starting that. 

I'll need to discuss this with my Head of Product, but I expect we'll go ahead with it.

I don't have a Play dashboard, because I don't produce any Android apps. The library is licensed out to third-party developers who produce apps that use it. We presume they advise their customers to use powerful devices; as far as we know, the Android apps are mostly used for mobile viewing of data produced by Windows apps using the Windows version of the library.  

Best,

John

[enh]

On Fri, Nov 3, 2023 at 6:36 AM John Dallman <jgdats...@gmail.com> wrote:
>
> On Tue, Oct 10, 2023 at 9:56 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:
>>
>> so if you care about this kind of thing, the only real reason i cann
>> think of not to enable this (and the reason why we haven't flipped the
>> ndk default yet) is that you might reasonably question whether the
>> [small] code size increase is valuable to enough [of your] users to
>> warrant it. and that's a question for you and your customers to
>> discuss (probably involving checking your Play dashboard to see what
>> devices your users are actually using).
>
>
> Well, I have it implemented and have done enough testing to get an idea of the effects.
>
> It increases the size of the text segment by about 0.7%, in a 60MB shared library.

(yeah, that sounds about right from what i remember from Chrome.)

> It slows down my performance benchmarks by 0.4% on a Snapdragon 845 running Android 9, where it's all no-ops, or 1.6% on a Snapdragon 8 Gen 1 with Android 12, where I can set off PAC and BTI traps at will. I presume that the slowdown on the 845 is just no-op time, and the larger slowdown when the traps are active is mostly due to the hash computations for PAC. Those computations can presumably be partly hidden by out-of-order execution, but not entirely.

that sounds about right, though i think that gets better in newer
cores ... quoting anandtech
(https://www.anandtech.com/show/18871/arm-unveils-armv92-mobile-architecture-cortex-x4-a720-and-a520-64bit-exclusive):
"""
One of the related security features of the latest Armv9.2
architecture is the introduction of a new QARMA3 Pointer
Authentication Code (PAC) algorithm. Arm claims the newer algorithm
reduces the CPU overhead of PAC to less than 1%, even on their little
cores, giving developers and handset vendors even less of a reason to
not enable the security feature. Most of these improvements revolve
around hardware integrity and security, with a combination of MTE and
native benefits through the 64-bit instruction and architecture, all
designed to make devices even more secure going into 2023 and beyond.
This fits with Arm's ethos to encourage a full switch to 64-bit over a
hybrid 64 and 32-bit marketplace.
"""
(the 'Q' in 'QARMA' stands for qualcomm, btw:
https://en.wikipedia.org/wiki/QARMA)

> In my routine testing, done in every overnight build, which takes about 18 hours on that Snapdragon 8 Gen 1, I have no regressions attributable to this change, which is good. I have a final level of testing to run, which will take about a week on that device; I wanted to get the performance effects before starting that.
>
> I'll need to discuss this with my Head of Product, but I expect we'll go ahead with it.

yeah, definitely a product decision, but i'm glad you got it to the
point where it's in their hands! you know where to find us if you hit
any technical problems (from your impressively extensive testing) :-)

> I don't have a Play dashboard, because I don't produce any Android apps. The library is licensed out to third-party developers who produce apps that use it. We presume they advise their customers to use powerful devices; as far as we know, the Android apps are mostly used for mobile viewing of data produced by Windows apps using the Windows version of the library.
>
> Best,
>
> John
>

> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqgmex9WKQpu3BpR5WXJb519hAjC%2BRSN9MrpYZZaER2feUg%40mail.gmail.com.

[John Dallman]

On Fri, Nov 3, 2023 at 9:45 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

> It slows down my performance benchmarks by 0.4% on a Snapdragon 845 running Android 9, where it's all no-ops, or 1.6% on  

> a Snapdragon 8 Gen 1 with Android 12, where I can set off PAC and BTI traps at will. I presume that the slowdown on the 845 

> is just no-op time, and the larger slowdown when the traps are active is mostly due to the hash computations for PAC. Those 

> computations can presumably be partly hidden by out-of-order execution, but not entirely.

> I'll need to discuss this with my Head of Product, but I expect we'll go ahead with it.

yeah, definitely a product decision, but I'm glad you got it to the
point where it's in their hands! You know where to find us if you hit

any technical problems (from your impressively extensive testing) :-)

My head of product is happy to go ahead with this. He'll trade 1.6% throughput for improved security on a mobile platform. 

The testing is so vast because this is a mathematical modeler, which is used in some very large applications and significant on Windows/Linux/macOS. We do full-scale testing on Android (and iPadOS), because that's easier than trying to decide what's a valid subset from 37 years of development tests and customer bugs. 

Best,

John

[enh]

On Tue, Nov 7, 2023 at 3:56 AM John Dallman <jgdats...@gmail.com> wrote:
>
>
>
> On Fri, Nov 3, 2023 at 9:45 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:
>>
>> > It slows down my performance benchmarks by 0.4% on a Snapdragon 845 running Android 9, where it's all no-ops, or 1.6% on
>>
>> > a Snapdragon 8 Gen 1 with Android 12, where I can set off PAC and BTI traps at will. I presume that the slowdown on the 845
>>
>> > is just no-op time, and the larger slowdown when the traps are active is mostly due to the hash computations for PAC. Those
>>
>> > computations can presumably be partly hidden by out-of-order execution, but not entirely.
>>
>> > I'll need to discuss this with my Head of Product, but I expect we'll go ahead with it.
>>
>> yeah, definitely a product decision, but I'm glad you got it to the
>> point where it's in their hands! You know where to find us if you hit
>> any technical problems (from your impressively extensive testing) :-)
>
>
> My head of product is happy to go ahead with this. He'll trade 1.6% throughput for improved security on a mobile platform.

nice!

looking to the future, it looks like we should just get QARMA3 "for
free" on suitable hardware:
https://patchwork.kernel.org/project/linux-arm-kernel/patch/20220131170654.6238...@arm.com/

> The testing is so vast because this is a mathematical modeler, which is used in some very large applications and significant on Windows/Linux/macOS. We do full-scale testing on Android (and iPadOS), because that's easier than trying to decide what's a valid subset from 37 years of development tests and customer bugs.

(yeah, an Android CTS run takes half an hour on 8 x86-64 servers
[citation needed:
https://source.android.com/docs/setup/create/cuttlefish-cts#run-cts]
and Android's not been around nearly that long!)

> Best,
>
> John
>
> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqgkYngWk_s19NxA3p%3D6G96-guw0pqv2WVkpC-ysHGKmzrA%40mail.gmail.com.

[John Dallman]

On Tue, Nov 7, 2023 at 9:44 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

looking to the future, it looks like we should just get QARMA3 "for free" on suitable hardware:
https://patchwork.kernel.org/project/linux-arm-kernel/patch/20220131170654.6238...@arm.com/

Good-oh. 

I have a new, but related question. I'm writing up a description of what PAC (and BTI) actually do, to explain what I've done to (a) our very skeptical build-mistress and (b) other software component people who support Android.

I thought that function pointers saved into memory would be PAC-signed and PAC-verified when they were used to make calls. A trivial example is not doing that, although it is using BTI, Should I expect the compiler to discard redundant PAC operations when it can see the whole flow of control in a single compilation? 

Thanks,

John

[enh]

it's actually worse than that --- function pointer protection just
keeps getting harder the more you think about it... there's all kinds
of weird stuff to worry about, from questionable (but common in some
contexts) code where the function type and function pointer type don't
exactly match, through stuff like laundering through void*, to whether
function pointer equality works between authenticated and
unauthenticated pointers, to give just a few examples. there's lots
more! (plus of course the critical "and how does anyone actually roll
any of this out?" question.)

people at various companies are working together on all this, but
there's nothing you can use in llvm yet... (for any OS, that i'm aware
of.)

> Thanks,

>
> John
>
> --
> You received this message because you are subscribed to the Google Groups "android-ndk" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/android-ndk/CAH1xqg%3Du8_NsrK_-hYsSX6Av9jcEPtX3bOQORzJqPxr2szgupg%40mail.gmail.com.

[John Dallman]

On Fri, Nov 17, 2023 at 8:56 PM 'enh' via android-ndk <andro...@googlegroups.com> wrote:

it's actually worse than that --- function pointer protection just
keeps getting harder the more you think about it... there's all kinds
of weird stuff to worry about, from questionable (but common in some
contexts) code where the function type and function pointer type don't
exactly match, through stuff like laundering through void*, to whether
function pointer equality works between authenticated and
unauthenticated pointers, to give just a few examples. there's lots
more! (plus of course the critical "and how does anyone actually roll
any of this out?" question.)

So I should explain this as "currently only applies to pointers saved and loaded within a function, still under development, but useful in its current form"?

Another question: The PAC and BTI crasher functions you supplied don't have bti instructions at their starts. Does BTI only apply to branches, and not calls?  

Thanks very much,

John

[John Dallman]

> Another question: The PAC and BTI crasher functions you supplied don't have bti instructions 

> at their starts. Does BTI only apply to branches, and not calls?  

Figured it out after some experiments and looking carefully at at ARM's manual. An important details about BTI is that it only applies to register-indirect branches, not all branches, which would presumably have larger overheads.  

Thanks again, 

John