PHC: "Where do those extra stacks come from?"

[Paul Bone]

TL;DR: we're rolling out PHC, currently at 1% soon at 10%.  If you already know about PHC that's the new information.

I've been filing a few extra crash reports with memory errors such as buffer overruns and use after frees.  I found a comment on one of them this morning "Where do these extra stacks come from?".  That's a great question and tells me I haven't communicated this widely enough yet.  I'd like to do that properly later, but for now I'd like this e-mail to serve as an informal introduction.

The Probabilistic Heap Checker (PHC) is a component in Firefox that will, probabilistically, redirect a `malloc()` request into a special area where it can perform extra checking for memory errors.  It can detect buffer overruns and use-after-free errors.  But the really cool thing is that it records the stack at the time of allocation and free.  So that for a use-after-free, the crash report constrains not only the stack where the error occurred, but stacks that describe the object's lifetime.

It also records the address and size of the memory allocation.

This information is behind "Protected Data Access", engineers with the appropriate crash-stats permission have access, it's also not symbolicised so far.  So most people are going to notice it when bugs are filed against their components.  They'll see the extra stacks in the bug report and might wonder where it came from if they don't have protected data access.

Right now this runs for everybody in Firefox Nightly (and has done for years), now we're rolling it out in Firefox Release, it's enabled for roughly 1% of our population in Firefox 120 and 121 and we already have 13 crash reports annotated with PHC stacks from december (not all of them are genuine).  We're planning to roll out to 10% of release in January which means we could expect ~130 crash reports (some percentage of which will be genuine bugs).

We're looking forward to having this new capability to find and diagnose memory errors.  I'll be filing bugs for these as appropriate and attaching these extra stacks.  I'll also continue to be making improvements to PHC in the new year.  Until then, if you're taking a break this time of year have a safe and happy holiday season.

Cheers.

[Christian Holler]

Thank you Paul for pushing this forward! PHC is a vital contribution to overall Firefox security and stability. If you as a developer see a bug with additional PHC information being filed in your component, I kindly ask you to prioritize this over regular crash investigation to fully utilize this new tool. In 2024, Paul will also be working with Suhaib and willkg to get these crashes to you automatically just like regular crash-stats bugs.

If you are interested to learn more about this tool, we do have a paper on it published at ICSE together with our colleagues from other companies that utilize the same technology: https://arxiv.org/abs/2311.09394

Happy holidays!

- Chris

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CANdLaqBq3JsDrbRSOPCMAFQF41N%3D%3DRjCFvojF4%3DpV9K2Tq2zAQ%40mail.gmail.com.

[Chris Peterson]

On which platforms is PHC enabled? Windows, macOS, Linux, and/or Android?

[Daniel Veditz]

Not a definitive answer, but I've seen crash reports on Windows, Mac,
and Linux, but none on Android. The pref (memory.phc.enabled) is true
on Fenix nightly but I don't know if Android is included in the
roll-out experiment. PHC is not very aggressive to keep the
performance and memory impacts down, so we won't see many of these
instrumented crashes until it's rolled out to a very large audience.

Another factor: PHC is only enabled on machines with 8Gb of memory
which might exclude a lot of android devices

[Paul Bone]

Yes, Android is excluded (for now).

 * Android + 32bit systems of any kind:

   PHC isn't even compiled-in, so the pref doesn't matter, but it is confusing to see the pref set to true (I can fix that).

 * 64bit desktop:

   PHC is compiled in, and now defers to the pref memory.phc.enabled.  Which is set by the experiment.

 * If PHC is compiled in and the pref is true:

   PHC checks if there is at least 8GB of RAM (another pref) and only enables if this is true.

Another question/clarification I saw elsewhere was about whether this will always be an experiment.  No. it's my intention that it'll eventually default-on in most cases.  It may have memory limits or even adjust how much memory it uses depending on the physical ram available.

[Gabriele Svelto]

On 08/01/24 00:49, Paul Bone wrote:
> Yes, Android is excluded (for now).

It might be worth noting that even when we enable PHC its effectiveness
might be lower than on other platforms. As Chris pointed out to me the
sessions on Android are much, much shorter than on desktop.

Gabriele