Removing Jemalloc 4

[Mike Hommey]

Hi,

We've tried to get off mozjemalloc for, apparently, close to 5 years
(date of the filing of bug 762449). We've had memory usage regressions
(like bug 1219914), and we've had perf regressions as per talos numbers
(things like bug 1138999), and those have never gone away (with
variations with each update of jemalloc >= 3).

My best bet at this point is that it's "just" a consequence of the
difference in memory layout due to the differences in how the allocators
work. That doesn't make it more okay.

Many updates to recent versions of jemalloc have been painful (usually
breaking everything except linux), and the current tip of the jemalloc
dev branch is not making things any better (see bug 1357301).

Furthermore, bug 1361258 has started to modify mozjemalloc with new
features for stylo, further deepening the API surface between both
allocators. While what was added in bug 1361258 is possible to implement
for jemalloc 4, the burden of continuing to maintain both allocators is
not really appealing considering the perspective of ever switching.

As much as it pains me, it's time to admit that switching to jemalloc >=
3 is not going to happen, and pull the plug once and for all. This
decision has taken way too long to be made, and I apologize for that.

This will happen with the landing of bug 1363992 in a few hours.

As for the things we were hoping jemalloc >= 3 would allow us to do
(essentially, heap partitioning), we'll be working on getting that on
mozjemalloc shortly (bug 1361258 and followups will actually get us
close on the necessary infrastructure), as well as cleaning up its code
(I have a local patch queue that removes 15% of jemalloc.c), and
probably converting it to C++ (at least for some RAII benefits, and
converting some of the gory macros to templates)

Mike.

[Nicholas Nethercote]

Just to add some context: glandium is deeply familiar with jemalloc4's
internals, having submitted numerous patches and fixes to it. And he has
spent *significant* time and effort on multiple occasions, on multiple
versions of jemalloc4, trying to avoid the performance regressions, without
success. If he can't get it into a state suitable for our use, I have
trouble imagining who else could.

In other words, this is not a hasty or premature decision, but one made
reluctantly based on experience.

Nick

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Eric Rahm]

Having been involved with jemalloc 3/4/5 work as well, I agree with Mike's
conclusions.

-e

[Tom Ritter]

My interest in jemalloc3/4 has always been with taking advantage of
it's partitioning capabilities to segment things like javascript
arrays for increased security against heap grooming and UAF
exploitation.

Is there a path forward with this in mozjemalloc? Plans, or would-take
changes, or just thoughts on what to do here?

-tom

[Mike Hommey]

On Tue, May 16, 2017 at 01:33:13AM -0500, Tom Ritter wrote:
> My interest in jemalloc3/4 has always been with taking advantage of
> it's partitioning capabilities to segment things like javascript
> arrays for increased security against heap grooming and UAF
> exploitation.
>
> Is there a path forward with this in mozjemalloc? Plans, or would-take
> changes, or just thoughts on what to do here?

See last paragraph of the email you quoted ;)

[Tom Ritter]

On Tue, May 16, 2017 at 1:48 AM, Mike Hommey <m...@glandium.org> wrote:
> On Tue, May 16, 2017 at 01:33:13AM -0500, Tom Ritter wrote:
>> My interest in jemalloc3/4 has always been with taking advantage of
>> it's partitioning capabilities to segment things like javascript
>> arrays for increased security against heap grooming and UAF
>> exploitation.
>>
>> Is there a path forward with this in mozjemalloc? Plans, or would-take
>> changes, or just thoughts on what to do here?
>
> See last paragraph of the email you quoted ;)

I have only the time I emailed to my defense. =(

-tom

[Steve Fink]

But 4 is bigger than 3, and much bigger than 0.9, so it must be way
better, right? ;-)

For the record, I have no reason to object to this plan, and
conceptually it seems like allocators always have to tune for something.
If mozjemalloc is at this time tuned to our workload, it seems very
plausible that jemalloc has over time optimized for something somewhat
different.

But I'm curious if you know anything about the intended future
directions of jemalloc, if there are any, and whether they align with
anything we need. Also, how much of the difficulty in maintaining
jemalloc perf across upgrades is specific to ARM?

Again, I am not challenging the decision. I agree that you're the best
person to make this call. I just like to know about the sweet spots of
different memory allocators.

[Nicholas Nethercote]

On Wed, May 17, 2017 at 4:28 AM, Steve Fink <sf...@mozilla.com> wrote:

>
> But I'm curious if you know anything about the intended future directions
> of jemalloc, if there are any, and whether they align with anything we need.

As far as I know the short answer is "whatever Facebook needs", because
jemalloc's author works for Facebook and Facebook is the primary consumer.

Nick

[Mike Hommey]

On Tue, May 16, 2017 at 11:28:02AM -0700, Steve Fink wrote:
> But I'm curious if you know anything about the intended future directions of
> jemalloc, if there are any, and whether they align with anything we need.

As Nick mentions, Jason Evans, the main developer on jemalloc (guess
where "je" comes from?) has been working at Facebook for a few years,
and Facebook needs is what mainly drives jemalloc development nowadays.
There are a few non-Facebook contributors, though.

Theoretically, many of those changes would benefit us. But in practice,
we've had many surprises.

If the current jemalloc was on par with mozjemalloc wrt performance and
memory usage, we may have more easily tracked and been able to influence
its development, but we've been playing catch-up and failed to make it
suit our needs, so that doesn't really help.

> Also, how much of the difficulty in maintaining jemalloc perf across
> upgrades is specific to ARM?

Not much, in fact. We've had regressions on all platforms, not
necessarily at the same time.


Mike

[jas...@canonware.com]

This is a disappointing outcome. You put a huge amount of work into porting and testing jemalloc, and more recently I've done a lot of work to improve reliability especially on Windows, primarily with Firefox in mind. Over the past few months we've fixed some long-standing performance issues that were peculiar to Windows. Even more recently, Qi Wang, David Goldblatt and I have done a lot of optimization in anticipation of the jemalloc 5.0.0 release, which is shaping up to be really solid. It isn't clear to me whether a final detailed Firefox+jemalloc[45] performance analysis happened after those fixes and optimizations; the bits of analysis I see evidence of in bugzilla suggest that some simple issues such as the behavior of decay-based purging may not have been taken into consideration.

My disappointment aside, thank you for all your contributions to jemalloc. You influenced its development in ways that continue to have a positive impact, especially as related to testing. Although all the jemalloc team members are currently employed by Facebook, we are developing jemalloc completely in the open in large part so that it is feasible to collaborate with other organizations and robustly handle diverse workloads. I wish Firefox would continue to be part of our target.