Per-addon memory accounting

[Nicholas Nethercote]

Hi,

In a previous thread
(http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/0a5c90a233bc4871?pli=1)
I asked about per-tab memory accounting. The topic of per-addon
memory accounting came up briefly, and I'd like to dig into it more
now.

The most substantive comment on this topic was from bsmedberg:

"We are actively working on moving at least jetpack-style addons into a
separate process. This was part of the original goal for the Jetpack SDK
1.0 but was not feasible this time. But even there, the addon will also
be able to run code in the context of content (content scripts), so
we'll have to do some combination of measurements. It probably isn't
feasible to do this work for old-style addons because they are basically
chrome and much of their code runs in the main browser window global."

Per-compartment reporters are shaping up to be a fantastic tool (eg.
see https://bugzilla.mozilla.org/show_bug.cgi?id=668871) so it would
be great to get some kind of per-addon memory reporting in the tree
ASAP, if it is at all possible. If that would only apply to
jetpack-style extensions, that's fine, though if anything could be
done for old-style extensions that would also be great. I'd love to
hear any suggestions anyone has.

The only lead I have on this stuff is that some jetpack extension
create their own JS compartment
(http://mxr.mozilla.org/mozilla-central/source/js/jetpack/JetpackChild.cpp#454).
This makes things easy, because the existing per-compartment
reporters will pick that up, though I don't know what name the
compartment will have. But from what I can tell not many extensions
create compartments in this way.

Thanks.

Nick

[johnjbarton]

As far as I know only binary code can even know that compartments exist.
They have no JS existence.
jjb

>
> Thanks.
>
> Nick

[Josh Matthews]

On 11-07-06 9:10 PM, Nicholas Nethercote wrote:
> The only lead I have on this stuff is that some jetpack extension
> create their own JS compartment
> (http://mxr.mozilla.org/mozilla-central/source/js/jetpack/JetpackChild.cpp#454).
> This makes things easy, because the existing per-compartment
> reporters will pick that up, though I don't know what name the
> compartment will have. But from what I can tell not many extensions
> create compartments in this way.

Note that the code linked is part of the implementation of
out-of-process Jetpack addons, and would need extra work to make memory
usage data show up in about:memory, just like content processes did.

Cheers,
Josh

[Alexandre Poirot]

As I said in this comment:
https://bugzilla.mozilla.org/show_bug.cgi?id=672443#c1
Jetpack is currently over-using sandboxes, so if we have some platform
capabilities to track memory used by scripts runned in one particular
sandbox, we would be able to measure memory usage of an addon.
Then, we have e10s move. It is currently a P1 for jetpack team:
https://wiki.mozilla.org/Features/Jetpack/Out-of-Process_Addons
We are still trying to figuring out what is the best approach from the
platform side of view.
Eddy Bruel is currently working on measuring CPU and memory cost of
content processes and former jetpack processes.

So in the upcoming e10s world, it would be really easy to measure
these process CPU/memory usage,
but Benjamin mentioned an important point: jetpack addons can execute
some "content scripts" (with page-mod API)
into content processes. So we get back to sandboxes. We are going to
execute these scripts in a sandbox that use same principal than the
page it is attached to.

[Boris Zbarsky]

On 7/19/11 10:25 AM, Alexandre Poirot wrote:
>So we get back to sandboxes. We are going to
> execute these scripts in a sandbox that use same principal than the
> page it is attached to.

I wonder whether it's worth it to change the sandbox constructor to have
a way of attaching information about who's creating the sandbox (as
opposed to the principal of the sandbox, which is all we have now) to
the sandbox and then expose that info on the compartment somehow. That
seems like the ideal thing from the about:memory pov.

-Boris

[Nicholas Nethercote]

Yes. I just landed on mozilla-inbound the patch from
https://bugzilla.mozilla.org/show_bug.cgi?id=672439 which
distinguishes all these same-named system compartments by their
address, so you can see how many system compartments there are, but
you don't know what each one represents.

Extra origin info would be great, I'm happy to write a patch if
someone can tell me what extra information would be suitable and where
I can get it from.

Nick

[Dave Townsend]

I'm assuming that anything loaded with Components.utils.import ends up in the system compartment? Simply typing those to the URL they were imported from would be an easy first step. Presumably something similar would work for JS components.

As for plain sandboxes, I suspect as Boris says we may just have to go and manually set those with some new argument to the sandbox constructor or something. Would be easy to include the add-on ID that way for restartless add-ons.

[Dave Townsend]

[Kyle Huey]

On Tue, Jul 19, 2011 at 5:22 PM, Nicholas Nethercote <n.neth...@gmail.com
> wrote:

> Extra origin info would be great, I'm happy to write a patch if
> someone can tell me what extra information would be suitable and where
> I can get it from.
>

We can store it on the xpc::CompartmentPrivate. What information would be
useful is more tricky ...

- Kyle

[Nicholas Nethercote]

On Tue, Jul 19, 2011 at 5:31 PM, Dave Townsend
<dtow...@oxymoronical.com> wrote:
> I'm assuming that anything loaded with Components.utils.import ends up in the system compartment? Simply typing those to the URL they were imported from would be an easy first step. Presumably something similar would work for JS components.

Does each thing loaded with Cu.import end up in its own compartment?

> As for plain sandboxes, I suspect as Boris says we may just have to go and manually set those with some new argument to the sandbox constructor or something. Would be easy to include the add-on ID that way for restartless add-ons.

Bug 673331 just landed on mozilla-inbound thanks to the efforts of a
new contributor, Sander van Veen. As a result,
Component.utils.Sandbox's optional second parameter can now contain a
property called "sandboxName", e.g.:

var sandbox = new Cu.Sandbox(this._window, {sandboxName: 'mySandbox'});

Each sandbox gets its own compartment (I think?), and sandboxes
created with a |sandboxName| specified will be identified as such in
about;memory. For example:

│ ├─────324,679 B (00.41%) -- compartment([System Principal], file:///home/njn/
.mozilla/firefox/li1f1wj6.bugzilla-tweaks/extensions/jid0-qBnIpLfDFa4LpdrjhAC6vB
qN20Q@jetpack/bootstrap.js, 0x7f2400294000)
│ │ ├──163,840 B (00.21%) -- gc-heap
│ │ │ ├───62,904 B (00.08%) -- objects
│ │ │ ├───44,608 B (00.06%) -- shapes
│ │ │ ├───35,144 B (00.04%) -- arena-unused
│ │ │ ├───18,112 B (00.02%) -- strings
│ │ │ ├────1,792 B (00.00%) -- arena-padding
│ │ │ └────1,280 B (00.00%) -- arena-headers
│ │ ├───65,536 B (00.08%) -- mjit-code
│ │ ├───36,734 B (00.05%) -- string-chars
│ │ ├───21,248 B (00.03%) -- object-slots
│ │ ├───21,141 B (00.03%) -- scripts
│ │ ├───15,704 B (00.02%) -- property-tables
│ │ └──────476 B (00.00%) -- mjit-data

The compartment name is not pretty, but you can tell that
bugzilla-tweaks created the sandbox.

Now, I believe all the Cu.Sandbox calls in core code have been
annotated, but lots of add-ons create their own sandboxes. So they'll
need to be changed to get this memory accounting, but at least they'll
get the option now; AFAIK this is the first feature that allows any
kind of per-add-on memory accounting. I think this is a big deal
because I've had requests from multiple add-on authors asking for any
help they can get to help improve memory usage.

Ehsan, if you want to modify Bugzilla Tweaks to use this new feature,
that would be a great first example :) Sander has written the patch
for you already:
https://bug673331.bugzilla.mozilla.org/attachment.cgi?id=550570.

Nick

[Robert Kaiser]

Dave Townsend schrieb:

> I'm assuming that anything loaded with Components.utils.import ends up in the system compartment? Simply typing those to the URL they were imported from would be an easy first step.

I actually don't think that would turn out well, as IIRC that code is
optimized to never load an imported module twice (which is one of the
reasons it has been introduced in the first place, AFAIK).

Robert Kaiser

--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community should think about. And most of the
time, I even appreciate irony and fun! :)

[Ehsan Akhgari]

On 11-08-21 7:17 PM, Nicholas Nethercote wrote:
> On Tue, Jul 19, 2011 at 5:31 PM, Dave Townsend
> <dtow...@oxymoronical.com> wrote:

>> I'm assuming that anything loaded with Components.utils.import ends up in the system compartment? Simply typing those to the URL they were imported from would be an easy first step. Presumably something similar would work for JS components.
>

> Does each thing loaded with Cu.import end up in its own compartment?
>

>> As for plain sandboxes, I suspect as Boris says we may just have to go and manually set those with some new argument to the sandbox constructor or something. Would be easy to include the add-on ID that way for restartless add-ons.
>

Well, this patch should be applied against the Add-on SDK, I think.
Once that lands in the Add-on SDK, dkl or glob (who are the new owners
of Bugzilla Tweaks) can release a new version based on an updated Add-on
SDK.

I filed bug 680958 for that.

Cheers,
Ehsan

[David Lawrence]

I see that this has landed now. When will a new version of the SDK be published publicly
or should I pulling my own from github and then rebuilding Bugzilla Tweaks based on that?
Right now I am still using SDK 1.0.

dkl

On 8/22/11 1:41 PM, Ehsan Akhgari wrote:
> On 11-08-21 7:17 PM, Nicholas Nethercote wrote:
>> On Tue, Jul 19, 2011 at 5:31 PM, Dave Townsend
>> <dtow...@oxymoronical.com> wrote:

>>> I'm assuming that anything loaded with Components.utils.import ends up in the system compartment? Simply typing those to the URL
>>> they were imported from would be an easy first step. Presumably something similar would work for JS components.
>>

>> Does each thing loaded with Cu.import end up in its own compartment?
>>

>>> As for plain sandboxes, I suspect as Boris says we may just have to go and manually set those with some new argument to the
>>> sandbox constructor or something. Would be easy to include the add-on ID that way for restartless add-ons.
>>

--
David Lawrence
d...@mozilla.com

[Myk Melez]

On 2011-09-06 9:20 AM, David Lawrence wrote:
> I see that this has landed now. When will a new version of the SDK be published publicly
> or should I pulling my own from github and then rebuilding Bugzilla Tweaks based on that?
> Right now I am still using SDK 1.0.

SDK 1.1 ships next Tuesday, September 13, but it will not include this
feature. SDK 1.2, which will include this feature, ships Tuesday,
October 25. (SDK development is on a train schedule
<https://wiki.mozilla.org/Jetpack/Development_Process> similar to and
synced with Firefox's, except with one fewer branch, and the releases
happen two weeks before Firefox releases).

You can pull from Github. Alternately, you can wait until we spin SDK
1.2a1, either next Tuesday or the Tuesday after (September 20). Either
way, note that AMO will not automatically repack addons built with
non-release versions of the SDK when new release versions are available.
So if you rebuild with a GitHub pull or a test build, you'll need to
update your addon manually.

-myk