startup performance of Addon SDK-based extensions

[Nickolay Ponomarev]

Hi,

Are there plans to track and improve the startup time of addon SDK based extensions in the 1.0 timeframe? The AMO team seems to talk about this regularly (see for example the AMO roadmap for 2011 or this post from 2010).

There are two reasons I ask:
1) It might turn out that to have acceptable performance we'll need to undo some of the changes adding modularity and abstractions to the SDK code.
2) We might need changes from the platform side (see Performance of evalInSandbox, Does evalInSandbox take advantage of compile caching?), which require prior planning.


I did some quick measurements on Mac OS X (10.5) with the latest Minefield and the latest Add-on SDK using getStartupInfo(). According to my tests,

• an empty extension (empty main.js) adds ~20 ms to the 700 ms baseline warm startup time (3%), which is tolerable, but
  
• the default extension created with 'cfx init' (adds a simple widget and loads the tabs module) adds more than 200 ms (28%) to the warm startup!
  

I ran some poor man profiling to check a hypothesis that most of this cost is in the module system (require()). It is very rough, but it showed how most (75%) of the time was spent:

• 25% (50ms) running es5.js in each of the 38 distinct modules loaded
• 15% (30ms) is other initialization operations with the sandbox (creating sandboxes, setting properties)
  
• 10% (20ms) is spent in fs.getFile(), presumably loading the modules from disk
  
• 25% (50ms) spent evaluating the module code itself, not counting the require() costs listed above.
  

Here are the 38 modules loaded. Here are the details of my instrumentation and the results: legend, data:

Counter	Hit count	Time spent, ms	Attributed to the securable-module infrastructure / to the user-level code?
start loading	38	37	user
fs.getFile()	38	19	s-m
up to createSandbox	38	0	s-m
createSandbox	38	17	s-m
setupSandbox, part 1	38	1	s-m
setupSandbox, part 2 - modifyModuleSandbox	38	54	s-m
setupSandbox, part 3	38	15	s-m
before evaluate	38	0	s-m
after evaluate	38	52	user
finished	38	0	s-m

Nickolay

[Dietrich Ayala]

Thanks for Nickolay, this is *fantastic* data.

Do you know if there already bugs for the platform problems?

Things that might help this:

* not packaging unused modules (which Brian is working on)
* minifying JS
* making sure we're properly optimized for omnijar
* ensuring module environment doesn't pull in unnecessary api-kit modules
* maybe doing something under the covers to allow us to use Cu.import?

Any other ideas?

I've cc'd Justin from AMO. Justin, is there a way for us to track Add-on SDK add-ons in AMO's performance monitoring right now? Is there something we can do to make that possible?

--
You received this message because you are subscribed to the Google Groups "mozilla-labs-jetpack" group.
To post to this group, send email to mozilla-la...@googlegroups.com.
To unsubscribe from this group, send email to mozilla-labs-jet...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mozilla-labs-jetpack?hl=en.

[KWierso]

Hopefully, Jetpack extensions can stop adding <unpack> true</unpack>
to install.rdf sometime in the future.

On Mar 13, 8:26 am, Dietrich Ayala <auton...@gmail.com> wrote:
> Thanks for Nickolay, this is *fantastic* data.
>
> Do you know if there already bugs for the platform problems?
>
> Things that might help this:
>
> * not packaging unused modules (which Brian is working on)
> * minifying JS
> * making sure we're properly optimized for omnijar
> * ensuring module environment doesn't pull in unnecessary api-kit modules
> * maybe doing something under the covers to allow us to use Cu.import?
>
> Any other ideas?
>
> I've cc'd Justin from AMO. Justin, is there a way for us to track Add-on SDK
> add-ons in AMO's performance monitoring right now? Is there something we can
> do to make that possible?
>

> On Sun, Mar 13, 2011 at 4:19 PM, Nickolay Ponomarev <asquee...@gmail.com>wrote:
>
>
>
>
>
>
>
> > Hi,
>
> > Are there plans to track and improve the startup time of addon SDK based
> > extensions in the 1.0 timeframe? The AMO team seems to talk about this

> > regularly (see for example the AMO roadmap for 2011<https://wiki.mozilla.org/AMO:Roadmap/2011#PERFORMANCE:_Add-ons_don.27...>or this
> > post from 2010<http://blog.mozilla.com/addons/2010/06/14/improve-extension-startup-p...>
> > ).

>
> > There are two reasons I ask:
> > 1) It might turn out that to have acceptable performance we'll need to undo
> > some of the changes adding modularity and abstractions to the SDK code.
> > 2) We might need changes from the platform side (see Performance of

> > evalInSandbox<http://groups.google.com/group/mozilla.dev.platform/browse_thread/thr...>,
> > Does evalInSandbox take advantage of compile caching?<http://groups.google.com/group/mozilla.dev.platform/browse_thread/thr...>),

> > which require prior planning.
>
> > I did some quick measurements on Mac OS X (10.5) with the latest Minefield

> > and the latest Add-on SDK using getStartupInfo()<http://blog.mozilla.com/tglek/2011/01/14/builtin-startup-measurement/>.
> > According to my tests,
>
> >    - an empty extension (empty main.js) adds ~20 ms to the 700 ms baseline

> >    warm startup time (3%), which is tolerable, but

> >    - the default extension created with 'cfx init' (adds a simple widget
> >    and loads the tabs module) adds *more than 200 ms (28%)* to the warm

> >    startup!
>
> > I ran some poor man profiling to check a hypothesis that most of this cost
> > is in the module system (require()). It is very rough, but it showed how
> > most (75%) of the time was spent:
>

> >    - 25% (50ms) running es5.js in each of the 38 distinct modules loaded
> >    - 15% (30ms) is other initialization operations with the sandbox
> >    (creating sandboxes, setting properties)
> >    - 10% (20ms) is spent in fs.getFile(), presumably loading the modules
> >    from disk
> >    - 25% (50ms) spent evaluating the module code itself, not counting the
> >    require() costs listed above.
>
> > Here are the 38 modules loaded<https://github.com/nickolay/addon-sdk/blob/TEST-securable-module-prof...>.
> > Here are the details of my instrumentation and the results: legend<https://github.com/nickolay/addon-sdk/blob/TEST-securable-module-prof...>,
> > data:
>
> > Counter Hit count Time spent, msAttributed to the

[Nickolay Ponomarev]

Hey Dietrich, thanks for your reply!

On Sun, Mar 13, 2011 at 4:26 PM, Dietrich Ayala <auto...@gmail.com> wrote:

Do you know if there already bugs for the platform problems?

Platform as in the Mozilla platform? Not that I know of... There are no obvious problems, but there could be some support for doing modules in the SDK more efficiently. Specifying the requirements is the key here, but I'm not up to doing that, unfortunately.

Modules do seem very similar to Cu.import if you remove the special handling of EXPORTED_SYMBOLS, add the ability to unload them at run-time (bug 481603, dtownsend has a WIP patch), and add the ability to create several instances of a single module (since the current SDK creates an independent set of modules sometimes, e.g. in tests or in the content-related modules).

Things that might help this:

* not packaging unused modules (which Brian is working on)
* minifying JS
* making sure we're properly optimized for omnijar

(These wouldn't help with the *warm* startup much, as I'm sure you know, but yeah, it's good for the cold startup. I hope we can do without minifying...)

* ensuring module environment doesn't pull in unnecessary api-kit modules
* maybe doing something under the covers to allow us to use Cu.import?

Any other ideas?

* Continuous builds, generating performance numbers to stop bad check-ins from landing (Am I dreaming?)
-- failing that, adding a simple perf test to cfx and keeping people aware of the importance of performance.
* Fixing or removing the "es5" hit.
* Possibly coalesce the sandbox initialization into a single call, since they appear to have a non-trivial cost.
* Changing the SDK modules to load things lazily (possibly making the require() call return a promise?), so that we don't load as much at startup.

On Sun, Mar 13, 2011 at 7:08 PM, KWierso <kwi...@gmail.com> wrote:

Hopefully, Jetpack extensions can stop adding <unpack> true</unpack>
to install.rdf sometime in the future.

Irakli is working on this <https://bugzilla.mozilla.org/show_bug.cgi?id=638742>, yay!

Nickolay

[Hernan Rodriguez Colmeiro]

On Sun, Mar 13, 2011 at 14:55, Nickolay Ponomarev <asqu...@gmail.com> wrote:
> * Changing the SDK modules to load things lazily (possibly making the
> require() call return a promise?), so that we don't load as much at startup.
>

I think Brian Warner is doing something related: switch to build-time
module resolution? (from run-time lookup)
[https://bugzilla.mozilla.org/show_bug.cgi?id=627607]

Hernán

[Dietrich Ayala]

On Mon, Mar 14, 2011 at 12:55 AM, Nickolay Ponomarev
<asqu...@gmail.com> wrote:
> * Continuous builds, generating performance numbers to stop bad check-ins
> from landing (Am I dreaming?)
> -- failing that, adding a simple perf test to cfx and keeping people aware
> of the importance of performance.

We need the continuous integration perf testing with mozilla-central
and some canonical SDK-based add-on.

I added Firefox Desktop support to Mark Finkle's test-harness add-on*,
so the next version will allow devs/users to run startup and pageload
tests locally.

* http://starkravingfinkle.org/blog/2011/02/zippity-using-the-crowd-to-collect-performance-data/

[Irakli Gozalishvili]

Nickolay first of all thanks for bringing this up, and for an interesting numbers!

Please see my other comments inline

On Sun, Mar 13, 2011 at 10:19, Nickolay Ponomarev <asqu...@gmail.com> wrote:

Hi,

Are there plans to track and improve the startup time of addon SDK based extensions in the 1.0 timeframe? The AMO team seems to talk about this regularly (see for example the AMO roadmap for 2011 or this post from 2010).

There are two reasons I ask:
1) It might turn out that to have acceptable performance we'll need to undo some of the changes adding modularity and abstractions to the SDK code.
2) We might need changes from the platform side (see Performance of evalInSandbox, Does evalInSandbox take advantage of compile caching?), which require prior planning.

I guess we could use function wrappers instead of sandboxes. I suspect it to perform better as no proxies will be used, but don't know if diff will be significant enough to make a difference.
 

I did some quick measurements on Mac OS X (10.5) with the latest Minefield and the latest Add-on SDK using getStartupInfo(). According to my tests,

• an empty extension (empty main.js) adds ~20 ms to the 700 ms baseline warm startup time (3%), which is tolerable, but
  
• the default extension created with 'cfx init' (adds a simple widget and loads the tabs module) adds more than 200 ms (28%) to the warm startup!
  

I ran some poor man profiling to check a hypothesis that most of this cost is in the module system (require()). It is very rough, but it showed how most (75%) of the time was spent:

• 25% (50ms) running es5.js in each of the 38 distinct modules loaded

Wow this is entirely unexpected as es5 only shims built-ins if they don't have `Object.create` and friends. Also it's not expected to do anything for FF >= 4.*

I guess we can just get rid of it since we no longer support 3.* or just eval it in sandboxes if it's on <= 3.*