So, what's the point of Cu.import, these days?

[David Teller]

Once again, there have been discussions on the feasibility of adding
static analysis to our JS code, possibly as part of MozReview. As usual,
one of the main problems is that we are not using standard JS, so we
pretty much cannot use standard tools.

One of the main differences between mozilla-central JS and standard JS
is our module system. We use `Components.utils.import`, while the rest
of the world is using `require`-style modules. If we could get rid of
`Cu.import`, we would be a very large step closer towards standard JS.

Which begs the question: what's the point of `Cu.import` these days?

Yes, I'm aware that it isolates code in separate compartments, and that
there is a benefit to isolating add-on code from platform code. However,
it is pretty unclear to me that there is any benefit in separating
compartments inside mozilla-central, rather than, say, relying upon
static analysis and/or reviews to ensure that nobody modifies
`Object.prototype` in funky ways.

If we decide to abandon the guarantees provided by compartments to
isolate mozilla-central modules from each other, it's not hard to imagine:
- semi-automated rewrites that could convert mozilla-central code to
RequireJS-style modules, all sharing a single compartment (per process);
- a backwards compatible, compartment-isolating implementation of
`Cu.import` for the sake of add-ons.

There would also be side-benefits in terms of memory usage, which is
always good to have.

So, can anybody think of good reason to not do this?

Cheers,
David

[Bobby Holley]

If the conversion is tractable and we end up with module ergonomics that
frontend developers are happy with, I'm certainly in favor of this from the
platform side. It would get us the 15-20MB of memory savings that bug
1186409 was pursuing without the smoke and mirrors.

bholley

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Bill McCloskey]

If we're going to do a mass conversion, shouldn't we try to move to ES
modules? There's some support for them in SpiderMonkey for chrome code, and
it would be great to move towards a future standard.

-Bill

[David Teller]

What's the current status of the implementation of ES6 modules? Also, to
use them in chrome code, can we support synchronous loading? Or would we
need to make the rewrite more complicated to support asynchronous loading?

[zbran...@mozilla.com]

If I understand correctly es module imports work differently from JSM's Cu.import in which scope they operate in.

JSM's are singletons, while importing in es imports code into your JS context.

How would you want to differentiate between those two modes?

Other things in the es import is that is that it cannot be loaded conditionally, while Cu.import can (that may be solved in the future ES import() proposal)

zb.

[jcop...@mozilla.com]

On Sunday, 25 September 2016 07:32:32 UTC+1, David Teller wrote:
> What's the current status of the implementation of ES6 modules?

ES6 modules are supported for chrome code, but not yet for content (pending spec related discussions that are not relevant for chrome).

It would be great if we could moving to using standard ES6 modules internally! If anyone is interested on working on converting the codebase then I can help with this.

Can you explain the requirement for synchronous loading? With ES6 modules all imports are determined statically and are loaded before the script is executed, and the spec does not currently provide an API to load a module, synchronously or otherwise.

Jon

[David Teller]

Ideally, it would be great to replace our current messy module loading
with something stricter. I suspect, however, that we have subtleties
that won't let us proceed. Let me detail a bit some of the problems that
might occur if we wish to rewrite existing code with a stricter module
loader.



* Side-effects

For one thing, I remember that some of our JS code defers loading its
dependencies (typically, using `XPCOMUtils.lazyModuleGetter`) to make
sure that this specific module is loaded after some startup code has
been properly initialized.

I don't remember the specifics, but I recall having seen it in or around
Services.jsm. I also recall that it is necessary for some tests that
mockup XPCOM components, so we need to ensure that the XPCOM components
have time to be installed before the code that depends upon them
actually instantiates them.

I suspect that this hairy behavior is quite the opposite of what ES6
modules are for, and that this may make it impossible to use them in
this context.



* Blocking C++ while JS code is being loaded

It is pretty common for C++ code to call JS code – typically, without
knowing that it's JS, thanks to XPCOM/XPConnect, expecting it to be a
regular function/method call.

If executing this JS code means that we need to somehow load modules,
this means that the loading needs to block the caller.

Is this the case already?



Cheers,
David

[Joshua Cranmer 🐧]

On 9/24/2016 5:13 PM, David Teller wrote:
> Which begs the question: what's the point of `Cu.import` these days?

One major difference between Cu.import and ES6/require-style modules is
that only one version of the script is created with Cu.import. This
allows you to make databases using Cu.import--every code that calls that
Cu.import file, whether a chrome JS file or an XPCOM component
implementation, will be guaranteed to see the same objects once the call
is made. There are definitely modules that rely on this.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

[David Teller]

In web content, that's also the case with ES6 and require-style modules.
I realize that it's a bit more complicated in chrome code, with all the
XUL + XBL + XPCOM + subscript loader, but I believe that we should be
able to reach the same result.

Cheers,
David

[Boris Zbarsky]

On 9/26/16 12:09 PM, David Teller wrote:
> In web content, that's also the case with ES6 and require-style modules.

No, only on a per-global basis.

Put another way, if you Cu.import the same thing from two instances of
browser.xul, you will get the same objects. But if import the same ES7
module from two different instances of the same webpage you get
_different_ objects.

-Boris

[David Teller]

I agree that my formulation was poor, but that's what I meant: in *a
single webpage*, all these modules behave the same wrt the underlying
objects.

[zbran...@mozilla.com]

So, it seems to me that we're talking about two aspects of module loading:


1) Singleton vs. per-instance

Cu.import allows us to share a single object between all the code that references it.

ES6 modules are not meant to do that.

2) Conditional vs. static

Cu.import allows us to decide *when* we're loading the code for side-effects, or even *if* we're going to load it at all.

if (needed) {
Cu.import(...);
}

or

XPCOMUtils.defineLazyModuleGetter(this, 'Services',
'resource://gre/modules/Services.jsm');

-----------------

The latter one may be resolved by some future ECMA proposals like:
- https://github.com/domenic/proposal-import-function
- https://github.com/benjamn/reify/blob/master/PROPOSAL.md

The former is a more tricky. I'm not sure how can we, within statement import world annotate the difference.
In the import-function world we could maybe do:

import('resource://gre/modules/Services.jsm', {singleton: true}).then();

but for static I don't see a semantically compatible way to annotate singleton reference.

zb.

[Anne van Kesteren]

On Mon, Sep 26, 2016 at 12:33 PM, <jcop...@mozilla.com> wrote:
> ES6 modules are supported for chrome code, but not yet for content (pending spec related discussions that are not relevant for chrome).

Can someone please publicly log this issue at
https://github.com/whatwg/html or wherever would be appropriate? It's
somewhat weird to keep the issue private and not ship the
implementation. Lots of folks are waiting on this feature and other
features (HTML modules) are blocked on it.


--
https://annevankesteren.nl/

[David Teller]

On 26/09/16 19:50, zbran...@mozilla.com wrote:
> So, it seems to me that we're talking about two aspects of module loading:
>
>
> 1) Singleton vs. per-instance
>
> Cu.import allows us to share a single object between all the code that references it.
>
> ES6 modules are not meant to do that.

If I understand ES6 modules correctly, two imports from the same webpage
will return the same module instance, right?

How hard would it be to consider all chrome code (of a JSRuntime) as a
single webpage? That's pretty much a requirement for any module loader
we would use for our chrome code.

> 2) Conditional vs. static
>
> Cu.import allows us to decide *when* we're loading the code for side-effects, or even *if* we're going to load it at all.
>
> if (needed) {
> Cu.import(...);
> }
>
> or
>
> XPCOMUtils.defineLazyModuleGetter(this, 'Services',
> 'resource://gre/modules/Services.jsm');
>
> -----------------
>
> The latter one may be resolved by some future ECMA proposals like:
> - https://github.com/domenic/proposal-import-function
> - https://github.com/benjamn/reify/blob/master/PROPOSAL.md
>
> The former is a more tricky. I'm not sure how can we, within statement import world annotate the difference.
> In the import-function world we could maybe do:
>
> import('resource://gre/modules/Services.jsm', {singleton: true}).then();
>
> but for static I don't see a semantically compatible way to annotate singleton reference.

I *think* that we can get rid all instances of the former, but I also
think that it's a multi-year project to do it all across our code.

@zb, do you think that it would be possible to have a migration path
from jsm towards ES6 modules that would let us do it one module at a
time? Let's assume for the moment that we can rewrite `Cu.import` to
somehow expose ES6 modules as jsm modules.

Also, how would you envision add-ons (which could add some kind of
modules dynamically) in a world of ES6 modules?


3) The issue of loading the source code

All module systems need to load their source before proceeding. If I
understand correctly, ES6 modules rely upon the same network stack as
the rest of Gecko to load the source code, while jsm rely only upon the
much more limited nsIJar* and nsILocalFile.

Barring any mistake, some of our network stack is written in JS. @zb, do
you see any way to untangle this?


4) The issue of pausing C++

There is still the issue of C++ code calling JS code and expecting it to
return only once it has entirely loaded + . Currently, this is made
possible by `Cu.import` performing a blocking read on the source file.

It is my understanding that ES6 modules, being designed for the web,
don't expect any kind of sync I/O, and "just" block `onload`.
Transitioning to ES6 modules would undoubtedly require some hackery here
to pause the calling C++ code.



5) The issue of the backstage wrapper

Currently, a number of tests rely upon `Cu.import` to expose the
backstage pass wrapper, i.e.

let {privateSymbol} = Cu.import(...);
// `privateSymbol` was not exported, but hey, it's still here.

Well, I *hope* it's just tests.

We would need a way to keep these tests working with ES6 modules.
Perhaps by requiring these tests to continue using a `Cu.import`
modified to work with ES6 modules.



That's all from the top of my head. At this stage, I suspect that the
best gain/effort ratio is migrating to RequireJS modules, but I'd be
happy to be proved wrong.

Cheers,
David

[Gijs Kruitbosch]

On 27/09/2016 10:28, David Teller wrote:
>
>
> On 26/09/16 19:50, zbran...@mozilla.com wrote:
>> So, it seems to me that we're talking about two aspects of module loading:
>>
>>
>> 1) Singleton vs. per-instance
>>
>> Cu.import allows us to share a single object between all the code that references it.
>>
>> ES6 modules are not meant to do that.
>
> If I understand ES6 modules correctly, two imports from the same webpage
> will return the same module instance, right?
>
> How hard would it be to consider all chrome code (of a JSRuntime) as a
> single webpage? That's pretty much a requirement for any module loader
> we would use for our chrome code.

I don't see how you would do this, because the globals *would* be
different for different windows (ie 2 copies of browser.xul windows),
and for XPCOM components. Even if our module loader had magic that let
this all happen without duplicating the modules themselves, it feels
like all kinds of static analysis and tools that we'd be doing this for
would break (because modules could never assume that |window| was a
thing in their global, or that it was always the same, but the tools
would assume that they could).

> 3) The issue of loading the source code
>
> All module systems need to load their source before proceeding. If I
> understand correctly, ES6 modules rely upon the same network stack as
> the rest of Gecko to load the source code, while jsm rely only upon the
> much more limited nsIJar* and nsILocalFile.

You've not really given enough detail here to explain why this is a
problem. You can pass chrome and jar: URIs to an XHR (obviously you get
security exceptions if you try this from the web...), and to
NetUtil.newChannel, etc. etc. - it's not clear to me why it'd be a
problem to use those to load the source code.

> Barring any mistake, some of our network stack is written in JS. @zb, do
> you see any way to untangle this?

This would only be a problem if you needed the JS-y bits of the network
stack to load those JS modules or components, which I don't think is the
case - that would surely also cause problems if it was the case with
Cu.import. Maybe I'm misunderstanding what problem you're trying to
identify?

~ Gijs

[David Teller]

I have opened bug 1305669 with one possible strategy for migrating
towards RequireJS.

Cheers,
David

[David Teller]

On 27/09/16 11:58, Gijs Kruitbosch wrote:
> On 27/09/2016 10:28, David Teller wrote:
>> How hard would it be to consider all chrome code (of a JSRuntime) as a
>> single webpage? That's pretty much a requirement for any module loader
>> we would use for our chrome code.
>
> I don't see how you would do this, because the globals *would* be
> different for different windows (ie 2 copies of browser.xul windows),
> and for XPCOM components. Even if our module loader had magic that let
> this all happen without duplicating the modules themselves, it feels
> like all kinds of static analysis and tools that we'd be doing this for
> would break (because modules could never assume that |window| was a
> thing in their global, or that it was always the same, but the tools
> would assume that they could).

I don't follow.

Fwiw, I'm thinking of Facebook's Flow, which is designed for use with
Node.js (so, no `window`) and modules.

>> 3) The issue of loading the source code
>>
>> All module systems need to load their source before proceeding. If I
>> understand correctly, ES6 modules rely upon the same network stack as
>> the rest of Gecko to load the source code, while jsm rely only upon the
>> much more limited nsIJar* and nsILocalFile.
>
> You've not really given enough detail here to explain why this is a
> problem. You can pass chrome and jar: URIs to an XHR (obviously you get
> security exceptions if you try this from the web...), and to
> NetUtil.newChannel, etc. etc. - it's not clear to me why it'd be a
> problem to use those to load the source code.

I'm talking about ES6 modules, which (if I read their code correctly)
use a built-in loading mechanism, already implemented in Gecko. Are you
talking of the same thing?

Of course, we could decide to write code using ES6 modules and compile
it away at build time. Is this what you had in mind?

>> Barring any mistake, some of our network stack is written in JS. @zb, do
>> you see any way to untangle this?
>
> This would only be a problem if you needed the JS-y bits of the network
> stack to load those JS modules or components, which I don't think is the
> case - that would surely also cause problems if it was the case with
> Cu.import. Maybe I'm misunderstanding what problem you're trying to
> identify?

Well, Cu.import doesn't have this problem because it doesn't rely on any
JS code – the only I/O, in particular, is performed through nsIJarURL
and nsILocalFile, both of which are implemented in C++.

But yeah, I may be wrong. If Necko's C++ code can handle gracefully (and
without failing) the fact that some of Necko's JS code is not loaded
yet, this may not be a problem. I'm not familiar enough with that part
of the code.


Cheers,
David

[David Bruant]

Le mardi 27 septembre 2016 14:49:36 UTC+2, David Teller a écrit :
> I have opened bug 1305669 with one possible strategy for migrating
> towards RequireJS.

RequireJS [1] is a peculiar choice for chrome code especially if your goal is static analysis.

From this thread and what I read in bug, it doesn't seem you want require.js.

I'll take a minute to try to setup some vocabulary. There are currently mostly 3 module formats in use in the front-end development.
* Aynchronous Module Definition (AMD) which first/main loader implementation is require.js [1]. Its motivation was mostly to provide a way to load modules asynchronously in front-end code. Example of how it looks:
* CommonJS (which Node.js is the main user right now, but front-end can use this syntax via tools like browserify and webpack). This form of module imports modules via the syntax 'var x = require('x')'
* ES6 modules which defines both the static syntax ('import' and 'export' keywords) and the (dynamic) module loader [2]. The latter is not ready yet but is getting there.

From what I've read, it looks like you want to transition to CommonJS, not AMD.
If you want a CommonJS loader, be aware that there is already one at https://hg.mozilla.org/mozilla-central/file/66a77b9bfe5d/devtools/shared/Loader.jsm


On the topic of transitioning, I don't maintain the Firefox codebase, so feel free to ignore anything I say below.
But for one-time top-level imports, the ES6 syntax seems like a better bet given from what I've read that they're supported in chrome and are the end-game.
As far as dynamic/conditional imports, there doesn't seem to be much value to move from Cu.import() to require() given it's unlikely static analysis tools will do anything with either anyway (I'm interested in being proven wrong here though) and the standard module loader [2] will beg for another rewrite eventually.

hope that helps,

David

[1] http://requirejs.org/
[2] https://whatwg.github.io/loader/ & https://github.com/whatwg/loader/pull/152/files

[David Teller]

You are right, I wrote RequireJS but I was thinking CommonJS, much as is
used currently in DevTools and Jetpack.

According to their documentation, Facebook's Flow analysis already
supports CommonJS modules [1]. Of course, they prefer ES6 modules. It
just remains to be seen whether we can migrate to these.

Cheers,
David

[1] https://flowtype.org/docs/modules.html#_

On 27/09/16 17:00, David Bruant wrote:
> Le mardi 27 septembre 2016 14:49:36 UTC+2, David Teller a écrit :
>> I have opened bug 1305669 with one possible strategy for migrating
>> towards RequireJS.
>
> RequireJS [1] is a peculiar choice for chrome code especially if your goal is static analysis.

[...]

> On the topic of transitioning, I don't maintain the Firefox codebase, so feel free to ignore anything I say below.
> But for one-time top-level imports, the ES6 syntax seems like a better bet given from what I've read that they're supported in chrome and are the end-game.
> As far as dynamic/conditional imports, there doesn't seem to be much value to move from Cu.import() to require() given it's unlikely static analysis tools will do anything with either anyway (I'm interested in being proven wrong here though) and the standard module loader [2] will beg for another rewrite eventually.
>
> hope that helps,
>
> David
>
> [1] http://requirejs.org/
> [2] https://whatwg.github.io/loader/ & https://github.com/whatwg/loader/pull/152/files

[Zibi Braniecki]

On Tuesday, September 27, 2016 at 2:28:54 AM UTC-7, David Teller wrote:
> If I understand ES6 modules correctly, two imports from the same webpage
> will return the same module instance, right?

I don't think this is a correct statement across globals.

When you load two modules in one js context, maybe, but when you have two browser.xul windows open and you load a JSM, it's shared between them.

> How hard would it be to consider all chrome code (of a JSRuntime) as a
> single webpage? That's pretty much a requirement for any module loader
> we would use for our chrome code.

So this opens up an interesting can of worms.
As we move into multi-process world, would we be interested in making our module loading code make it less impossible to chunk chrome into separate processes?

> I *think* that we can get rid all instances of the former, but I also
> think that it's a multi-year project to do it all across our code.

I don't see how or why would we want to get rid of all instances of the former.

It seems to me that we use the nature of singletons quite a lot - for example, I'm working on a replacement for chrome registry for l10n resources and I use runtime global cache by just having a single cache object in my JSM.

> @zb, do you think that it would be possible to have a migration path
> from jsm towards ES6 modules that would let us do it one module at a
> time? Let's assume for the moment that we can rewrite `Cu.import` to
> somehow expose ES6 modules as jsm modules.

I don't see a reason why wouldn't it be possible. We could even start by just promoting the new method for new code.

> Also, how would you envision add-ons (which could add some kind of
> modules dynamically) in a world of ES6 modules?

I linked to the ECMA proposals that give us nested/conditional imports.
I believe that we should go this route.

> It is my understanding that ES6 modules, being designed for the web,
> don't expect any kind of sync I/O, and "just" block `onload`.
> Transitioning to ES6 modules would undoubtedly require some hackery here
> to pause the calling C++ code.

Quite the opposite.

The first version of es6 modules is synchronous, even static, from the perspective of the user.

Only the import() function proposal introduces async way to load modules.

I see a huge value in both so I'd be happy if we implemented both internally and through this participated in the evolution of the import function proposal.


personal preference would be to not settle on the intermittent module loading API. If we want to move, let's go all the way and do this right.

One idea that came to mind for how could we differentiate between singleton loading and in-context loading using statements would be to differentiate by path.
Not sure if it feels right or a dirty hack, but sth like this:

import { Registry } from 'resource://gre/modules/L10nRegistry.jsm';

would work like Cu.import,

import { Registry } from 'resource://gre/modules/my-file.js';

would load my-file in-context.

Does it sound like a cool way of solving it or a terrible way of complicating it?

zb.

[David Teller]

On 27/09/16 19:35, Zibi Braniecki wrote:
> On Tuesday, September 27, 2016 at 2:28:54 AM UTC-7, David Teller wrote:
>> If I understand ES6 modules correctly, two imports from the same webpage
>> will return the same module instance, right?
>
> I don't think this is a correct statement across globals.
>
> When you load two modules in one js context, maybe, but when you have two browser.xul windows open and you load a JSM, it's shared between them.
>
>> How hard would it be to consider all chrome code (of a JSRuntime) as a
>> single webpage? That's pretty much a requirement for any module loader
>> we would use for our chrome code.
>
> So this opens up an interesting can of worms.
> As we move into multi-process world, would we be interested in making our module loading code make it less impossible to chunk chrome into separate processes?

That's too many negations for my poor brain :)

I think that we want to keep the current behavior of
one-chrome-module-has-only-one-instance-per-JSRuntime. Anything else
will introduce impossible-to-track bugs. From what I read below in your
message, I believe that we agree.

>> I *think* that we can get rid all instances of the former, but I also
>> think that it's a multi-year project to do it all across our code.
>
> I don't see how or why would we want to get rid of all instances of the former.

"why": because you wrote "The former is a more tricky." in your previous
message. If it's not, I'm quite happy to not remove them :)

For reference, "the former" is a snippet such as:

if (needed) {
Cu.import(...);
}

to which I would add

function foo() {
Cu.import(...);

}


>
> It seems to me that we use the nature of singletons quite a lot - for example, I'm working on a replacement for chrome registry for l10n resources and I use runtime global cache by just having a single cache object in my JSM.
>
>> @zb, do you think that it would be possible to have a migration path
>> from jsm towards ES6 modules that would let us do it one module at a
>> time? Let's assume for the moment that we can rewrite `Cu.import` to
>> somehow expose ES6 modules as jsm modules.
>
> I don't see a reason why wouldn't it be possible. We could even start by just promoting the new method for new code.

For this, we need first to make sure that two distinct .jsm modules/.xul
files/whatever chrome stuff that can load a ES6 module will receive the
same object when loading the same url, right? This seems like a pretty
important first step.

>> Also, how would you envision add-ons (which could add some kind of
>> modules dynamically) in a world of ES6 modules?
>
> I linked to the ECMA proposals that give us nested/conditional imports.
> I believe that we should go this route.

Ok. I think that works, but we should check with the addons team.

>> It is my understanding that ES6 modules, being designed for the web,
>> don't expect any kind of sync I/O, and "just" block `onload`.
>> Transitioning to ES6 modules would undoubtedly require some hackery here
>> to pause the calling C++ code.
>
> Quite the opposite.
>
> The first version of es6 modules is synchronous, even static, from the perspective of the user.
>
> Only the import() function proposal introduces async way to load modules.

I'm talking of the perspective of the embedder (here, Gecko). Reading
the implementation of ES6 modules, I have the impression that loading is
sync from the perspective of the JS & DOM but async from the perspective
of the embedder. Am I wrong?

If so, I fear that we're going to end up in gotcha cases whenever C++
calls JS code.

> I see a huge value in both so I'd be happy if we implemented both internally and through this participated in the evolution of the import function proposal.
>
>
> personal preference would be to not settle on the intermittent module loading API. If we want to move, let's go all the way and do this right.

Well, my personal preference is whatever doesn't require us to rewrite
the entire codebase of Firefox :)

>
> One idea that came to mind for how could we differentiate between singleton loading and in-context loading using statements would be to differentiate by path.
> Not sure if it feels right or a dirty hack, but sth like this:
>
> import { Registry } from 'resource://gre/modules/L10nRegistry.jsm';
>
> would work like Cu.import,
>
> import { Registry } from 'resource://gre/modules/my-file.js';
>
> would load my-file in-context.
>
> Does it sound like a cool way of solving it or a terrible way of complicating it?

I think it complicates stuff. Among other things, we have code that's
designed to be loaded both as a jsm and as a CommonJS module, and I'm
pretty sure that this would break all sorts of havoc.

Cheers,
David

[David Teller]

I have posted a draft of a plan for migrating from JSM to ES6 modules here:

https://gist.github.com/Yoric/2a7c8395377c7187ebf02219980b6f4d

Cheers,
David

[Kris Maglione]

On Sun, Sep 25, 2016 at 12:13:41AM +0200, David Teller wrote:
>So, can anybody think of good reason to not do this?

One major problem I see with this is that we currently lazily
import most modules the first time that a symbol they export is
referenced. If we move to CommonJS or ES6 modules, we need to
either:

a) Load essentially *all* of our Chrome JS at startup, before we
even draw the first window. Maybe the static dependency handling
of ES6 modules would make that more tractable, but I'd be
surprised.

b) Manually import modules whenever we need them. That might be
doable in CommonJS or with the proposed future dynamic imports
of ES6, but with a lot of additional cognitive overhead.

c) Use CommonJS, but with a lazy import helper. I wouldn't mind
that approach so much, but I think that it would pretty much
nullify any advantage for static analysis.

or,

d) Some hybrid of the above.

Frankly, I've been considering transitioning the code I work
with to CommonJS for a while, mainly because easier for outside
contributors to cope with (especially if they're used to Node).
Cu.import tends to hide the names of the symbols it exports
(which shows in how often our ESLint hacks fail to guess at what
it exports), and even defineLazyModuleGetter takes some getting
used to.

The main things that have been stopping me are the lack of
support for lazy imports, and the unfortunate impact that the
SDK loader has on debugging, with its mangling of exceptions,
and the source URL mangling imposed by the subscript loader. But
those problems can be overcome.

[David Teller]

\o/, let's join forces :)

I admit that I haven't thought at all about the impact on exceptions. If
we migrate to ES6 modules, then the problem is something that we don't
need to handle. If we migrate to CommonJS modules with a loader built
into XPConnect, I think that we can solve this without blood or pain.
More on this later.

I'm not entirely scared of lazy modules. There are obvious difficulties,
but I don't think that they are insurmountable.

I'll answer separately for CommonJS and ES6 modules, because the
problems we'll be facing are clearly different.

* CommonJS

As you mention in c), I'm pretty sure that we can trivially port
`defineLazyModuleGetter` to CommonJS, without changes to the client
code. See [1] for a few more details.

This will not be sufficient to get static analysis to understand lazy
imports, but I imagine that we can fix this as follows:

1. Introduce a `lazyRequire` function with the same signature and scope
as `require` and with the semantics of `defineLazyModuleGetter`.

2. Either teach our open-source linters that `lazyRequire` is `require`
or use Babel to rewrite `lazyRequire` to `require` before static analysis.



* ES6 modules

Indeed, ES6 modules don't like lazy imports. Theoretically, we could
port `defineLazyModuleGetter` using `processNextEvent` footgun magic,
but I would really hate to go in this direction.

I have put together in [2] a possible plan to migrate to ES6 modules
without breaking `defineLazyModuleGetter` – pending confirmation from
@jonco that this can be done. Essentially, past some point in the
migration, `Cu.import` becomes a sync version of the ES7's `import()`
function.

If this works (and it's a pretty big "if"), we can use more or less the
same steps as above.


Cheers,
David


[1] https://gist.github.com/Yoric/777effee02d6788d3abc639c82ff4488
[2] https://gist.github.com/Yoric/2a7c8395377c7187ebf02219980b6f4d

[Zibi Braniecki]

On Tuesday, September 27, 2016 at 2:01:09 PM UTC-7, David Teller wrote:

> "why": because you wrote "The former is a more tricky." in your previous
> message. If it's not, I'm quite happy to not remove them :)
>
> For reference, "the former" is a snippet such as:
>
> if (needed) {
> Cu.import(...);
> }
>
> to which I would add
>
> function foo() {
> Cu.import(...);
> }

It's tricky for migration, but it's extremely useful for performance and memory consumption.

I believe we want both nested and conditional imports.

zb.

[Nick Fitzgerald]

On Wed, Sep 28, 2016 at 1:19 AM, David Teller <dte...@mozilla.com> wrote:

>
> * CommonJS
>

​Just a heads up: the devtools have been using CommonJS modules and lazily
requiring modules for a couple years now. If you don't want to jump
directly to ES modules, reusing our infrastructure is probably a good idea.
We also run eslint on this stuff, but I don't know exactly how much
integration it has with our lazy requires.

http://searchfox.org/mozilla-central/search?q=lazyRequireGetter&case=false&regexp=false&path=
​