Loader simplifications

[Irakli Gozalishvili]

Hi Folks,

Last week we've talking a lot about changes we'll be making to a current jetpack architecture (to support multiprocess architecture or reduce complexity). Some of changes discussed were concerning a module loader. I'd  like to highlight ideas we've talked about, maybe even continue discussion here:

1. Special form of `require`:
   
   require('chrome!some/module')  
   
   Which would translate into:
   
   - Load 'some/module' in the chrome process.
   - Return message pipe channel, that can be used to exchange messages with 'some/module' loaded in chrome process.
   
   
   
2. Weather or not we need modules on the chrome process:
   
   Loader code is big, complex and has many dependencies. Minimizing code running on chrome is a primary goal of multiprocess  
   architecture, there for avoiding bunch of (module loader) code is a good idea. 
   
   
3. Module loader simplification:
   
   Current module loader was designed with a lot of flexibility in mind, which was great at a time of experimentation, but now this flexibility adds complexity. Simple module loader would allow us share modules on chrome, addon and content processes. I proposed idea of using   
   platform capabilities (https://developer.mozilla.org/en/JavaScript_code_modules/Using) for module loading. I also took time to implement a prototype that already follows manifest generated by linker and also understands relative (./foo/bar ../bar) and absolute (package/module) type require.  
   
   https://github.com/mozilla/addon-sdk/pull/212
   
   This can be used as a replacement of the following combination:
   
   https://github.com/mozilla/addon-sdk/blob/master/python-lib/cuddlefish/app-extension/bootstrap.js
   https://github.com/mozilla/addon-sdk/blob/master/python-lib/cuddlefish/app-extension/components/harness.js
   https://github.com/mozilla/addon-sdk/blob/master/packages/api-utils/lib/cuddlefish.js
   https://github.com/mozilla/addon-sdk/blob/master/packages/api-utils/lib/securable-module.js
   
   Would require some changes to a linker though, in order to avoid few hacks used right now. 
   
   
4. Making stdlib package-less.
   Current layout of SDK modules is complicated, packages, addon-kit, api-utils. Would be nicer to have just lib. This also would kill primary reason for complex module search logic.  
   
   
5. Simplifications to the module search:
   Current module search logic is quite complicated. NodeJS on the other hand understands only two forms of require:
   
   - Relative 
   
   require('./foo')
   require('../bar')
   
   - Absolute
   
   require('foo/bar') // requires `bar` module from `foo` package
   
   Hopefully with #4 we will be able to get rid of complicated search algorithm. (Also with #4 require('panel') will still work)
   
   
   

I'm most curious to hear your feedback on #3 (specially yours @warner as it closely relates to a linker).

I think simplified module search #4 and loader #3 which depends on package-less stdlib #4 would help a lot in implementing E10S, so I'd like to take those steps first. 

Regards
--
Irakli Gozalishvili
Web: http://www.jeditoolkit.com/
Address: 29 Rue Saint-Georges, 75009 Paris, France

[Christoph Dorn]

On 11-07-16 9:05 AM, Irakli Gozalishvili wrote:
> 1. Special form of `require`:

>
> require('chrome!some/module')
>
> Which would translate into:
>
> - Load 'some/module' in the chrome process.
> - Return message pipe channel, that can be used to exchange messages
> with 'some/module' loaded in chrome process.

-1

I am against require hints (by prefixing "hint!") other than for "text!"
or other plugins that only act on a string/file and return the result
(without memoizing the initialized module).

Placing "chrome!" in require() is IMO the wrong as it requires the
developer to use "chrome!" wherever the module is required in the
application. What if one forgets to add it in one place? Will the same
module always be returned (or piped)?

Also, when looking at the module to be included in chrome, how do we
know it is intended to run in chrome? i.e. how would the developer know
when to add "chrome!". This is a decision the average developer should
*NOT* have to make especially when using stock modules.

IMO, modules belong into packages and packages define their internal
environment. Furthermore the loader can be instructed to load packages
into different runtimes (processes) based on the setup of the SDK or
application.

To make a chrome process module available I would suggest something like
the following:

1) Define a package called "contentSDK" which contains
"privilegedModule" and mark the package to run in the content processes.
2) Define a package called "chromeSDK" which contains
"privilegedModule" and mark the package to run in the chrome process.
3) When a module requires("contentSDK/privilegedModule") a bridge is
setup by "contentSDK/privilegedModule" to
requires("chromeSDK/privilegedModule") in the chrome process using
whatever API will be available for that.

This allows "privilegedModule" in the two packages to setup a stock or
custom communication channel. "contentSDK/privilegedModule" could even
communicate with module running in a different application.

Obviously there is a lot more detail to this than I have outlined, but I
hope it shows the package-based approach where packages and the way they
are used defines the runtime for the contained modules VS asking the
developer to declare this with every module require and using a magic
message pipe that the chrome module may not support (because it sends
too much data, etc...).

> 2. Weather or not we need modules on the chrome process:

>
> Loader code is big, complex and has many dependencies. Minimizing
> code running on chrome is a primary goal of multiprocess
> architecture, there for avoiding bunch of (module loader) code is a
> good idea.

We absolutely need modules on the chrome process. Modules are the way
forward and to not support them in all contexts is a big fail in my books.

You could argue that modules on the chrome process are not necessarily
needed for simple jetpack extensions (the primary audience initially)
but they are important when building more advanced applications such as
what I am doing.

If the loader is too complex use a simpler one which should be trivial
as it does not need to know much about security as it all runs in chrome.

> 3. Module loader simplification:

>
> Current module loader was designed with a lot of flexibility in
> mind, which was great at a time of experimentation, but now this
> flexibility adds complexity. Simple module loader would allow us
> share modules on chrome, addon and content processes. I proposed
> idea of using
> platform capabilities
> (https://developer.mozilla.org/en/JavaScript_code_modules/Using) for
> module loading. I also took time to implement a prototype that
> already follows manifest generated by linker and also understands
> relative (./foo/bar ../bar) and absolute (package/module) type require.
>
> https://github.com/mozilla/addon-sdk/pull/212
>
> This can be used as a replacement of the following combination:
>
> https://github.com/mozilla/addon-sdk/blob/master/python-lib/cuddlefish/app-extension/bootstrap.js
> https://github.com/mozilla/addon-sdk/blob/master/python-lib/cuddlefish/app-extension/components/harness.js
> https://github.com/mozilla/addon-sdk/blob/master/packages/api-utils/lib/cuddlefish.js
> https://github.com/mozilla/addon-sdk/blob/master/packages/api-utils/lib/securable-module.js
>
> Would require some changes to a linker though, in order to avoid few
> hacks used right now.

Not sure about the impact of this. I would have to see if it breaks my
CommonJS loader I have sitting on top of jetpack:

https://github.com/pinf/loader-js/tree/master/demos/JetpackExtension

> 4. Making stdlib package-less.

> Current layout of SDK modules is complicated, packages, addon-kit,
> api-utils. Would be nicer to have just lib. This also would kill
> primary reason for complex module search logic.

-1

Packages are essential. If you expect your users to use packages you
should be using them for the SDK.

There is no reason why packages should complicate module search logic if
you have a clean idea of what packages are and do. The lookup can be
completely deterministic with no searching.

> 5. Simplifications to the module search:

> Current module search logic is quite complicated. NodeJS on the
> other hand understands only two forms of require:
>
> - Relative
>
> require('./foo')
> require('../bar')
>
> - Absolute
>
> require('foo/bar') // requires `bar` module from `foo` package

+1

> Hopefully with #4 we will be able to get rid of complicated search
> algorithm. (Also with #4 require('panel') will still work)

Question is will "foo" be "foo" for *ALL* modules in the entire addon or
will "foo" be only mapped for the current package?

I don't have a preference either way because I am using the PINF loader
to bootstrap a full CommonJS environment with package mappings for my
addons:

https://github.com/pinf/loader-js/tree/master/demos/JetpackExtension

> I think simplified module search #4 and loader #3 which depends on
> package-less stdlib #4 would help a lot in implementing E10S, so I'd
> like to take those steps first.

I understand the need to simplify and streamline, but please do not put
developer comfort in front of good architectural and logical design. You
can always add sugar (APIs, defaults, docs) later to make it easier for
newcomers.

Christoph

[Irakli Gozalishvili]

Sorry if did not make myself clear, changes described will not affect add-on developers relying on high level APIs! 

On Saturday, 2011-07-16 at 20:15 , Christoph Dorn wrote:

On 11-07-16 9:05 AM, Irakli Gozalishvili wrote:

1. Special form of `require`:

require('chrome!some/module')

Which would translate into:

- Load 'some/module' in the chrome process.
- Return message pipe channel, that can be used to exchange messages
with 'some/module' loaded in chrome process.

-1

I am against require hints (by prefixing "hint!") other than for "text!"
or other plugins that only act on a string/file and return the result
(without memoizing the initialized module).

Placing "chrome!" in require() is IMO the wrong as it requires the
developer to use "chrome!" wherever the module is required in the
application. What if one forgets to add it in one place? Will the same
module always be returned (or piped)?

Also, when looking at the module to be included in chrome, how do we
know it is intended to run in chrome? i.e. how would the developer know
when to add "chrome!". This is a decision the average developer should
*NOT* have to make especially when using stock modules.

That's not something average user will have to write, care or even know about. This is really low level stuff that SDK hackers will use to implement E10S.

 

IMO, modules belong into packages and packages define their internal
environment. Furthermore the loader can be instructed to load packages
into different runtimes (processes) based on the setup of the SDK or
application.

To make a chrome process module available I would suggest something like
the following:

1) Define a package called "contentSDK" which contains
"privilegedModule" and mark the package to run in the content processes.
2) Define a package called "chromeSDK" which contains
"privilegedModule" and mark the package to run in the chrome process.
3) When a module requires("contentSDK/privilegedModule") a bridge is
setup by "contentSDK/privilegedModule" to
requires("chromeSDK/privilegedModule") in the chrome process using
whatever API will be available for that.

This allows "privilegedModule" in the two packages to setup a stock or
custom communication channel. "contentSDK/privilegedModule" could even
communicate with module running in a different application.

Obviously there is a lot more detail to this than I have outlined, but I
hope it shows the package-based approach where packages and the way they
are used defines the runtime for the contained modules VS asking the
developer to declare this with every module require and using a magic
message pipe that the chrome module may not support (because it sends
too much data, etc...).

Moving away from packages is on the list, so packages approach is no go. Even if we ignore that, I really don't like mirrored modules approach, we tried similar thing before and main issue was that it's not 1 on 1 mapping. Usually diff modules from add-on process will need to get a message pipe channel associated with a same X module on the chrome process. Your suggested approach will require lot's of boilerplate code and implicit magic in the loader, loading same named modules in the other processes. Also, we actually will have three processes: chrome, addon and content so explicit way of obtaining message channel is a way more convenient way to go. Again, I don't thing you will have to care about it at all!     

 

2. Weather or not we need modules on the chrome process:

Loader code is big, complex and has many dependencies. Minimizing
code running on chrome is a primary goal of multiprocess
architecture, there for avoiding bunch of (module loader) code is a
good idea.

We absolutely need modules on the chrome process. Modules are the way
forward and to not support them in all contexts is a big fail in my books.

You could argue that modules on the chrome process are not necessarily
needed for simple jetpack extensions (the primary audience initially)
but they are important when building more advanced applications such as
what I am doing.

If the loader is too complex use a simpler one which should be trivial
as it does not need to know much about security as it all runs in chrome.

The whole point of the multiprocess work we are doing is to offload work from the chrome process to the add-on process. Doing anything complex  there goes against the main goal. I'd recommend doing advanced stuff on the add-on process instead.

stdlib in nodejs is not contained in packages, but that does not stops anyone from creating those. Advantage, there is that modules from

stdlib can be required easily: require('net'). We also want `require('panel')` to be short and simple as it's what most of users will type frequently.

As a user you won't notice any change.  

 

5. Simplifications to the module search:
Current module search logic is quite complicated. NodeJS on the
other hand understands only two forms of require:

- Relative

require('./foo')
require('../bar')

- Absolute

require('foo/bar') // requires `bar` module from `foo` package

+1

Hopefully with #4 we will be able to get rid of complicated search
algorithm. (Also with #4 require('panel') will still work)

Question is will "foo" be "foo" for *ALL* modules in the entire addon or
will "foo" be only mapped for the current package?

I don't quite understand your question, but stdlib modules will behave similar to nodejs require('net'). Again, users should not see any difference from this change.  

I don't have a preference either way because I am using the PINF loader
to bootstrap a full CommonJS environment with package mappings for my
addons:

https://github.com/pinf/loader-js/tree/master/demos/JetpackExtension

I think simplified module search #4 and loader #3 which depends on
package-less stdlib #4 would help a lot in implementing E10S, so I'd
like to take those steps first.

I understand the need to simplify and streamline, but please do not put
developer comfort in front of good architectural and logical design. You
can always add sugar (APIs, defaults, docs) later to make it easier for
newcomers.

Sorry, for not making it clear from the very beginning, as noted above, non of the changes described are going to affect add-on developers relying  on high level APIs.  

 

Regards!

Christoph

--
You received this message because you are subscribed to the Google Groups "mozilla-labs-jetpack" group.
To post to this group, send email to mozilla-la...@googlegroups.com.
To unsubscribe from this group, send email to mozilla-labs-jet...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mozilla-labs-jetpack?hl=en.

[Christoph Dorn]

Ah, ok.

>> IMO, modules belong into packages and packages define their internal
>> environment. Furthermore the loader can be instructed to load packages
>> into different runtimes (processes) based on the setup of the SDK or
>> application.
>>
>> To make a chrome process module available I would suggest something like
>> the following:
>>
>> 1) Define a package called "contentSDK" which contains
>> "privilegedModule" and mark the package to run in the content processes.
>> 2) Define a package called "chromeSDK" which contains
>> "privilegedModule" and mark the package to run in the chrome process.
>> 3) When a module requires("contentSDK/privilegedModule") a bridge is
>> setup by "contentSDK/privilegedModule" to
>> requires("chromeSDK/privilegedModule") in the chrome process using
>> whatever API will be available for that.
>>
>> This allows "privilegedModule" in the two packages to setup a stock or
>> custom communication channel. "contentSDK/privilegedModule" could even
>> communicate with module running in a different application.
>>
>> Obviously there is a lot more detail to this than I have outlined, but I
>> hope it shows the package-based approach where packages and the way they
>> are used defines the runtime for the contained modules VS asking the
>> developer to declare this with every module require and using a magic
>> message pipe that the chrome module may not support (because it sends
>> too much data, etc...).
>>
>>
> Moving away from packages is on the list, so packages approach is no go.

Can you elaborate. Surely only for internal stuff and not user packages?

> Even if we ignore that, I really don't like mirrored modules approach,
> we tried similar thing before and main issue was that it's not 1 on 1
> mapping. Usually diff modules from add-on process will need to get a
> message pipe channel associated with a same X module on the chrome
> process. Your suggested approach will require lot's of boilerplate code
> and implicit magic in the loader, loading same named modules in the
> other processes. Also, we actually will have three processes: chrome,
> addon and content so explicit way of obtaining message channel is a way
> more convenient way to go. Again, I don't thing you will have to care
> about it at all!

Well, only to the point of being able to run the PINF loader in the
various processes as it is the foundation of my addons. I suppose I will
have to try it out when the work is done and report back.

>>> 4. Making stdlib package-less.
>>> Current layout of SDK modules is complicated, packages, addon-kit,
>>> api-utils. Would be nicer to have just lib. This also would kill
>>> primary reason for complex module search logic.
>>
>> -1
>>
>> Packages are essential. If you expect your users to use packages you
>> should be using them for the SDK.
>>
>> There is no reason why packages should complicate module search logic if
>> you have a clean idea of what packages are and do. The lookup can be
>> completely deterministic with no searching.
>>
>>
> stdlib in nodejs is not contained in packages, but that does not stops
> anyone from creating those.

That is besides the point. In my view, you either have a pure
package-based system where all/most code resides in logical
groupings/packages that are individually versioned or you don't. Looks
like jetpack is not going to, and that is fine, but just not consistent
with where I thought jetpack was heading.

Having two packages for the public and internal stuff makes complete
sense to me.

> Advantage, there is that modules from
> stdlib can be required easily: require('net'). We also want

That is irrelevant. One can easily map one or more packages to the
global system search path (require.paths).

>>> 5. Simplifications to the module search:
>>> Current module search logic is quite complicated. NodeJS on the
>>> other hand understands only two forms of require:
>>>
>>> - Relative
>>>
>>> require('./foo')
>>> require('../bar')
>>>
>>> - Absolute
>>>
>>> require('foo/bar') // requires `bar` module from `foo` package
>>
>> +1
>>
>>> Hopefully with #4 we will be able to get rid of complicated search
>>> algorithm. (Also with #4 require('panel') will still work)
>>
>> Question is will "foo" be "foo" for *ALL* modules in the entire addon or
>> will "foo" be only mapped for the current package?
>
> I don't quite understand your question, but stdlib modules will behave
> similar to nodejs require('net'). Again, users should not see any
> difference from this change.

So "foo/bar" will sit on global `require.paths` for all modules in all
packages vs having "foo" mapped to `require("foo/*")` in the package
descriptor (package.json file) via "mappings" (CommonJS/Mappings/C).

Where will package names be defined? By the package.json file of the
package? How are you going to avoid package name conflicts?

Christoph