Asynchronous module loading
Christoph Dorn
(Firefox Extension) to provide module loading capabilities and a
comprehensive set of base modules.
The extension creates sandboxes (XUL:browser) for given origin domains
and inserts chiron as well as app code into the sandboxed page.
In this setup I need chiron to be able to load modules not just from
the origin domain, but also from mozilla chrome://. To get around the
same origin policy when chiron loads modules I delegate the actual
loading of the file to the mozilla chrome context via asynchronous
events.
Everything is working beautifully except for some scope issues when
the module code is evaluated.
You can see my changes to modules.js here [1].
I was wondering if chiron already supports asynchronous module
loading? If not, is this something that could be added? It could look
something like this:
require('TestModule',function(){
// Module Loaded
});
The code in modules.js is difficult to follow for me and before I
attempt to add this I wanted to ask if it would work, it makes sense
to add and if anyone with more experience could add this much faster
than me.
Looking forward to your feedback.
Thanks for a great library!
Christoph
[1] - http://github.com/cadorn/chiron/blob/f6a8798efc43a2f148020a070537906aa807a7c0/src/chiron/modules.js
Kris Kowal
<christ...@christophdorn.com> wrote:
> I am in the process of incorporating chiron into my own project
> (Firefox Extension) to provide module loading capabilities and a
> comprehensive set of base modules.
Thanks for considering Chiron. I've been participating in a project
to construct a standard library for JavaScript that's presently called
"ServerJS" [1]. The version of Chiron you're working with was
constructed to conform to the Securable Modules [2] proposal that Ihab
Awad and I put forth to the group. There are roughly seven
server-side implementations of the module system, including one with
which I've been working, Narwhal [3]. You might find some of these
projects interesting since we share a common goal: to provide a body
of standard, interoperable JavaScript modules.
> The extension creates sandboxes (XUL:browser) for given origin domains
> and inserts chiron as well as app code into the sandboxed page.
Neat workaround.
> In this setup I need chiron to be able to load modules not just from
> the origin domain, but also from mozilla chrome://. To get around the
> same origin policy when chiron loads modules I delegate the actual
> loading of the file to the mozilla chrome context via asynchronous
> events.
To make this workaround safe, you will probably want to split your
architecture such that the loader exists in the chrome context and
operates as a frozen or sealed "capability object" with attenuated
authority, such that you validate that module identifiers can only
acquire pure-javascript modules within a managed section of the chrome
URI space and provide inert module factory functions, and also
"evaluate" said module factory functions in a context where the
global/primordial objects cannot be modified (deeply frozen on
references). Sandboxes can then safely be shared.
> Everything is working beautifully except for some scope issues when
> the module code is evaluated.
You may also run into unidentifiable Array objects if the scripts are
running in separate contexts. Each context gets its own Array type.
> I was wondering if chiron already supports asynchronous module
> loading? If not, is this something that could be added? It could look
> something like this:
>
> require('TestModule',function(){
> // Module Loaded
> });
We're presently discussing the future of asynchroneity on ServerJS [4]
[5]. Summarily, a callback approach is not likely to pan-out in the
long run, but using E-like "Promises" using the "ref_send" library, or
something similar, preferably more usable, would be an alternative.
The trouble is that, if any module may be loaded asynchronously, all
module loads must be treated as asynchronous. So the operative syntax
would be more likely to look like:
require('module').observe(function (module) {
});
But it would have to be ubiquitously applied. I've only glanced at
your patch, but it doesn't look like it's nearly enough code to handle
all the wonky cases you can run into when you mix synchronous and
asynchronous style module loading. For example, if you synchronously
load a module that asynchronously loads one of its dependencies,
there's a danger that you will use its API before it is ready. I had
a working version of Chiron three or so years ago that used
exclusively asynchronous style (script injection, at the time) so I
could probably assist in getting a loader that uses a dependency graph
instead of the stack to guarantee module observer execution order, but
it would involve a complete overhaul.
At any rate, it's a tough nut to crack. I'm looking into two
approaches as stopgaps. First, preloading, where the most frequently
used modules are bundled*minified*compressed and sync XHR is only used
when there's a memo miss on the module factories. This approach is
probably not useful to you since you cannot fall back to a synchronous
solution. Second would be a transform, that would translate
synchronous code to asynchronous code with different loaders and
loader semantics for the respective domains. This approach is fraught
with hazards since, with a naïve implementation, events might be
interpolated between the call and the return of any require function
that would violate your invariants (threads would no longer be
guaranteed mutual exclusion).
> The code in modules.js is difficult to follow for me and before I
> attempt to add this I wanted to ask if it would work, it makes sense
> to add and if anyone with more experience could add this much faster
> than me.
Yeah, modules.js is really thick :-)
It's my hope to develop Narwhal to the point that it can be used
client-side as well as server-side. It would be great to bring you on
board.
Kris Kowal
[1] - http://groups.google.com/group/serverjs
[2] - https://wiki.mozilla.org/ServerJS/Modules/SecurableModules
[3] - http://github.com/tlrobinson/narwhal/tree/master
[4] - http://groups.google.com/group/serverjs/browse_thread/thread/6b2f70e57423db36
[5] - http://groups.google.com/group/serverjs/browse_thread/thread/4fc999c996462a99
Christoph Dorn
> to construct a standard library for JavaScript that's presently called
> "ServerJS" [1]. The version of Chiron you're working with was
> constructed to conform to the Securable Modules [2] proposal that Ihab
> Awad and I put forth to the group. There are roughly seven
of it is over my head and too low level for my interests. I am glad it
is resulting in some nice libraries like chiron and Narwhal.
> server-side implementations of the module system, including one with
> which I've been working, Narwhal [3]. You might find some of these
> projects interesting since we share a common goal: to provide a body
> of standard, interoperable JavaScript modules.
further along with my prototype I am revisiting this decision. Narwhal
could be a better fit as it seems to provide more of the base
functionality that I am also looking at implementing.
Let me give you a little background about my project. I started
FirePHP [1] about 2 years ago which has grown to about 50,000 active
users. One of my goals with FirePHP was to include extensibility to
allow users to load their own "renderers" for logging data that are
injected into the Firebug console. While FirePHP has been successful I
have only had limited success with others writing extensions for it.
Having learnt from FirePHP and wanting to add "real" extensibility has
resulted in a new project called FireWidgets [2]. FireWidgets is a
generic widget system that you can embed into your application to
create UIs (as well as backend components, Mozilla Extensions and
Eclipse Views) using JavaScript modules.
The idea is to create one standard runtime environment primarily for
UI widgets that can be embedded into applications. The FireWidgets
project is one among several intended to improve the PHP toolchain [3]
but has many uses beyond that. The next version of FirePHP will be
built on top of FireWidgets.
> To make this workaround safe, you will probably want to split your
> architecture such that the loader exists in the chrome context and
> operates as a frozen or sealed "capability object" with attenuated
> authority, such that you validate that module identifiers can only
> acquire pure-javascript modules within a managed section of the chrome
> URI space and provide inert module factory functions, and also
> "evaluate" said module factory functions in a context where the
> global/primordial objects cannot be modified (deeply frozen on
> references). Sandboxes can then safely be shared.
time I simply create a sandbox in a browser and let chiron work within
the sandbox. We can refactor this towards a more refined solution as
we run more use-cases through it.
I have put up a demo here [4]. You need Firefox 3, install the
"DevExtension" and click on "HelloWorld" test. This test will load a
widget, construct a sandbox and inject chiron into it. You should get
an alert if everything is working properly. You can find the source
code here [5] (on the "cj" branch).
> We're presently discussing the future of asynchroneity on ServerJS [4]
> [5]. Summarily, a callback approach is not likely to pan-out in the
> long run, but using E-like "Promises" using the "ref_send" library, or
> something similar, preferably more usable, would be an alternative.
> The trouble is that, if any module may be loaded asynchronously, all
> module loads must be treated as asynchronous. So the operative syntax
> would be more likely to look like:
>
> require('module').observe(function (module) {
>
> });
>
> But it would have to be ubiquitously applied. I've only glanced at
> your patch, but it doesn't look like it's nearly enough code to handle
> all the wonky cases you can run into when you mix synchronous and
> asynchronous style module loading. For example, if you synchronously
> load a module that asynchronously loads one of its dependencies,
> there's a danger that you will use its API before it is ready. I had
> a working version of Chiron three or so years ago that used
> exclusively asynchronous style (script injection, at the time) so I
> could probably assist in getting a loader that uses a dependency graph
> instead of the stack to guarantee module observer execution order, but
> it would involve a complete overhaul.
the sandbox while I was refactoring my prototype to use joose and
chiron. My situation may be a bit different as I can ask chrome to
inject script files into the sandbox.
My goals are to:
- provide a sandbox bound to a specific domain
- allow sandbox code to load modules and make XHR requests cross-
domain based on a pre-established contract provided in a module spec
file
- provide seamless and secure asynchronous communication between
modules/widgets within the same sandbox, between sandboxes, with
modules running in chrome and server components written in any popular
server language
> At any rate, it's a tough nut to crack. I'm looking into two
> approaches as stopgaps. First, preloading, where the most frequently
> used modules are bundled*minified*compressed and sync XHR is only used
> when there's a memo miss on the module factories. This approach is
> probably not useful to you since you cannot fall back to a synchronous
> solution. Second would be a transform, that would translate
> synchronous code to asynchronous code with different loaders and
> loader semantics for the respective domains. This approach is fraught
> with hazards since, with a naïve implementation, events might be
> interpolated between the call and the return of any require function
> that would violate your invariants (threads would no longer be
> guaranteed mutual exclusion).
development if modules have to constantly be packaged. It could also
work for loading third party modules/widgets but would require the
publisher to process the sources into the bundles.
Requiring asynchronous loading may be the better solution as it would
also work in development without modification. I can easily require
FireWidget modules to use asynchronous loading calls but it would
exclude a potentially sizable number of modules that are coded for
synchronous loading including chiron etc...
I think we will need a loader that can combine several of these
approaches based on what you are trying to do with your widget/module.
> It's my hope to develop Narwhal to the point that it can be used
> client-side as well as server-side. It would be great to bring you on
> board.
core module system for chrome and sandboxes. Unfortunately the
intricacies of doing a proper integration and seeing what changes we
may need to make to Narwhal is pretty much over my head. I would
rather spend my time on building widgets that will run on the platform
to test what we can do with it.
I have been preparing my users (FirePHP) to test FireWidgets as alphas
become available. If we can collaborate on getting Narwhal integrated
I can focus on writing some cool demo apps and building an active
testing community around these.
I have plans to fully integrate Firebug and other debugging tools into
FireWidgets which will allow us to debug modules in realtime. This
could prove to be a great platform to develop ServerJS modules even if
the target platform is not FireWidgets.
I am open to suggestions and would love to work together on this.
Looking forward to your reply.
Christoph
[1] - http://ww.firephp.org/
[2] - http://github.com/cadorn/firewidgets/tree/master
[3] - http://workshop.christophdorn.com/
[4] - http://widgetdevapp.workshop.firewidgets.org/
[5] - http://github.com/cadorn/firewidgets/tree/cj
Kris Kowal
<christ...@christophdorn.com> wrote:
> I am open to suggestions and would love to work together on this.
I think you need a promise-based version of modules.js, but
unfortunately this means you won't be server-js, narwhal, or chiron
compatible in short order. Since I'm likely to need to do such a
thing eventually (although probably as a compilation target), and
since I consider Mark Miller a sort of mentor who I owe the favor of
learning how to work with Promises, I will probably get around to this
eventually. A hybrid would be messy and unlikely to bring any joy to
your users. I'll make an announcement here when I have something to
share.
Kris Kowal
Christoph Dorn
> I think you need a promise-based version of modules.js, but
> unfortunately this means you won't be server-js, narwhal, or chiron
> compatible in short order. Since I'm likely to need to do such a
>
> thing eventually (although probably as a compilation target), and
> since I consider Mark Miller a sort of mentor who I owe the favor of
> learning how to work with Promises, I will probably get around to this
> eventually. A hybrid would be messy and unlikely to bring any joy to
> your users. I'll make an announcement here when I have something to
> share.
>
I have specific ideas about what I want the final system to be able to
do, but there is a lot of room for how it is implemented underneath.
Our discussion has given me some ideas about other possibilities. With
FireWidgets I hope to explore what can be done by loading dynamic
modules into the browser in terms of customizing the browser and
extensions as well as connect modules to website functionality.
Reviewing our discussion and the FireWidgets architecture I am thinking
about a revised design:
- Treat chrome and snadbox as two completely different entities with
different technologies involved
Chrome:
- Integrate Narwhal to provide a chrome-based "server" that can be
used to load applications comprised of modules.
- Server modules can have complete access to all of chrome (including
browser and other extensions) just like browser extensions have.
- To load an application into the server the user "installs" the
application by trusting the module just like users trust extensions now
- Once an application module is trusted, dependant modules are
automatically trusted and loaded
- Provide an in-browser HTTP gateway for the server using Jack. This
can then be used to access application modules from web pages and
sandboxes in a controlled way
- Integrate Firebug and other debugging tools into the server
Sandbox:
- Stick to a simple "browser" where anything goes (limited by same
origin policy)
- Provide an optional API to access chrome via the HTTP service (this
can allow sandboxed applications access to all of chrome's features in a
monitored way)
- Applications can be completely pre-loaded by installing them as
complete bundles. If additional code is required the application can
deal with it's own loading which can include using the HTTP service as a
proxy to load cross domain. This would allow for synchronous and
asynchronous loading depending on how you choose to setup your sandboxed
application.
- Over time standardize to one or several application
toolkits/libraries/frameworks that provide base functionality to
encourage a more interchangeable ecosystem of sandbox modules.
Do you think Narwhal is suitable for this and Jack could be used tied
into something like the mozilla HTTP server [1]? Is this something that
fits with your Narwhal on the client interest?
Christoph
[1] - https://developer.mozilla.org/En/Httpd.js/HTTP_server_for_unit_tests