Module Loading

232 views
Skip to first unread message

bolinfest

unread,
Jul 23, 2010, 11:14:10 AM7/23/10
to Closure Library Discuss
Hi, I have some issues with module loading, so I wanted to get
feedback before putting together a patch. The library code that is
creating problems for me is in
goog.module.ModuleLoader.prototype.loadModulesInternal:

// In prod, we don't load via a script tag because it is difficult
to
// determine if the script has been loaded and to handle errors
conditions.
if (this.getDebugMode() && goog.userAgent.GECKO) {
// In debug mode on FF, we do not load via an XHR + eval as the
script will
// not show in firebug.

/* CODE IS LOADED VIA SCRIPT TAGS */
} else {
/* CODE IS LOADED VIA goog.net.BulkLoader */
}

The first issue is that debug mode only applies to Firefox. Why is
this? It is not obvious that setDebugMode() has no effect for non-
Gecko-based browsers.

The second issue is that a goog.net.BulkLoader is always used, even if
the URIs that contain the modules are on a different domain. Ideally,
I think that ModuleLoader should check to see whether the URIs are on
the same domain first before opting to use a goog.net.BulkLoader. As
the existing comment in the code points out, the JS should be made
available on the same domain so that it can be loaded with an XHR so
that failures are easier to monitor, but I can imagine scenarios where
someone might serve their JS on the same domain in production. (For
example, resources may be spread across subdomains to get around the
browser's connection limit.)

The simplest solution that would unblock me would be to drop the "&&
goog.userAgent.GECKO" clause from the conditional. In production, I
happen to serve the JS for the modules on the same domain, so the
second item isn't an issue for me personally, but I could fix it while
I'm already in the code if it is an issue for others.

If anyone is curious why I even have this problem, it is because I'm
adding support for modules to plovr (http://plovr.org/). Because plovr
is run on a separate domain during development (localhost:9810), the
main page (which is probably running on localhost:80 or localhost:
8080) needs to be able to load the modules from plovr, so XHRs won't
work.

Nick Santos

unread,
Jul 23, 2010, 12:06:18 PM7/23/10
to closure-lib...@googlegroups.com
On Fri, Jul 23, 2010 at 11:14 AM, bolinfest <boli...@gmail.com> wrote:
> Hi, I have some issues with module loading, so I wanted to get
> feedback before putting together a patch. The library code that is
> creating problems for me is in
> goog.module.ModuleLoader.prototype.loadModulesInternal:
>
>  // In prod, we don't load via a script tag because it is difficult
> to
>  // determine if the script has been loaded and to handle errors
> conditions.
>  if (this.getDebugMode() && goog.userAgent.GECKO) {
>    // In debug mode on FF, we do not load via an XHR + eval as the
> script will
>    // not show in firebug.
>
>    /* CODE IS LOADED VIA SCRIPT TAGS */
>  } else {
>    /* CODE IS LOADED VIA goog.net.BulkLoader */
>  }
>
> The first issue is that debug mode only applies to Firefox. Why is
> this?

IE and Webkit-based browsers will evaluate the script tags in a
non-deterministic order (*not* in the order they are added to the
document), so they will simply not work in this mode.

>
> The second issue is that a goog.net.BulkLoader is always used, even if
> the URIs that contain the modules are on a different domain. Ideally,
> I think that ModuleLoader should check to see whether the URIs are on
> the same domain first before opting to use a goog.net.BulkLoader. As
> the existing comment in the code points out, the JS should be made
> available on the same domain so that it can be loaded with an XHR so
> that failures are easier to monitor, but I can imagine scenarios where
> someone might serve their JS on the same domain in production. (For
> example, resources may be spread across subdomains to get around the
> browser's connection limit.)
>
> The simplest solution that would unblock me would be to drop the "&&
> goog.userAgent.GECKO" clause from the conditional. In production, I
> happen to serve the JS for the modules on the same domain, so the
> second item isn't an issue for me personally, but I could fix it while
> I'm already in the code if it is an issue for others.
>
> If anyone is curious why I even have this problem, it is because I'm
> adding support for modules to plovr (http://plovr.org/). Because plovr
> is run on a separate domain during development (localhost:9810), the
> main page (which is probably running on localhost:80 or localhost:
> 8080) needs to be able to load the modules from plovr, so XHRs won't
> work.
>

Some people have asked for a CrossDomainBulkLoader, but I don't know
if anyone has looked into how hard it would be. It's not as simple as
changing an if condition. Most people I know just use server-side
forwarding to play make-believe that the cross-domain resource is on
the same domain, but I don't know how widespread that practice is.

Nick

bolinfest

unread,
Jul 23, 2010, 1:29:16 PM7/23/10
to Closure Library Discuss
Hmm, what happens if you do something like:

var html = '<script src="foo.js"></script><script src="bar.js"></
script><script src="baz.js"></script>';
var node = goog.dom.htmlToDocumentFragment(html);

and then append node? Would that guarantee order?

On Jul 23, 12:06 pm, Nick Santos <nicksan...@google.com> wrote:

Nick Santos

unread,
Jul 23, 2010, 2:30:39 PM7/23/10
to closure-lib...@googlegroups.com
not sure. have you tried it?

bolinfest

unread,
Jul 23, 2010, 6:42:41 PM7/23/10
to Closure Library Discuss
OK, I tested this out by creating several test pages. In my own
experiments and Googling, I could not figure out a way to just append
the <script> tags in such a way to enforce load order. As Nick
mentioned, order is preserved on Firefox but not in Chrome:

http://bolinfest.com/closure/scriptdeterminism.html
http://bolinfest.com/closure/scriptdeterminism2.html

However, (on Chrome 5 on Linux, at least), using the onload handler of
a <script> tag to trigger the load of the next <script> tag does
indeed force the scripts to load in order (though the onload handler
for <script> is reportedly flaky):

http://bolinfest.com/closure/scriptdeterminism3.html

Unfortunately, loading script tags in series is also fairly slow.
Nevertheless, I think that having slow debug behavior that works
across domains is much better than having none at all.

Would a patch that adds this behavior to ModuleLoader in debug mode
for non-Gecko based browsers be welcome?

I suppose in the extreme case, I could have plovr serve the contents
of each file of the module as JSONP and then concatenate everything
together after everything has loaded.

Curious to hear your thoughts,
Michael

On Jul 23, 2:30 pm, Nick Santos <nicksan...@google.com> wrote:
> not sure. have you tried it?
>

Nick Santos

unread,
Jul 23, 2010, 7:01:34 PM7/23/10
to closure-lib...@googlegroups.com
On Fri, Jul 23, 2010 at 6:42 PM, bolinfest <boli...@gmail.com> wrote:
> OK, I tested this out by creating several test pages. In my own
> experiments and Googling, I could not figure out a way to just append
> the <script> tags in such a way to enforce load order. As Nick
> mentioned, order is preserved on Firefox but not in Chrome:
>
> http://bolinfest.com/closure/scriptdeterminism.html
> http://bolinfest.com/closure/scriptdeterminism2.html
>
> However, (on Chrome 5 on Linux, at least), using the onload handler of
> a <script> tag to trigger the load of the next <script> tag does
> indeed force the scripts to load in order (though the onload handler
> for <script> is reportedly flaky):
>
> http://bolinfest.com/closure/scriptdeterminism3.html
>
> Unfortunately, loading script tags in series is also fairly slow.
> Nevertheless, I think that having slow debug behavior that works
> across domains is much better than having none at all.
>
> Would a patch that adds this behavior to ModuleLoader in debug mode
> for non-Gecko based browsers be welcome?

How slow is "slow"? I guess slow is better than nothing. Go for it.

Idly thinking about the larger problem:

Ideally, we would pressure browser developers to just provide some
standard way to do this.

You could use the "// @sourceURL" trick, but it means that everyone
has to modify their serving infrastructure to add a @sourceURL and
serve the raw JS somewhere, which is a PITA. And even if you use the
"// @sourceURL" trick, it still doesn't help with stack traces:
http://code.google.com/p/v8/issues/detail?id=672

There are also progressively hackier things that you could do with
JSONP, etc. But none of them are really satisfying.

Nick

Ojan Vafai

unread,
Jul 23, 2010, 11:03:16 PM7/23/10
to closure-lib...@googlegroups.com
On Fri, Jul 23, 2010 at 3:42 PM, bolinfest <boli...@gmail.com> wrote:
OK, I tested this out by creating several test pages. In my own
experiments and Googling, I could not figure out a way to just append
the <script> tags in such a way to enforce load order. 

I don't know if it helps in this case, but if you document.write the script tags, they get loaded in order. The issue with document.write is that you need to do it before the load event in order to avoid clobbering the page. The order of script execution is clearly specified in the HTML5 spec when document parsing is involved (hence document.write). I couldn't find anything in the spec about appending via script. I didn't look too deeply though.

I'm attaching a modified version of your test page that uses document.write.

Ojan
scratch.html

Alex Russell

unread,
Jul 24, 2010, 12:04:01 AM7/24/10
to closure-lib...@googlegroups.com
On Fri, Jul 23, 2010 at 4:01 PM, Nick Santos <nicks...@google.com> wrote:
> On Fri, Jul 23, 2010 at 6:42 PM, bolinfest <boli...@gmail.com> wrote:
>> OK, I tested this out by creating several test pages. In my own
>> experiments and Googling, I could not figure out a way to just append
>> the <script> tags in such a way to enforce load order. As Nick
>> mentioned, order is preserved on Firefox but not in Chrome:
>>
>> http://bolinfest.com/closure/scriptdeterminism.html
>> http://bolinfest.com/closure/scriptdeterminism2.html
>>
>> However, (on Chrome 5 on Linux, at least), using the onload handler of
>> a <script> tag to trigger the load of the next <script> tag does
>> indeed force the scripts to load in order (though the onload handler
>> for <script> is reportedly flaky):
>>
>> http://bolinfest.com/closure/scriptdeterminism3.html
>>
>> Unfortunately, loading script tags in series is also fairly slow.
>> Nevertheless, I think that having slow debug behavior that works
>> across domains is much better than having none at all.
>>
>> Would a patch that adds this behavior to ModuleLoader in debug mode
>> for non-Gecko based browsers be welcome?
>
> How slow is "slow"? I guess slow is better than nothing. Go for it.
>
> Idly thinking about the larger problem:
>
> Ideally, we would pressure browser developers to just provide some
> standard way to do this.

Message received:

http://wiki.ecmascript.org/doku.php?id=strawman:simple_modules
http://wiki.ecmascript.org/doku.php?id=strawman:simple_modules_examples

> You could use the "// @sourceURL" trick, but it means that everyone
> has to modify their serving infrastructure to add a @sourceURL and
> serve the raw JS somewhere, which is a PITA. And even if you use the
> "// @sourceURL" trick, it still doesn't help with stack traces:
> http://code.google.com/p/v8/issues/detail?id=672
>
> There are also progressively hackier things that you could do with
> JSONP, etc. But none of them are really satisfying.

The Dojo approach to this has been to treat x-domain loading as the exotic
case and assume a tool had been run in order to enable it. That
let the build system do the hackish JSONP things while the loader only
needs to determine if the requested module base path is on or off
the local domain before requesting it.

Not perfect, but it at least allows workable CDN hosting of modules
with the significant caveat that you can't optimize nearly as
aggressively.

Regards

bolinfest

unread,
Jul 25, 2010, 6:02:48 PM7/25/10
to Closure Library Discuss
Hi Ojan, yes, that technique certainly works if you can use
document.write(), though the problem that I am mainly trying to solve
is for modules that are loaded dynamically, which almost always means
after the page has been loaded, so document.write() is not an option
in that case.

Also, doesn't document.write() result in doing synchronous script
loads?

After rethinking the JSONP stuff, I realized that concatenating
everything together would defeat my original goal, which was to load
each script tag separately so that it is easier to match up line
numbers in the loaded JS with the original JS, and that sort of thing.
I'll try to put the "onload" patch together this week.

Though as Nick points out, the best solution to this problem requires
working with browser vendors to provide an appropriate API.

On Jul 23, 11:03 pm, Ojan Vafai <o...@chromium.org> wrote:
>  scratch.html
> 2KViewDownload

bolinfest

unread,
Jul 27, 2010, 7:09:16 PM7/27/10
to Closure Library Discuss
I filed a bug:

http://code.google.com/p/closure-library/issues/detail?id=197

and submitted a patch today:

http://codereview.appspot.com/1848053/diff/1/2

any takers to review it?

Hochhaus, Andrew

unread,
Sep 20, 2012, 5:48:51 PM9/20/12
to closure-lib...@googlegroups.com
We have been exploring goog.debug.ErrorReporter with closure-compiler
V3 source maps to do server side stack trace logging (and
deobfuscation) for client side errors experienced in production. Good
browser support for stack traces (Error.stack) exists in IE10, Safari
6, FF, Chrome and Opera. However, of these, only Chrome supports "//@
sourceURL=" for Error.stack _stack traces_ (as opposed to in the
client side developer tools). This makes using
ModuleLoader#setSourceUrlInjection not optimal for production
exception logging.

For the non-chrome browsers, keeping stack trace information intact
could be better achieved by appending <script> tags
(ModuleLoader#setDebugMode). However, this comes with the problem that
not all browsers guarantee the order to appended script tags (after
the page has loaded). Therefore ModuleLoader resorts to serial script
loading in debug mode (potentially seriously hurting download
performance).

As best I can tell, this summaries the pros/cons of the two methods of
loading modules currently supported in ModuleLoader:

XHR w/ eval
+ parallel downloading
+ good error handling
+ ordered script execution
- double evaluation penalty (during module parsing)
- worse debugging/stack traces [@sourceURL only currently works on chrome]

Append <script>
+ native parsing performance
+ better debugging
- unordered script execution (on some browsers) results in serial download

As a middle of the road solution, what about the following hack?

* XHR to download the module (don't eval it)
* Ensure server sets caching headers
* Do file download related error handling using the XHR
* After XHR completes, drop string on the floor, instead append
<script> tag (serially, respecting module load order -- this should be
a browser cache hit)
* Use timeout w/ module notification for "stage 2" error handling

This would result in:

+ parallel downloading
+ good stack traces
+ ordered script execution
+ native parsing performance
- horrible perf if browser cache disabled/fails
- middle of the road error handling

However, maybe someone has an idea for how to mitigate the downsides. Thoughts?

-Andy

Max Nikitin

unread,
Jul 8, 2013, 8:22:50 AM7/8/13
to closure-lib...@googlegroups.com
> worse debugging/stack traces [@sourceURL only currently works on chrome] 
It doesn't work for me. How can I fix it? Is it google chrome problem?
Reply all
Reply to author
Forward
0 new messages