Error when trying to use sjs:bundle

[pcxunl...@gmail.com]

A bit of background: I created a Google Chrome Extension called Tab Organizer (https://github.com/Pauan/tab-organizer). It currently uses the Google Closure Compiler and I want to port it to use Conductance instead. I believe doing this will dramatically simplify the async code and reduce/remove bugs.

I'm already used to compiling, and the system is set up for compiling, so using the "sjs:bundle" module to load up my SJS files into a single JS file makes sense to me. I tried running the following command...

    conductance bundle --output="build/js/main.js" "./src/server/main.app"

...but I got this error message:

    Error: No module ID found for file:///home/pauan/Programming/Extensions/tab-organizer/src/server/main.sjs (missing a resource mapping?)

After doing a bit of testing, it seems the error message occurs if you try to load a file that doesn't exist. But, I checked and the file definitely *does* exist.

I then tried loading up the "sjs:bundle" module directly:

    @ = require("sjs:bundle")

    @create({

      output: "build/js/main.js",

      sources: [ "./src/server/main.sjs" ]

    })

But I got the same error message. Doing require("./src/server/main.sjs") in the Conductance shell works fine. 

FYI, I'm using the latest commit from the master branch of the Conductance repository on Github.

[pcxunl...@gmail.com]

I looked in the source code for the sjs:bundle module, and it seems that it throws an error for any file:// URL:

    if (!(id .. str.startsWith('file://'))) return id;

    throw new Error("No module ID found for #{id} (missing a resource mapping?)");

This seems strange to me.

[Tim Cuthbertson]

Hi,

Yep, the way the bundle module works is a little tricky. Even though it might be perfectly capable of resolving your modules right now (on disk, using their file://) paths, it _also_ needs to know the full URL those modules will live at from the browser's point of view, so that at runtime it can return the bundled module only in response to a require() on the matching URL. That's why it fails on file:// URLs, because it knows the browser will never be trying to load those.

So the "missing a resource mapping?" error is a somewhat subtle hint, but if you check the help text / docs you can see how to set resource mappings:

    --resource=ARG          Set the runtime URL (or server path) for an on-disk
                            location, e.g: --resource
                            components=/static/sjs/components --resource
                            /lib/nodejs/sjs=http://example.org/sjs NOTE: The
                            URLs used here must match the URLs used by your
                            running application, otherwise the bundled version
                            will be ignored.

If your server just serves up the current directory at the root, it's often as simple as adding:

--resource .=/

which will mean that all paths that resolve locally to `file://<PWD>/XXX` will be bundled into modules with a runtime URL of `document.location.origin + "/XXX"`.

This all seems a little more effort to understand than it should, but we haven't figured out a way to simplify the correspondence between "files on disk right now" and "URLs at runtime" so far. Suggestions welcome :)

Cheers,
 - Tim.

[pcxunl...@gmail.com]

Yup, after I sent my message I tried adding a "resources" and it worked. Perhaps you could just make "resources" optional, having it default to { ".": "/" }? Or at the very least make it clearer in the docs: there's no mention that "resources" is required, so I figured it was optional. In fact, since it's essentially required, throwing a helpful error message if it's not defined might be a good idea.

Now that I have the bundle created, I tried loading it up as a Chrome Extension. It correctly loaded my "./src/server/main" script (from the bundle), but then I got this error:

    Uncaught Error: Don't know how to load module at mho:std.sjs 

The "mho:std" module is included in the bundle. I tried adding in a "mho:" prefix to require.hubs, but I got the same error.

[Tim Cuthbertson]

On Wednesday, August 27, 2014 9:16:38 AM UTC+10, pcxunl...@gmail.com wrote:

Yup, after I sent my message I tried adding a "resources" and it worked. Perhaps you could just make "resources" optional, having it default to { ".": "/" }? Or at the very least make it clearer in the docs: there's no mention that "resources" is required, so I figured it was optional. In fact, since it's essentially required, throwing a helpful error message if it's not defined might be a good idea.

Technically it's not always required, though. If you bundle only modules that appear under an existing `require.hubs` entry (e.g sjs:, mho:, and any custom hub), it'll assume these same hubs are already defined on the client, in which case you don't need a resource mapping. I'll update the docs to clarify which options are (usually) required though.

I think ".":"/" is frequently a good default, but would lead to hard to diagnose issues whenever it's incorrect - everything would work fine, but the bundled URLs would be wrong and so everything would be loaded from the server anyway!

FWIW when using serving an .app file's bundled dependencies with Conductance we do take care of this automatically, since we know both the on-disk location and URL.

 

Now that I have the bundle created, I tried loading it up as a Chrome Extension. It correctly loaded my "./src/server/main" script (from the bundle), but then I got this error:

    Uncaught Error: Don't know how to load module at mho:std.sjs 

The "mho:std" module is included in the bundle. I tried adding in a "mho:" prefix to require.hubs, but I got the same error.

Hmm, that should definitely work - but yes, you will need to make sure mho: is actually on require.hubs at runtime. I don't think that error has anything to do with the bundle functionality, it sounds like SJS can't even resolve the module into a URL using require.hubs (which happens before it even checks the bundle). If you remove any reference to the generated bundle, I expect you'll see the same error.

I just tried it out, and it seems to work fine for me as a normal page:

$ conductance bundle mho:std --output bundle.js

[ ... ]

$ cat index.html

<script src="/__sjs/stratified.js"></script>

<script src="bundle.js"></script>

<script type="text/sjs">

require.hubs.unshift(["mho:","/__mho/modules/"])

require('mho:std').info("hi!");

</script>

When served via `conductance serve` (which gives me the __sjs and __mho paths for free, but doesn't otherwise affect how plain .html files are served), this works fine. Maybe double-check that your `mho:` hub is an _array_ of two strings as above, rather than just passing two arguments to unshift?

As an aside, it sounds like you're using .app files, but those are generally served up by the Conductance server. I'd assume a chrome extension would have to just use .html and .sjs files, because it wouldn't know what to do with an .app file. That doesn't sound like the source of your problem, but it seems odd regardless.

Cheers,

 - Tim.

[pcxunl...@gmail.com]

On Tuesday, August 26, 2014 2:34:46 PM UTC-10, Tim Cuthbertson wrote:

require.hubs.unshift(["mho:","/__mho/modules/"])

Using that worked. Earlier I tried doing this:

    require.hubs.addDefault(["mho:", ""])

But that didn't work. I also tried this:

    // Use the file: hub

    require.hubs.addDefault(["mho:", require.hubs[require.hubs.length - 3][1]])

But that didn't work either. Thanks to your message, I tried this:

    require.hubs.addDefault(["mho:", "/"])

And now it works. I assume this is because of how it handles URL concatenation. For instance, when I had "resources": { ".": "/" } it created the URL "//src/server/main.sjs", so I had to instead use { ".": "" }. It would be nice to have better error messages to detect these situations, but I got it working now at least.

I'd assume a chrome extension would have to just use .html and .sjs files, because it wouldn't know what to do with an .app file. That doesn't sound like the source of your problem, but it seems odd regardless.

Right, I'm using normal .sjs files, so I had to create my own HTML file that loads SJS and sets up the hubs and requires my main module.

I also had to manually add in the "sjs:http" and "sjs:regexp" modules to make "sjs:std" work, and the "sjs:service" module to make "mho:std" work. The dependency finder didn't pick them up.

My goal is to have a single program that will package up all my SJS files, bundle them into a single precompiled minified JS file, and have a minified stratified.js runtime that doesn't include the compiler (since I'm precompiling the SJS files). And I want to make sure that all of the modules are taken only from the bundle, no external file loading, to minimize file requests.

I understand my needs are pretty uncommon, and perhaps I would be the only one to benefit from it. But I figure that if somebody is going to use the bundle module, they are probably going to use it "all the way", compiling all their SJS into a single file. If so, they would benefit from a stratified.js that doesn't need the compiler. Similar to how people using Require.js bundle their files together and then use Almond (https://github.com/jrburke/almond) at runtime.

Also, Chrome Extensions aren't allowed to use eval, but currently stratified.js seems to require it. If I precompile SJS files using the bundler, how difficult would it be to create a stratified.js runtime that doesn't use eval/new Function/etc.? Right now I'm using a workaround to make eval work, but I don't know how long that will work, so I'd rather have an eval-free runtime.

I understand that right now SJS parses <script type="text/sjs"> blocks, and so I would have to find a different way to do things like register hubs, if I wanted to completely remove eval.

I'd be happy to help out with the coding on the SJS side, though it would take time for me to understand the existing code and tools.

[Tim Cuthbertson]

On Wednesday, August 27, 2014 12:03:44 PM UTC+10, pcxunl...@gmail.com wrote:

On Tuesday, August 26, 2014 2:34:46 PM UTC-10, Tim Cuthbertson wrote:

require.hubs.unshift(["mho:","/__mho/modules/"])

Using that worked. Earlier I tried doing this:

    require.hubs.addDefault(["mho:", ""])

But that didn't work. I also tried this:

    // Use the file: hub

    require.hubs.addDefault(["mho:", require.hubs[require.hubs.length - 3][1]])

But that didn't work either. Thanks to your message, I tried this:

    require.hubs.addDefault(["mho:", "/"])

And now it works. I assume this is because of how it handles URL concatenation. For instance, when I had "resources": { ".": "/" } it created the URL "//src/server/main.sjs", so I had to instead use { ".": "" }. It would be nice to have better error messages to detect these situations, but I got it working now at least.

Ahh yes, it's just using string concatenation - we should definitely add some better error detection around this. I think it should be possible to normalize this so it'll work the same regardless of whether you give it a slash or not.

 

I'd assume a chrome extension would have to just use .html and .sjs files, because it wouldn't know what to do with an .app file. That doesn't sound like the source of your problem, but it seems odd regardless.

Right, I'm using normal .sjs files, so I had to create my own HTML file that loads SJS and sets up the hubs and requires my main module.

I also had to manually add in the "sjs:http" and "sjs:regexp" modules to make "sjs:std" work, and the "sjs:service" module to make "mho:std" work. The dependency finder didn't pick them up.

For a bit of background, the bundler is conservative - so it will only bundle things that it can statically determine will _definitely_ be loaded.

This will often miss dependencies that are loaded dynamically, such as in the body of an if branch or a function. The solution is to manually include them in the bundle sources list, or use an explicit @require annotation in the module that uses it:

https://conductance.io/reference#sjs:%23language/metadata::%40require

In this case, those modules were simply missing from the @require annotations in sjs:std and mho:env - I've added them in now, thanks!

 

My goal is to have a single program that will package up all my SJS files, bundle them into a single precompiled minified JS file, and have a minified stratified.js runtime that doesn't include the compiler (since I'm precompiling the SJS files). And I want to make sure that all of the modules are taken only from the bundle, no external file loading, to minimize file requests.

I understand my needs are pretty uncommon, and perhaps I would be the only one to benefit from it. But I figure that if somebody is going to use the bundle module, they are probably going to use it "all the way", compiling all their SJS into a single file. If so, they would benefit from a stratified.js that doesn't need the compiler. Similar to how people using Require.js bundle their files together and then use Almond (https://github.com/jrburke/almond) at runtime.

Yes, I've thought about this in the past, but you're right, it's a bit of an uncommon case. Unfortunately, there are a number of common ways to use bundles that conflict with these requirements:

 - you usually want the browser to cache `stratified.js` separately (it'll be modified much less frequently than your bundle)

 - for the fastest load times, you will often bundle only modules that are needed on page load, leaving additional modules to load lazily over HTTP if/when they're needed.

That's not to say we shouldn't support the fully-bundled case you describe, it's just not our sole aim with the bundle module.

BTW, if you want everything as one file you can simply concatenate the generated bundle.js and stratified.js into one file (in that order).

I did try stripping the compiler out a while back. It's not as simple as just precompiling bundled modules, as we distribute some builtin modules as SJS source, so you also need to precompile those as well. In fact, stripping out the compiler makes stratified.js _larger_ because of this.

 

Also, Chrome Extensions aren't allowed to use eval, but currently stratified.js seems to require it. If I precompile SJS files using the bundler, how difficult would it be to create a stratified.js runtime that doesn't use eval/new Function/etc.? Right now I'm using a workaround to make eval work, but I don't know how long that will work, so I'd rather have an eval-free runtime.

That I can help you with! As a bit of an experiment a while back, we created the `stratified-aot.js` version of the runtime for use in Chrome apps (which also forbid `eval`). We haven't publicized this (it's rather experimental), but it does exist in the git repo :). This version does precompile the builtin modules I mentioned above, so I believe you could strip the compiler entirely. This mode really hasn't seen any testing, so there could be bugs lurking. But if you're game, you should be able to remove:

         "tmp/c1.js.min",

from the stratified-aot client build step (line 184 of src/build/buildscript.sjs @https://github.com/onilabs/stratifiedjs/blob/2a29dc37fdbe59b76c4f940f17736ed6bc8fdf02/src/build/buildscript.sjs#L184), and then run:

    ./sjs src/build/buildscript.sjs stratified-aot.js

If you decide to go this route, do let us know how you fare. We haven't had much cause to work on these sorts of environments, but I think it could be pretty cool (e.g chromebook apps could surely benefit from some stratification).

 

I understand that right now SJS parses <script type="text/sjs"> blocks, and so I would have to find a different way to do things like register hubs, if I wanted to completely remove eval.

You can use a `main` attribute on the stratified.js tag to specify a main module. I believe this will get around the eval issue, as long as the given module appears in a precompiled bundle.

<script src="stratified-aot.js" main="./init"></script>

(from http://onilabs.com/stratifiedjs#clientside).

The code in stratified.js to figure out which is the "main script tag" uses some heuristics, so if you're concatenating your own stratifiedjs+bundle file, you may still need to name it "stratified-<something>.js".

Cheers,

 - Tim.

 

[pcxunl...@gmail.com]

> In this case, those modules were simply missing from the @require annotations
> in sjs:std and mho:env - I've added them in now, thanks!

Yup, I figured that, which is why I specified which modules were missing from sjs:std and mho:std.

> you usually want the browser to cache `stratified.js` separately (it'll be
> modified much less frequently than your bundle)

I realized that after I posted my message: I'm actually using SJS in 3 different HTML pages, so having stratified.js be separate from the bundle is useful even in my use-case (not for latency reasons, but for file size reasons: having 3 copies of stratified.js means a bigger file size for my extension!).

> for the fastest load times, you will often bundle only modules that are
> needed on page load, leaving additional modules to load lazily over HTTP
> if/when they're needed.

Yup, that's good advice, though not applicable to my use-case (where all files are loaded from the disk no matter what).

> That's not to say we shouldn't support the fully-bundled case you describe,
> it's just not our sole aim with the bundle module.

Of course. I completely agree with having multiple ways to bundle things up, because people have different use cases.

> That I can help you with! As a bit of an experiment a while back, we created
> the `stratified-aot.js` version of the runtime for use in Chrome apps (which
> also forbid `eval`). We haven't publicized this (it's rather experimental),
> but it does exist in the git repo :).

I did see "stratified-aot.js" in the repo, but I figured it was intended for precompiling SJS code (I was unaware that even some builtins are written in SJS).

I replaced "stratified.js" with "stratified-aot.js" and used the "main" attribute to load my app, and it worked, no more eval! I'll let you know if I run into any problems/bugs.

I'll also try stripping out the compiler and see how well that works.

> The code in stratified.js to figure out which is the "main script tag" uses
> some heuristics, so if you're concatenating your own stratifiedjs+bundle
> file, you may still need to name it "stratified-<something>.js".

It won't work on older browsers, but document.currentScript (https://developer.mozilla.org/en-US/docs/Web/API/document.currentScript?redirectlocale=en-US&redirectslug=DOM%2Fdocument.currentScript) should help out a lot with that. You can then use a fallback for older browsers.

[pcxunl...@gmail.com]

I just tried out the build steps you gave, and it seems to work fine. Here's the file size stats:

    stratified.js  120,286 bytes
    stratified-aot.js  138,601 bytes
    stratified-aot-no-compiler.js  81,198 bytes

So it's definitely worth it for me to use the no-compiler version, especially since I need to use the aot version anyways (because of eval restrictions).

[pcxunl...@gmail.com]

I also tried heavily minifying them with UglifyJS2, here are the results:

    stratified.min.js  93,903 bytes
    stratified-aot.min.js  104,008 bytes
    stratified-aot-no-compiler.min.js  64,162 bytes

So the end result is that "stratified-aot-no-compiler.min.js" is about half the size of "stratified.js", which sounds great to me.

I'm sure you've already considered automatically minifying the SJS source code, perhaps you just don't want an extra dependency on something like UglifyJS?

[Tim Cuthbertson]

Thanks for the stats :) Did you use mangling, or just compression? I haven't used uglifyJS much, so I'm not sure what guarantees it provides about not breaking exported symbols.

Yeah, partially it's the extra dependency, and partially the concern that this translation could introduce its own bugs in our VM, which is hopefully unlikely but could be very hard to track down.

Also if you use gzip compression on your server, the difference between minified and unminified JS often isn't that big a deal, especially since our compiled output already strips comments. Of course that means we really ought to add gzip compression support to Conductance sometime soon if we want to make that argument ;)

Cheers,

 - Tim.

[pcxunl...@gmail.com]

I used the highest settings: full compression, mangling, and optimizations (including unsafe optimizations). UglifyJS shouldn't mangle global names (unless you tell it to), it only mangles local names. And it doesn't mangle names if there's eval/with in scope. So it's pretty safe. And it's very well tested, too, so bugs are unlikely.

Gzip is a great idea, but I don't think it'll benefit me personally, since my files are run locally from a .zip (since that's how Chrome Extensions work). It would be nice to have for people using servers, though. In fact, there's already some gzip modules out there, and I think Node.js comes with some built-in support as well.