Re: [amd-implement] AMD delivering ES6 modules

[johnj...@chromium.org]

On Monday, March 3, 2014 3:22:45 PM UTC-8, James Burke wrote:

On Wed, Feb 19, 2014 at 11:56 AM, John Hann <jo...@unscriptable.com> wrote:
> I've been trying to use the spec *as is* and it's been very frustrating.
> It's not even close to being usable for anything but toy apps, imho.  Maybe
> we should start a discussion about how we could help steer the TC39 team?
> Does anybody wish to debate the ES6 Loader spec here?  Or should we do it
> somewhere else?

I agree. Any transpiler work has been of the trivial kind and a module system is really only usable when the module loader works. And AFAIK, to date, the loaders used by the transpilers have been AMD-based, or CJS.

We have to start somewhere.  I work on traceur's Loader and I am keen to hear about loader improvements. 

 

I have tried to give private feedback to the ES module designers ...

To be clear, I am not a module designer.

 

A list of changes I would prefer to see 

I’m skipping background descriptions and use case backup for these since I’m talking to an audience that should already have context:

1) Solve module inlining better. The committee feels this is a separable concern, that requires new network layer features, but there is a recognition it needs to exist via `loader.define(id, stringOfJavaScript)`. 

However, that is a horrible way to inline modules, and the other network layer stuff is still really speculative. It takes away time from people on committee from solving more of the actual module spec. Solve what people do today first. See about the speculative network layer changes like package URLs later. 

I gave specific proposals in this area, but they were discarded due to the inertia of past decisions/consensus.

I'm unsure of what you mean by 'inlining'.

 

2) The <module> HTML tag should die in a fire: the first script loaded likely needs to set up loader config, so it is a plain script, there is System.import for starting module loading, and better module inlining will address other inline module definition concerns. This is another example of effort that takes away from wrapping up modules and the module loader work. 

It also just looks like <newscript> tag: why not change more of the language if a new HTML tag is needed? (I don’t want that though) It also creates extra cognitive burden on any developer, using modules or not, when looking at the HTML docs for what tag to use for a script.

I think you misunderstand 3 things:

  1) HTML tags are not the business of TC39.

  2) The tag is not currently being advocated.

  3) <script type='module'> would amount to an anonymous module, that is 99% the same as a plain script.

 

3) They need to have module-specific variable(s) to allow the module a module-specific loader.import() that supports relative ID resolution, and loading those modules in the same loader instance that loaded the current module. They also need to expose the equivalent of `module.id` and `module.uri`. Those id and url properties might have something in the spec soon for them, but it is not clear how they will solve the local loader issue yet.

We have added two module-specific variables to our implementation.

 

4) Allow export default or named exports but not both. That would solve the issue that started this thread, and it would also allow `System.import(‘a’)` to just call a() if ‘a' exported a function as default. How it looks like now, you would have to do a.default() or some other named property, which is goofy, and indicates how `export default` was tacked on to the design later. This “one or the other” behavior is basically what AMD and CJS modules do now anyway, so a strong precedent for it.

On a per module basis or at the language level?  I'd like 'default' to die, but it seems that it was added for the benefit of AMD/CJS ;-)

 

5) The module loader needs a better way to load loader hooks. This is exactly what AMD loader plugins do: have a separator in the module ID to indicate the plugin module ID vs the plugin resource ID, and load the plugin, use its exports as the loader hooks for those kinds of IDs.

This will avoid a lot of footguns around people not overriding the existing ES loader hooks correctly, allows delayed loading and running of those hooks until there is a resource that needs them, and it would allow for the normalize and translate hooks to be synchronous. Or not if they want to be super flexible, keep those hooks async, but the other benefits still stand.

This will likely get pushback because of 6) though.

The loader hooks is not the problem IMO, the problem is using a mutable global for system configuration in a system that support module loading.  Too many hooks are async, though I think translate() may be a good candidate. However in a world where promises can do nothing wrong, I believe this battle is not one to fight.

 

6) Better baseline. This is likely the most contentious, and least likely to happen, so I put it last:

There will likely be claims that different JS envs need the capability to be different and many concerns are just for the browser. However, what most developers want are consistency: the same system works everywhere.

I agree with this much for sure. IMO the System should work for 99.9% of developers out of the box.

 

The core behavior of module loading in the ES design is really driven by browser constraints anyway, so favor that as the default consistency.

It is fine to allow per JS-environment variations, to allow for example, node code to have the nested module lookup. That will be possible with the imperative overrides of loader hooks. However, using the browser constraints and basic declarative config would give a better baseline. That declarative config can live fine alongside the imperative overrides.

I don't follow this part. 

 Other parts in this area:

* relative ID resolution should just be built in to the base loader instead of the base loader doing nothing.

Yes.

 

* a basic, declarative config. It is important that this config is declarative (loader.config({})) instead of imperative (i.e., System.paths.foo = ‘’) to make it easier for build tools to find those calls and process them. It also allows that config to come from declarative file formats like JSON. The basic config in AMD loaders is a great place to start since it has had real world use.

Consistency is more important.

 

* treat inlined modules as a core use case, not something just for the browser. Just as async loading of module bodies is treated as a core use case.

* The loader plugin thing.

This item #6 is probably the biggest fundamental issue that has colored my feedback and its reception. So be aware of any base line premises or design goals that are driving your feedback. If they do not match with who is designing ES modules, it will make it hard to communicate effectively.

....

Anyway, if you want to give feedback to them, maybe the above will enable you to be more successful.

I propose to make traceur's loader fully functional with a base class Loader per spec and extensions to solve real problems with real use case issues. Guy Bedford is deep into the latter with his work on ESML and systemjs.  Help us.  Starting from their results -- which have a lot of really nice aspects -- and arguing for improvements makes sense as a strategy to me.

jjb

[Karolis Narkevicius]

> I'd like 'default' to die, but it seems that it was added for the benefit of AMD/CJS ;-)

This is not funny :( If `default` has been added for the benefit of AMD/CJS and now AMD/CJS people are having trouble dealing with this feature (e.g this thread, or https://github.com/square/es6-module-transpiler/issues/85) - something is very wrong. Seems like misdesign or miscommunication that lead to misdesign?

Karolis


On Tuesday, 4 March 2014 06:03:37 UTC, James Burke wrote:

An important correction: I received a private note indicating that the ES designers did not ignore the feedback, they just could not respond to it due to length and mixing of concerns. I got the impression from a different response that a good chunk of it was just ignored. 

I apologize for the mischaracterization. They are all trying hard and doing research into existing systems. It has felt opaque and hard to understand the roots of some decisions. As an implementer I like more details, and I felt there was not a full understanding about existing module uses. The ES designers see it differently, they feel like they have put in significant work to understand.

Some notes inline:

On Mon, Mar 3, 2014 at 5:56 PM, Guy Bedford <guybe...@gmail.com> wrote:

For module inlining, myself and John Barton have been working on a `System.register` inlining method, which would work analogously to the named defines in AMD, while remaining CSP compatible and not dealing with string sources.

Maybe that will work for AMD/CJS code upconverted to work inside an ES module loader, but that will not work for ES modules because of the way that mutable slots via `import` work: they are very difficult to emulate in existing code. I thought a little bit about it in a throw-away experiment I did (you can ignore most of the rest of it):

https://github.com/jrburke/module#mutable-slots

but really if language keywords are used for module syntax, inlining needs something similar. I did some sketching of that here, but note that is just a sketch to talk about the issues involved, nothing real:

https://github.com/jrburke/module/blob/master/docs/existing/inlining.md

 

Technical details aside, it fits quite neatly into the existing loader pipeline allowing for bundles (https://github.com/systemjs/systemjs/blob/master/lib/system-bundles.js#L11).

There are currently two key issues I have with the spec at the moment which are:

1. Circular references - providing support for CommonJS-style circular reference handling

The mutable slots from `import` is to allow the module to have a working reference to an export even though it may not actually have the exported value yet. The CJS style circular referencing should not be needed, and I do not believe cannot be supported natively in an ES loader for ES modules.

This is one of the big things the CJS/node approach got incorrect: expecting a sync require() call to fetch and execute at the point of the call is not realistic to support in the browser. I do not think, or expect the ES system to go to incredible measures to make up for that choice, but leave it for node to use the hooks into the ES loader API to provide a require() that can load legacy node modules in that matter, probably using System.set() to jam in the legacy exports into the module loader (if they make them visible at all).

 

2. Deferred execution - currently ES6 modules defer their execution until they are loaded. But AMD and CommonJS modules in the ES6 loader don't. System.register would suffer from this - I really think they should get the same treatment.

Maybe I am unsure what you mean by defer execution, but AMD loader should definitely not execute the factory function until that module is part of a top level require([]) dependency chain. I believe it aligns well with the ES model. Maybe you mean that the instantiate hook needs more specification. CJS behavior in this case seems undefined, as they are always synchronously loaded and executed, they did not support a use case of being loaded but not executed.

James

[johnj...@chromium.org]

My complaint about 'default' has nothing to do with how well or poorly it helps AMD/CJS. My objection is entirely based on experience with ES6: this feature costs complexity for developers without giving them significant new power.  It forces you to read imports and exports more carefully when you should be paying attention to the things imported or exported.  Furthermore, it is one of the few parts of the module syntax which could be cleanly lopped off without having the rest of the story fall apart.

That said, I don't believe this feature is a high priority one way or another. We should focus on the missing features of the Loader. We can have real impact by defining, implementing, and using these features. 

jjb

[Guy Bedford]

On 4 March 2014 01:25, Karolis Narkevicius <karo...@gmail.com> wrote:

> I'd like 'default' to die, but it seems that it was added for the benefit of AMD/CJS ;-)

This is not funny :( If `default` has been added for the benefit of AMD/CJS and now AMD/CJS people are having trouble dealing with this feature (e.g this thread, or https://github.com/square/es6-module-transpiler/issues/85) - something is very wrong. Seems like misdesign or miscommunication that lead to misdesign?

This thread was initially started to share the transpile output that solves #85. We have now integrated this fix into the Traceur AMD compilation output, so that interop DOES work.

To summarise, AMD modules are imported by the default property:

import $ from 'jquery';

The above ES6 can be transpiled into AMD and work correctly for loading jQuery when jQuery is an AMD module, both in ES6 loaders like SystemJS and in existing AMD loaders today.

Sharing the interoperability solution was my primary goal with this thread. It seems I failed to adequately communicate that!

 

--
You received this message because you are subscribed to the Google Groups "amd-implement" group.
To unsubscribe from this group and stop receiving emails from it, send an email to amd-implemen...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[johnj...@chromium.org]

On Tuesday, March 4, 2014 9:50:35 AM UTC-8, Guy Bedford wrote:

On 4 March 2014 01:25, Karolis Narkevicius <karo...@gmail.com> wrote:

> I'd like 'default' to die, but it seems that it was added for the benefit of AMD/CJS ;-)

This is not funny :( If `default` has been added for the benefit of AMD/CJS and now AMD/CJS people are having trouble dealing with this feature (e.g this thread, or https://github.com/square/es6-module-transpiler/issues/85) - something is very wrong. Seems like misdesign or miscommunication that lead to misdesign?

This thread was initially started to share the transpile output that solves #85. We have now integrated this fix into the Traceur AMD compilation output, so that interop DOES work.

To summarise, AMD modules are imported by the default property:

import $ from 'jquery';

The above ES6 can be transpiled into AMD and work correctly for loading jQuery when jQuery is an AMD module, both in ES6 loaders like SystemJS and in existing AMD loaders today.

Sharing the interoperability solution was my primary goal with this thread. It seems I failed to adequately communicate that!

Guy, can you outline the role of 'default' export in your implementation and the consequences if the feature did not exist?  You've established that the feature is sufficient, I'm just curious about what makes it necessary.

jjb

[Guy Bedford]

The only thing `default` does is act as a sugar. Since AMD (and CommonJS) modules can export any type of value, we need to cater for this. If we want to load an AMD module from inside an ES6 module, we have to then write the full module to some export name. Lets say we called it "amdmodule":

export var amdmodule = amdFactory.apply(global, deps);

Now whenever we import an AMD module from within an ES6 module, we have to use this name:

import { amdmodule as $ } from 'jquery';

The default sugar just "tells us" that this export name is `default` (it is just another export like the others), and then gives us a sugar syntax for importing from it:

import $ from 'jquery';

It is simply there as a convenience for loading existing AMD and CommonJS modules (and also for loading ES6 modules that want to have a single export of course).

 

jjb

[Guy Bedford]

On 3 March 2014 22:03, James Burke <jrb...@gmail.com> wrote:

An important correction: I received a private note indicating that the ES designers did not ignore the feedback, they just could not respond to it due to length and mixing of concerns. I got the impression from a different response that a good chunk of it was just ignored. 

I apologize for the mischaracterization. They are all trying hard and doing research into existing systems. It has felt opaque and hard to understand the roots of some decisions. As an implementer I like more details, and I felt there was not a full understanding about existing module uses. The ES designers see it differently, they feel like they have put in significant work to understand.

Thanks for clarifying and sharing your thoughts on this. Let me know if any of my responses are unclear. If another discussion format or location would work better here, I'm happy to discuss further.

 

Some notes inline:

On Mon, Mar 3, 2014 at 5:56 PM, Guy Bedford <guybe...@gmail.com> wrote:

For module inlining, myself and John Barton have been working on a `System.register` inlining method, which would work analogously to the named defines in AMD, while remaining CSP compatible and not dealing with string sources.

Maybe that will work for AMD/CJS code upconverted to work inside an ES module loader, but that will not work for ES modules because of the way that mutable slots via `import` work: they are very difficult to emulate in existing code. I thought a little bit about it in a throw-away experiment I did (you can ignore most of the rest of it):

Firstly, note that no conversion needs to be done to make AMD and CommonJS work in an ES module loader - compatibility layers can make existing code work in the ES6 module loader without any adjustments.

Adequately making mutable slots work is also a limitation of polyfill approaches. Until we have fully transitioned into ES6, I would suggest educating users not to assume this functionality.

But you are exactly right - the System.register inlining method is only adequate during the polyfill phase as the limitation is we lose circular dependency support.

The hope is that by that time (when ES6 is supported in the majority of browsers), a suitable bundling system will be available.

Personally I would really like to see the `module 'name' { ... }` tag brought back, it is strongly needed to enable bundling when ES6 has got majority browser adoption. It's certainly on the list of things to advocate!

 

https://github.com/jrburke/module#mutable-slots

but really if language keywords are used for module syntax, inlining needs something similar. I did some sketching of that here, but note that is just a sketch to talk about the issues involved, nothing real:

https://github.com/jrburke/module/blob/master/docs/existing/inlining.md

 

Technical details aside, it fits quite neatly into the existing loader pipeline allowing for bundles (https://github.com/systemjs/systemjs/blob/master/lib/system-bundles.js#L11).

There are currently two key issues I have with the spec at the moment which are:

1. Circular references - providing support for CommonJS-style circular reference handling

The mutable slots from `import` is to allow the module to have a working reference to an export even though it may not actually have the exported value yet. The CJS style circular referencing should not be needed, and I do not believe cannot be supported natively in an ES loader for ES modules.

This is one of the big things the CJS/node approach got incorrect: expecting a sync require() call to fetch and execute at the point of the call is not realistic to support in the browser. I do not think, or expect the ES system to go to incredible measures to make up for that choice, but leave it for node to use the hooks into the ES loader API to provide a require() that can load legacy node modules in that matter, probably using System.set() to jam in the legacy exports into the module loader (if they make them visible at all).

My point is we can get all existing code in AMD and CommonJS to work in the ES6 loader without any changes necessary, which makes adoption really nice.

But as soon as we hit circular dependencies, it all breaks down.

 

 

2. Deferred execution - currently ES6 modules defer their execution until they are loaded. But AMD and CommonJS modules in the ES6 loader don't. System.register would suffer from this - I really think they should get the same treatment.

Maybe I am unsure what you mean by defer execution, but AMD loader should definitely not execute the factory function until that module is part of a top level require([]) dependency chain. I believe it aligns well with the ES model. Maybe you mean that the instantiate hook needs more specification. CJS behavior in this case seems undefined, as they are always synchronously loaded and executed, they did not support a use case of being loaded but not executed.

The ES6 loader will parse the syntax tree and link the bindings, but it will only execute a module when it is actually required. There is a complete separation between linking and execution. For example:

<script type="module" name="my-module">

  export var some = 'export';

  console.log('executed');

</script>

The above will only execute when I do `System.import('my-module')`.

But, if instead of ES6, I had written AMD in the above:

<script type="module" name="my-module">

  define(function() {

    console.log('executed');

    return { some: 'export' };

  });

</script>

The execution will happen as soon as that tag is run (via `System.define`).

This difference in execution between native ES6 and other module types seems entirely unnecessary to me, but it is quite a subtle point I understand.

 

James

[James Burke]

On Tue, Mar 4, 2014 at 4:31 PM, Guy Bedford <guybe...@gmail.com> wrote:

But, if instead of ES6, I had written AMD in the above:

<script type="module" name="my-module">

  define(function() {

    console.log('executed');

    return { some: 'export' };

  });

</script>

The execution will happen as soon as that tag is run (via `System.define`).

This difference in execution between native ES6 and other module types seems entirely unnecessary to me, but it is quite a subtle point I understand.

Oh, I suppose I do not understand how that define() gets translated to System.define() call. But in regular AMD code:

<script>

define(‘my-module’, function() {

  // This function not run until my-module is part

  // of a require chain

});

// the factory function run as part of handling

// this call

require([‘my-module’]);

</script>

James

[Guy Bedford]

System.define has the following form:

System.define('module-name', "... string source (ES6 or not) ");

This is the bundling method that gives us CSP issues, and also the bundling method behind the `<script type="module">` tag which I've used in the example.

An unnamed module tag is the same as System.module, which does execute.

So in this way we get the distinction between named defines / bundles and immediate execution.

In the bundle case we're talking about here, the factory function would be executed even when just populating the bundle, for code that is AMD or CommonJS or global. That is exactly my complaint with having different deferred execution.

[Ben Hockey]

On Tuesday, March 4, 2014 6:31:05 PM UTC-6, Guy Bedford wrote:

But, if instead of ES6, I had written AMD in the above:

<script type="module" name="my-module">

  define(function() {

    console.log('executed');

    return { some: 'export' };

  });

</script>

The execution will happen as soon as that tag is run (via `System.define`).

This difference in execution between native ES6 and other module types seems entirely unnecessary to me, but it is quite a subtle point I understand.

to me, this looks to be desirable.  `define` is executed immediately and then the execution of the factory is deferred until something else depends on this module.  if `define` is not executed immediately, how/when do you know what dependencies need to be fulfilled before the module's factory can be executed?

also, with circular dependencies, where is the problem?  if AMD has this solved today (which it has solved via the "exports" dependency) and a compatibility layer can load AMD via an ES6 module loader then what is missing?  can't the AMD compatibility layer manage the circular dependencies just as AMD loaders do today?  i would assume that if the compatibility layer is controlling the execution of the factory (NOT the module i.e. the call to define, but the factory provided in the call to define) then it should have everything necessary to handle circular dependencies - it can defer the execution of the factory and provide the objects passed to the factory.  isn't that all that's necessary to continue to handle circular dependencies in the same way they are handled today?

thanks,

ben...

[Guy Bedford]

Thanks Ben for the feedback, responses below.

On 5 March 2014 06:46, Ben Hockey <neonst...@gmail.com> wrote:

On Tuesday, March 4, 2014 6:31:05 PM UTC-6, Guy Bedford wrote:

But, if instead of ES6, I had written AMD in the above:

<script type="module" name="my-module">

  define(function() {

    console.log('executed');

    return { some: 'export' };

  });

</script>

The execution will happen as soon as that tag is run (via `System.define`).

This difference in execution between native ES6 and other module types seems entirely unnecessary to me, but it is quite a subtle point I understand.

to me, this looks to be desirable.  `define` is executed immediately and then the execution of the factory is deferred until something else depends on this module.  if `define` is not executed immediately, how/when do you know what dependencies need to be fulfilled before the module's factory can be executed?

Yes, define does need to be called to get the dependencies. Unfortunately the point here is that even the factory function gets called. When modules have been fetched, they are linked, and then only executed when required. But AMD and CommonJS module are executed during linking instead of afterwards, which is my complaint. The module pipeline expects the modules to define a complete Module instance at linking, so the compatibility layer is forced to execute too soon. Since the whole process is controlled by the spec, there is no way around this without a spec change.

This makes it impossible to have AMD bundles that only execute the factory function when required, leading to a performance loss on startup, as well as the fact that there is this forking in behaviour between native modules and other modules.

 

also, with circular dependencies, where is the problem?  if AMD has this solved today (which it has solved via the "exports" dependency) and a compatibility layer can load AMD via an ES6 module loader then what is missing?  can't the AMD compatibility layer manage the circular dependencies just as AMD loaders do today?  i would assume that if the compatibility layer is controlling the execution of the factory (NOT the module i.e. the call to define, but the factory provided in the call to define) then it should have everything necessary to handle circular dependencies - it can defer the execution of the factory and provide the objects passed to the factory.  isn't that all that's necessary to continue to handle circular dependencies in the same way they are handled today?

Again, this is set by the loading pipeline. The loading pipeline goes to each AMD and CommonJS module, one by one, and asks for its full Module object to be returned. Note also that Module objects are immutable. So by definition that process can't handle any circular dependencies at all, since there is no way for a reference to be created to a module object, without that module object already being finalised.

As above, this can't be dealt with by the compatibility layer since the above logic is baked into the spec itself.

I summarised my views on circular references here - https://github.com/jorendorff/js-loaders/issues/102#issuecomment-36415182, including a proposal for allowing module shells to enable underlying mutable module objects.

 

thanks,

ben...

[Ben Hockey]

i think i understand a little better now.  so, if ES6 would allow for a compatibility layer to execute an AMD module (define is called) and then at a later time have the compatibility layer provide the module's value (the factory is executed) then circular dependencies and bundling could work.  is that right?

[Guy Bedford]

ES6 does allow for a compatibility layer to execute an AMD module, but the compatibility layer gets the AMD module to execute its factory immediately at the same time it checks its dependencies.

Basically AMD support looks more or less like this in the instantiate hook:

instantiate: function(load) {

  var deps, factory;

  global.define = function(_deps, _factory) {

    deps = _deps;

    factory = _factory;

  }

  eval(load.source);

  return {

    deps: deps,

    execute: factory

  }

}

The instantiate function then runs the execute function for a module, after running the execute for each of its dependencies.

This same restriction stops circular dependencies from working, since each module is expected to immediately return a complete module object.

Bundling is a separate issue though, and we've managed to find a bundling solution that works with the loader hooks. Happy to share this info as well.

On 5 March 2014 14:33, Ben Hockey <neonst...@gmail.com> wrote:

i think i understand a little better now.  so, if ES6 would allow for a compatibility layer to execute an AMD module (define is called) and then at a later time have the compatibility layer provide the module's value (the factory is executed) then circular dependencies and bundling could work.  is that right?

--

[John Hann]

Whoa. This thread is out of control.  I want to respond, but it would be an utter mess.  Way too many topics.  It's time to start some new threads, folks.  (I admit to be the first one to take this thread off-topic! :/ )

I have to say that I appreciate the work you guys, Guy and John B, are doing.  Unfortunately, I completely disagree with your proposed transpilation of ES6 to AMD.  I believe I have a much simpler solution.  Let's start a new thread on this topic?  I promise to try not to derail it. :)

James: your efforts to steer the TC39 guys is also awesomely appreciated.  Thanks for the advice, above.  I've been preparing some notes, code samples, etc. over the past two weeks and plan to launch a salvo of proposals to both this group and to the js-loaders github repo to address the serious problems with the current Loader spec.  I wish we only needed to tweak the spec, but it's deeply flawed.  :'(

Regards,

-- John

[John Hann]

Hey James, 

I just went back and read your points.  We're in agreement on 95% of your statements about the problems with the Loader spec. (yay)  I think we can make named+default exports work, though.  After attempting to build an extensible ES6 style loader, I have a few other problems to add to your list.  

Unfortunately, the current Loader spec needs to be thrown out, and a new one needs to be started.  I have no idea how to approach this without alienating (aka "seriously pissing off") the TC39 folks.  My current plan is to use hard evidence to show them that there are flaws (hopefully gaining some credibility while I do that) and then to find ways to influence them in the right direction.  I've got some ideas about which flaws to present first, but it's going to be hard to get them to throw out the work they've done so far.

If anybody else has a great plan to start from scratch, I'm all ears.  

Anyway, keep up the great work.  All of you.  I'm proud of this group.

Regards,

-- John

[James Burke]

On Thu, Mar 6, 2014 at 3:20 AM, John Hann <jo...@unscriptable.com> wrote:

Unfortunately, the current Loader spec needs to be thrown out, and a new one needs to be started.  I have no idea how to approach this without alienating (aka "seriously pissing off") the TC39 folks.  My current plan is to use hard evidence to show them that there are flaws (hopefully gaining some credibility while I do that) and then to find ways to influence them in the right direction.  I've got some ideas about which flaws to present first, but it's going to be hard to get them to throw out the work they've done so far.

If anybody else has a great plan to start from scratch, I'm all ears.  

Starting with “current spec needs to be thrown out” will likely not get you very far. Best to get agreement on problems first, make sure they weigh the use cases the same as you. 

Also best to first start privately, they prefer a layer of the onion design approach — starting the design discussion small and gradually expanding it out to more people. It is near impossible to do this kind of design work with big groups. And see suggestions from my earlier message.

They are good people wanting to do good, and they are smart, have studied and made serious attempts at listening. Just difficult given the scope of the design space, and how we weigh use cases differently. The more code-based data points you have for identifying problem areas and use cases, the better.  It is more likely that to make progress it will mean iterating on what they have already.

I am super burned out on this though, and for my own sanity expect to disengage from discourse about modules and loaders for a while and just focus on my own code and work priorities to recharge.

James

[John Hann]

On Thu, Mar 6, 2014 at 2:16 PM, James Burke <jrb...@gmail.com> wrote:

I am super burned out on this though, and for my own sanity expect to disengage from discourse about modules and loaders for a while and just focus on my own code and work priorities to recharge.

I hear you James.  I've been mentally preparing myself.  Thanks again for the advice. If I make any progress, I'll let you know.  

If anybody would like to join me in my endeavor, please reach out to me on any channel.  I could use the help. :)

-- John

[Guy Bedford]

We've been playing around with ES6 module loading, compiling ES6 modules into AMD and then loading that AMD in an ES6 loader.

This workflow allows ES6 modules to be used in production, with AMD doing the transport.

The thing is, there needs to be some meta information for an ES6 loader to know that the AMD module is in fact an ES6 module that has been transpiled.

I've suggested an output like the following:

define([], function() {

}, 'transpiled-module')

An ES6 AMD compatibility layer can then pick up on the above meta information and know to treat the module as an ES6 module directly and not as an AMD module. This is needed due to the incompatibility of export structures.

My worry is whether this would break existing AMD loaders if the above output was used to run ES6 modules through RequireJS and other AMD loaders (this being the other, and for now, the more important use case).

It would be great to hear from implementors if the above output would cause issues in loaders.

Thanks!

[John Hann]

This would break in curl.js. :(

[Guy Bedford]

Thanks John, that was my worry.

I think compatibility in this case is more important.

Will consider some other options.

[Taka Kojima]

What about:

define([], function() {

}).meta({transpiled : true});

As far as I'm aware nothing in the AMD-spec relies on the return value from define, so this should work ok.

[James Burke]

Guy,

Do you know what is supposed to happen when System.import() is used, when:

1) the module just has a default export
2) has named exports

So, for 1), assume a default exports a function:

System.import('a').then(function(a){
//a() or a.default() or something else?
})

If a.default() that looks undesirable, and I would prefer if ES6
modules only allowed a default or named exports, in which case a()
becomes a clear possibility. It would also bridge the discrepancies
with existing loaders. Maybe they can get a() to work regardless.

I believe Erik Arvidsson filed a bug about only allowing one or the
other, but do not know if that got any resolution. So if you have
further info in this area, feel free to share.

James

[Guy Bedford]

Yes this default access is undesirable, so we change it.

We can override the default behaviour of System.import, which is what I do in SystemJS here - https://github.com/systemjs/systemjs/blob/master/lib/system-core.js.

Then, when there is only a default property, we get the default property only. This then allows AMD and CommonJS modules to work nicely with ES6.

The issue I've had above is exactly because of this handling needing a sub-exception for compiled ES6!

[Guy Bedford]

If there are worries about the mutability of System, note that System is entirely designed to be mutable to customize the loading - that is how one changes System.normalize etc.

[James Burke]

On Tue, Feb 18, 2014 at 2:22 PM, Guy Bedford <guybe...@gmail.com> wrote:
> Yes this default access is undesirable, so we change it.
>
> We can override the default behaviour of System.import, which is what I do
> in SystemJS here -
> https://github.com/systemjs/systemjs/blob/master/lib/system-core.js.
>
> Then, when there is only a default property, we get the default property
> only. This then allows AMD and CommonJS modules to work nicely with ES6.
>
> The issue I've had above is exactly because of this handling needing a
> sub-exception for compiled ES6!

Right, and I prefer to see the way to fix this by asking the spec to
be different. Because the System.import style is sub-par compared to
the import syntax, even when just using pure ES6 modules. I can see
where it may be claimed that the import syntax is just sugar on the
default assignment, but the import vs System.import API pairing is
uneven, and ideally fixed.

James

[Guy Bedford]

I don't think the import asymmetry is quite as bad, since we simply write:

import $ from 'jquery';

when loading an AMD module.

So it's really just the System.import function which benefits from the sugar.

Perhaps the solution is as simple as the following: When calling System.import, and getting a module with only one export called "default", make the returned value that property instead of the module object.

But I think this can be solved in user code just as easily, and getting an idea like the above into the spec would be hard!

[James Burke]

On Tue, Feb 18, 2014 at 2:47 PM, Guy Bedford <guybe...@gmail.com> wrote:
> I don't think the import asymmetry is quite as bad, since we simply write:
>
> import $ from 'jquery';
>
> when loading an AMD module.
>
> So it's really just the System.import function which benefits from the
> sugar.

That, and using ES6 modules transpiled to AMD, which I think is the
issue that started this thread. Particularly if a module was only
allowed to either export default or do named exports: it would be
clear what to do.

And I think it is an entirely reasonable thing for a transpiler to
enforce that rule: only allow transpiling from ES6 to AMD for a module
that either uses export default or uses named exports, but not both.

On a practical level, I do not see doing both in the same module as
useful. For web modules, I expect export default will be used most of
the time, to get finer grained modules that work well for
optimizations and cutting out code that is not needed. And node folks
prefer export default too.

That, or they just attach named properties to the exports object, same
for AMD: either default export or named exports but not both.

> But I think this can be solved in user code just as easily, and getting an
> idea like the above into the spec would be hard!

The spec should not be considered done. It does not have an actual
runnable implementation, and these are the sorts of issues that should
be expected when trying to plug it all together.

James

[Guy Bedford]

Ok, I've come up with a new solution along these lines:

define([], function(a, b) {

  if (a && a.__defaultOnly)

    a = a['default'];

  if (b && b.__defaultOnly)

    b = b['default'];

  return {

    export: 'value',

    __transpiledModule: true

  };

});

The extra meta checks above don't affect backwards compatibility for using ES6 in AMD.

Then an AMD compatibility layer can read when a module is a "__transpiledModule", and store the module as the full module object in the Loader registry.

When importing, the first two lines allow ES6 modules to be used directly, while AMD modules, stored as:

Module({

  __defaultOnly: true,

  default: {...AMD Module...}

});

Will have the AMD module plucked out.

We need two separate metas because we have this "jump up" and "jump down" using the default property as the carrier.

I really think this is a good flexible option for ensuring bidirectional compatibility. We get to use defaults and named exports together, there are no edge cases.

James

[John Hann]

> The spec should not be considered done. It does not have an actual
runnable implementation, and these are the sorts of issues that should
be expected when trying to plug it all together.

I've been trying to use the spec *as is* and it's been very frustrating.  It's not even close to being usable for anything but toy apps, imho.  Maybe we should start a discussion about how we could help steer the TC39 team?  Does anybody wish to debate the ES6 Loader spec here?  Or should we do it somewhere else?

James

[Guy Bedford]

I just wanted to do a check to see if there are any red flags with the previous output I described?

Basically, an ES6 module looking like:

```

import { a } from 'a';

export var p = 5;

```

would be transpiled to AMD:

```

define(['a'], function($_0) {

  if ($_0 && $_0.__defaultOnly) $_0 = $_0['default'];

  var a = $_0['a'];

  return {

    p: 5,

    __transpiledModule: true

  };

});

```

and then populated in the module registry (thanks to the __transpiledModule flag) as:

Module({

  __defaultOnly: true,

  default: {

    p: 5,

    __transpiledModule: true

  }

});

The callbacks to the execute function are then the module objects as above.

• If the dependency "a" here was an AMD module itself, we would try to read its "a" property with the above `import { a } from 'a'`.
• If we had instead used `module a from 'a'`, we would get the full AMD module.
• If we had instead used `import A from 'a'`, we would have got a.default (which doesn't make sense).
• If the dependency "a" here was an ES6 module, then our import statement transpiles exactly as needed for ES6.

Does the above sound sensible?

[Guy Bedford]

My sincere apologies for wasting time detailing this logic, but I need to make a correction. The system above is very close to what is needed, but there is a slight alteration required in the output.

Firstly, the AMD compatibility layer should already only return direct AMD objects (the default property when __defaultOnly is present), so this ISNT needed in the transformation.

But ES6 modules transpiled into AMD then need to undo this transformation rather with the following output:

define(['a'], function($_0) {

  if (!($_0 instanceof Module)) $_0 = { default: $_0 };

  var a = $_0['a'];

  return {

    p: 5,

    __transpiledModule: true

  };

});

I mainly just wanted to ensure that this was being fully communicated at the AMD level, because this output needs to be standardised. The above is then the correct structure of the solution.