Multiversion/scoped API
James Burke
https://groups.google.com/group/amd-implement/browse_thread/thread/44429707a29b7f4d
# Question
How should an AMD loader work if module 'a' and module 'b' both want
to use module 'c', but different versions of 'c'?
# Possible solution
## Configuration
It seems common for AMD loaders to support a paths: {} config that
allows mapping a module ID to a path segment. Perhaps we should
standardize some of these config values.
What could work is then extending paths to have its own nested config, so:
{
paths: {
'a': {
//This value is the normal path value used if needing
//to map a somewhere else normally (the usual, string-only
//value for a paths config):
path: 'some/other/path/to/a',
//baseUrl here would default to a's location,
//but a different one could be set, which would
//relative to a's location.
baseUrl: './node/modules',
//a list of modules that are scoped to a's location.
scope: ['c', 'd'],
//any special path mappings inside a. Not
//required, particularly if listing in "scope" is enough.
//However, could have nested config. If a module
//is listed here, it is considered "scoped" so it does
//not have to be in the scope: config above.
paths: {
'c': {
scope: ['f']
}
}
}
}
}
## Attaching scope to the loaded script
This would work similar to how the module ID is attached to an
anonymous module -- add an attribute on the script tag that mentions
the scope the script tag load -- any define calls in there are are
scoped to that value.
## Combined modules
For modules combined in a file, we would need a way to include both
'c' modules in a file. What about something like:
define.scope('a', function (define) {
//'c' defined in this case would only be visible/used by 'a'.
define('c', function () {});
//Can have nested values, the define passed in to this function
//already has a scope of 'a', so this next one would have
//an absolute scope of 'a/c':
define.scope('c', function (define) {
define('g', ['h'], function (h) {
//h is found by walking up the scopes, unless 'c' had
//an explicit scope setting for it.
});
});
});
define.scope('b', function (define) {
define('c', function () {});
});
## Loader plugins
We probably need to figure out what that means for the loader plugin
API. I would hope it would be fairly transparent though, since each
scope would have its own version of the plugin loaded, and so its
internal loading/tracking, particularly for the build time APIs would
automatically be scoped. But it probably needs a bit more thought,
maybe just trying it in code.
## Summary
What I like about this approach is that it is module ID-centric, not
path centric -- for built files all that is serialized are module IDs
and nothing related to paths, and the output could be nested into
other scopes.
This is just a quick sketch to start the conversation. It is not fully
baked. In particular, I'm not sure 'scope' is the right word, but
feels roughly right and it is short.
Feel free to propose alternatives/problems.
James
Ryan Fitzer
> I'm not sure 'scope' is the right word, but
> feels roughly right and it is short.
In your example, 'c' would be scoped to 'a'. Would that include any
local vars declared by 'a'?
For example, if 'a' declared `var someVar = true` and 'c' referenced
`someVar`, could one expect 'c' to use the value 'a' defined?
I see how it could work in the combined file, but what about
dynamically loaded files. They would need to evaluate (eval(), with(),
etc.) the scoped modules, if I'm understanding you correctly.
Regarding the config syntax, I think that using two different props,
`scope` for short notation, `path` for the long form, would be
confusing. I realize one could use either/or, but it sounds like both
could be used at the same time. As you mentioned, "scope" may not be
the right term. If I understand you proposal correctly, "local" would
seem a better choice. He's my example:
{
paths: {
'a': {
// use short notation
local: {
modules: ['c', 'd']
},
// or long notation
local: {
'c': {
local: {
modules: ['f']
}
}
},
// which can keep on going
local: {
'c': {
local: {
'f': {
local: {
modules: ['j']
}
}
}
}
}
}
}
}
Expressing the relationship between 'f' and 'a' via
`a.local.c.local.f` makes more readable sense to me: "f is local to c,
which is local to a". This pattern also consolidates the API under one
prop and preserves the short and long forms.
Since I haven't built an AMD loader, take my feedback from the point
of view of a user of loaders :)
Ryan
James Burke
> This one is quite the brain twister. It took me a bit to wrap my head around it.
>
>> I'm not sure 'scope' is the right word, but
>> feels roughly right and it is short.
>
> In your example, 'c' would be scoped to 'a'. Would that include any
> local vars declared by 'a'?
No, c cannot see any local vars produced inside a's factory function.
I think this confusion you bring up is one reasons why "scope" is
probably the wrong word for it.
I like 'local' now better than 'scope'.
This would mean 'modules' could not be a module name, since local.c
and local.modules are allowed above.
Exploring the main design goal a bit more:
Some modules are just local and can use the default file
conventions/no additional config. For those modules, we just need to
list them. Arrays are nice for that.
However, there are some modules that may have their own local config,
in which an object is nice for that.
'local' could just be an array of items, and if the item had more
config, it is an object instead of a string:
local: [
'c',
'd',
{'e': {
local: []
}}
]
I suppose that is not too bad, but the extra {} for each entry that
has further dependencies is kind of ugly. But maybe the hope is to
only have fairly shallow hierarchies, so it may not be that bad. I can
live with it.
In any case, I prefer your suggestion 'local' over 'scope' now, and
that would change the define API to define.local().
James
Ryan Fitzer
Well yeah, when you write it like that. A little whitespace goes a long way :)
I agree that it's not as visually succinct, but describing and
understanding the API will be much simpler.
Ryan
Ben Hockey
John Hann
Ben Hockey
Ben Hockey
Ben Hockey
James Burke
> dojo has a packageMap which is part of a package configuration. it
> basically says that for this package, anywhere you see this module id, use
> this other one instead. i know that james would like to move away from
> packages but this is probably the best argument i see for keeping them -
> they act as a sandbox for module ids. a package can reach out and reference
> another package without being overly concerned about what id is used to
> reference that package. this is along the lines of what node_modules
> provides. packageMap allows the package consumer to map any "global" id
> into the package's "local" space.
The thing I do not like about packages is the runtime "main"
translation. I think it is wrong to have to always require
configuration to find the main module, this should just be a
convention thing. One way would be to just have dependencies do a
require('packageName/main'). Or have a package manager elevate
packageName/main.js to packageName.js. Otherwise it means there will
be large config blocks because every package needs an entry.
I like packages and a package.json for specifying dependencies and
versions of dependencies that need to be processed at install time,
but once it gets to run time, a package.json-equivalent should not be
mandatory just to start loading a module in that package. That is
favoring configuration over convention.
And there is just a tension between paths and packages.
> var require = {
> packages: [
> {
> name: 'a',
> packageMap: {
> c: 'c1-7'
> }
> },
So I might be able to live with the Dojo pattern, but need to talk
some things out:
1) I would like to see this as a generic 'map': config, and have it
applicable both at the "top level" of the config, and inside paths:
config -- in other words allow this mapping for things that do not
need the "main" lookup, but follow directly from a paths config. So,
if I have
paths: {
'a/b': {
map: {
'c': 'c-1'
}
}
}
it would mean for module 'a/b' or 'a/b/e' their 'c' is really 'c-1'.
2) It seems to me you also need a smart tool to rename any named
modules. So if 'c' did do a define('c', function (){}); placing it at
c-1 means a rename sweep. Not too bad, but it is extra work for a
translation tool. Patrick in the other thread was not too keen on
having to touch the module internals, just wanted to do a define()
wrapping at most. It could also probably figure out the config, but
that just needs to process package.json contents, not do AST or regexp
surgery.
3) How to translate node_modules code to this pattern? It sounds like
the tool that bundles up the code would need to make up IDs to make
this happen. Hmm. So this could be a problem. I'm not sure how
reliable it is to depend on a package.json for naming.
My first pass at an algorithm:
* For each node_modules package, get its name and version.
* For each of the dependencies in package.json, figure out their name
and version and where it was installed.
* Create a top-level module ID for each nested dependency. If there is
only one 'foo' then this is easy. If there are two 'foos' then name
the top level 'foo-version'. Care must be taken to confirm the source
of the module pull -- did the dependencies: {} in the package.json
indicate just a version number, so it is assumed a package registry
has resolved the name, or is it an URL. If an URL, use that instead of
version number in the top level id, 'foo-url'.
So I think this breaks down because it depends on knowing what module
was fetched from which package repository. For instance, if 'a' and
'b' both get 'c' version 1 but they got c from two different
repositories, there could be a problem disambiguating them.
Anyone have thoughts on how this might work? The choice of package
repo is configured in npm config IIRC, so there are no artifacts on
disk to indicate the origin? And we cannot always assume npm is in
play.
This is not a problem with the 'local' proposal, because it always
allows file-specific mappings. It could use a system like the Dojo one
for normal use, to avoid sending duplicate copies of the same file,
but for cases when it is unclear go to full local mode.
I do not think this would be a problem for a "smart" package manager
that knows about this kind of AMD config. I expect it would set up the
config for the user on package install. However, the case Patrick
brings up in the other thread is about code that was managed by some
other package manager, or just files on disk that maybe inherits some
node_modules from somewhere.
How big of an issue is this?
In summary, I like the dojo approach because it allows for de-duping,
but it requires a lot more knowledge to get 100% correct for deeply
nested node_modules trees, knowledge that I am not sure are available.
Maybe we need a combo approach -- pick up Dojo's packageMap, but just
use it as 'map', and allow at top level of config and in paths as well
as packages, but use local approach for the extreme cases?
The possible downside with that is the define.local() API
introduction, for cases that may not need it?
Ben Hockey
On Friday, December 9, 2011 5:02:42 PM UTC-5, James Burke wrote:
The thing I do not like about packages is the runtime "main"
translation. I think it is wrong to have to always require
configuration to find the main module, this should just be a
convention thing. One way would be to just have dependencies do a
require('packageName/main'). Or have a package manager elevate
packageName/main.js to packageName.js. Otherwise it means there will
be large config blocks because every package needs an entry.
I like packages and a package.json for specifying dependencies and
versions of dependencies that need to be processed at install time,
but once it gets to run time, a package.json-equivalent should not be
mandatory just to start loading a module in that package. That is
favoring configuration over convention.
And there is just a tension between paths and packages.
> var require = {
> packages: [
> {
> name: 'a',
> packageMap: {
> c: 'c1-7'
> }
> },
So I might be able to live with the Dojo pattern, but need to talk
some things out:1) I would like to see this as a generic 'map': config, and have it
applicable both at the "top level" of the config, and inside paths:
config -- in other words allow this mapping for things that do not
need the "main" lookup, but follow directly from a paths config. So,
if I havepaths: {
'a/b': {
map: {
'c': 'c-1'
}
}
}
it would mean for module 'a/b' or 'a/b/e' their 'c' is really 'c-1'.
2) It seems to me you also need a smart tool to rename any named
modules. So if 'c' did do a define('c', function (){}); placing it at
c-1 means a rename sweep. Not too bad, but it is extra work for a
translation tool. Patrick in the other thread was not too keen on
having to touch the module internals, just wanted to do a define()
wrapping at most. It could also probably figure out the config, but
that just needs to process package.json contents, not do AST or regexp
surgery.
3) How to translate node_modules code to this pattern? It sounds like
the tool that bundles up the code would need to make up IDs to make
this happen. Hmm. So this could be a problem. I'm not sure how
reliable it is to depend on a package.json for naming.My first pass at an algorithm:
* For each node_modules package, get its name and version.
* For each of the dependencies in package.json, figure out their name
and version and where it was installed.
* Create a top-level module ID for each nested dependency. If there is
only one 'foo' then this is easy. If there are two 'foos' then name
the top level 'foo-version'. Care must be taken to confirm the source
of the module pull -- did the dependencies: {} in the package.json
indicate just a version number, so it is assumed a package registry
has resolved the name, or is it an URL. If an URL, use that instead of
version number in the top level id, 'foo-url'.So I think this breaks down because it depends on knowing what module
was fetched from which package repository. For instance, if 'a' and
'b' both get 'c' version 1 but they got c from two different
repositories, there could be a problem disambiguating them.Anyone have thoughts on how this might work? The choice of package
repo is configured in npm config IIRC, so there are no artifacts on
disk to indicate the origin? And we cannot always assume npm is in
play.
This is not a problem with the 'local' proposal, because it always
allows file-specific mappings. It could use a system like the Dojo one
for normal use, to avoid sending duplicate copies of the same file,
but for cases when it is unclear go to full local mode.I do not think this would be a problem for a "smart" package manager
that knows about this kind of AMD config. I expect it would set up the
config for the user on package install. However, the case Patrick
brings up in the other thread is about code that was managed by some
other package manager, or just files on disk that maybe inherits some
node_modules from somewhere.How big of an issue is this?
In summary, I like the dojo approach because it allows for de-duping,
but it requires a lot more knowledge to get 100% correct for deeply
nested node_modules trees, knowledge that I am not sure are available.Maybe we need a combo approach -- pick up Dojo's packageMap, but just
use it as 'map', and allow at top level of config and in paths as well
as packages, but use local approach for the extreme cases?The possible downside with that is the define.local() API
introduction, for cases that may not need it?
James Burke
'packages', but my summary at the end.
On Fri, Dec 9, 2011 at 7:02 PM, Ben Hockey <neonst...@gmail.com> wrote:
>> And there is just a tension between paths and packages.
> [snip]
> i'm lost with this comment - what's the tension you're referring to?
I just meant with 'paths' the config is usually much simpler, just
setting a base location for modules that start with that module ID
prefix. It is still configuration though. Just seems lighter weight,
no need for a 'main' property because there is no flexibility.
If the "package", by which I mean "a directory of modules", is just
placed at baseUrl, then no config is needed, particularly if the code
that consumes that "package" asks for the "main" module via
require('packageName/main").
But ideally we could just install modules so that the
"packageName/main.js" file was installed as "packageName.js", and
installed in the baseUrl. Then no configuration is needed. That is
really what I desire -- getting away with *no* config for basic apps.
If when installing a package, and it has dependencies, if they are
unique dependencies, just place them at baseUrl with that
packageName.js + packageName/ layout and it all works with no config.
If there is a conflicting dependency, then do the dojo trick of
installing 'c' at 'c-17' in baseUrl and put in a config that says "for
this set of modules, when it asks for 'c', use 'c-17', like the
'packageMap' used by dojo.
My concern in my last message is that I do not think there is enough
info to know if two 'c' packages/set of modules are really the same or
different. There are clues in the package.json though, maybe it is
enough. If name, version match, and for node packages, if "homePage"
and "author" match up, then that is probably enough info to confirm a
match or if something is different.
>> it would mean for module 'a/b' or 'a/b/e' their 'c' is really 'c-1'.
>
> i wouldn't oppose this but it seems like configuring something like this per
> module is not going to reduce the amount of config compared with packages.
> ...and i don't think it makes sense to apply this to a/b/e. my reasoning
> is based on this - if we were to define the location of a/b via paths, it
> would not affect the location of a/b/e. the same should be true of anything
> else configured by the paths config. we've set a precedent that paths
> configuration applies to the single module specified in the config and that
> if you want a config to apply to a sub tree you use packages.
Hmm, the paths, at least in requirejs is for the full subtree. An
'a/b' paths config would apply to finding a/b/c. That was how it
worked in Dojo in the old days, unless I'm really losing it.
So for me, the only difference between 'packages' and 'paths' config
is that 'packages' has a reliance on a 'main' lookup. If that was
removed, paths would be enough, and we could add this new mapping
capability just under paths.
The other option would be to remove paths, and always use packages,
but that is really awkward for single JS file libraries that are
common on the web.
The basic tension is between traditional JS on the web and the idea of
a directory for a deliverable for a set of modules. And that it is not
clear how to best deliver packageName.js+packageName/ combo that would
work well for path resolutions. But I want that to work.
So it may just require more time to sort out. If we support both
'packages' and 'paths', then I would like to have the same
capabilities for 'paths' as far as dependency mappings, so I like just
having the property called "map" instead of "packageMap".
>> 2) It seems to me you also need a smart tool to rename any named
>> modules. [snip]
> this is why you should always use anonymous modules. a tool would then only
> need to process package.json - this could be the same tool that initially
> fetches the modules too.
Yeah this one is workable.
> i think if you're going to try and load the same modules in node and in the
> browser then if you're using amd in the browser it might be best to use an
> amd loader in node as well. despite what can be achieved via UMD or other
> methods to try and normalize the different environments, consistent mapping
> of dependencies is still an issue. if you have a "flat" module space then
> no problem but as soon as you rely on node_modules or AMD paths/packages
> config then it's no longer a simple matter.
I'm a bit more optimistic. I do think there is a discrepancy with how
node treats module IDs more like paths internally vs AMD's module ID
approach, but for the standard "package" setup where './' relative
paths are used inside the set of modules, and "absolute" IDs for
outside dependencies, I can see a straightforward reuse.
> in my mind the solution is a package manager that understands (a yet to be
> specified/agreed on) AMD config and automatically manages a config.js for
> you. the same config file could also make up part of a build config -
> additional config, output dir and layers, etc would be in a profile but the
> profile could point to config.js in order to reuse the module resolution
> configuration.
Right, I think that is a similar approach to what Kris Zyp's cpm does,
at least last time I looked. It gets a bit weird because the config
needs to get loaded before main references any dependencies.
So I've been thinking of just injecting the config into the main.js at
the top. It would be great to minimize the config in general,
particularly for this case, but just in general, if we can get most
apps running with no config, that is great.
The stuff in this thread is talking about the extreme edges, where
lots of things with incompatible modules are delivered, and that needs
config, but I would like it as light as possible.
> i'm not sure that its worth the effort to specifically account for
> node_modules - we should be pushing towards using an AMD package manager and
> using an AMD loader in node. there's no reason that we can't fetch modules
> from the npm repository if those were specified as dependencies and we can
> put them on disk in a way that makes sense to AMD. i believe cpm would be a
> good starting point for this kind of tool. if i understand it correctly, it
> creates top level modules to point to "main" modules in packages.
I think the node_modules approach is a concrete one used by JS now,
that has this concern. I want to make sure we can handle the same
cases. If the mapping approach for Dojo cannot handle that case, then
I think there is a problem. But I think it is starting to work out.
> i don't care to add another API if we don't need it. we should be strongly
> resisting any additional APIs. we've done a good job of pushing a lot of
> things to plugins and although this is not a problem to be solved by
> plugins, we still need to be fighting to keep the API surface area as small
> as possible. however, if we were to make any API additions, i'm thinking
> that a define.configure(config) would be a good alternative to
> require(config) so that an amd loader in node could simply define a global
> `define` and avoid needing to work around the "local" require typically
> injected by node. that's really another discussion for another time though.
For a standard config call, then it may make more sense to use
define.config(). For me, it is not so much the node case, but for
script loaders in the browser, some have expressed interest in not
having a global and a local require, which may lead to developer
confusion.
As you mention though, we can sort that out after getting the behavior
of this mapping support figured out, and how to apply it in the
config.
Summary:
I'm coming around to the packageMap idea, but I would prefer to see it
as 'map' and allow it to be used in 'paths' config too.
James
Ben Hockey
On Monday, December 12, 2011 4:59:53 PM UTC-5, James Burke wrote:
Hmm, the paths, at least in requirejs is for the full subtree. An
'a/b' paths config would apply to finding a/b/c. That was how it
worked in Dojo in the old days, unless I'm really losing it.
Relative module IDs should be relative to the reference module ID,not the reference module URL.
Rawld
> This is a follow up from this thread:
> https://groups.google.com/group/amd-implement/browse_thread/thread/44429707a29b7f4d
>
> # Question
>
> How should an AMD loader work if module 'a' and module 'b' both want
> to use module 'c', but different versions of 'c'?
>
Thanks to Ben for publishing dojo's soln already to this thread. I'll
chime in direct response to James' msg:
This problem has been solved with bdLoad (which is rotting since dojo
adopted it) and dojo with packageMaps as described here:
http://livedocs.dojotoolkit.org/loader/amd#relocating-module-namespaces
There's a longer description (although a bit dated, the theory is
correct) here:
http://bdframework.org/bdLoad/docs/bdLoad-tutorial/bdLoad-tutorial.html#nameClashes
I've heard that some don't like the package concept. I strongly
disagree. First, it provides a mechanism to specify and understand a set
of interdependent modules. These modules often have a set of common
properties that are easiest to understand when considered at the package
level (compared to the individual module level). As far as the loader is
concerned, the key properties are the external dependencies which may
clash with other packages' external dependencies...which brings up a key
point: the question posed on this thread is not complete; it should read:
How should the loader work when two trees of modules both depend on
another tree of modules with the same root id, say "X", but the two
trees need different implementations the "X" tree (maybe different
versions, maybe completely different libraries)?
Packages make this kind of reasoning very straightforward...both during
run-time and build-time.
fwiw, I don't find the package "main" property particularly important
(you can certainly use packages without it). And, iiuc, we've all
stopped implementing support for the lib property. So to specify some
package, say "A", you need only write:
packages[{
name:"A", location:"path-to-A-(perhaps-relative-to-baseUrl)"
}]
That doesn't seem too much for all the benefit. And you don't even need
this if you don't want to do fancy thinks like solve mapping probs...I
certainly agree that packages ought to be optional.
--Rawld
Rawld
>
> The thing I do not like about packages is the runtime "main"
> translation. I think it is wrong to have to always require
> configuration to find the main module, this should just be a
> convention thing. One way would be to just have dependencies do a
> require('packageName/main'). Or have a package manager elevate
> packageName/main.js to packageName.js. Otherwise it means there will
> be large config blocks because every package needs an entry.
That's a separate discussion. We can ignore the whole "main" property
for this discussion without effect.
The discussion below seems far, far more complex than packageMaps. E.g.,
renaming tools and such are not required.
The only possible problem area is where a module author includes an
explicit module-id...which should be outlawed. Even then, the problem
could be solved by ignoring the module-id...which would work in most cases.
James Burke
but then a summary at the end, hopefully to start the process of
closing this thread…
On Wed, Dec 14, 2011 at 12:24 AM, Rawld <rg...@altoviso.com> wrote:
> Packages make this kind of reasoning very straightforward...both during
> run-time and build-time.
>
> fwiw, I don't find the package "main" property particularly important (you
> can certainly use packages without it). And, iiuc, we've all stopped
> implementing support for the lib property. So to specify some package, say
> "A", you need only write:
>
> packages[{
> name:"A", location:"path-to-A-(perhaps-relative-to-baseUrl)"
> }]
>
> That doesn't seem too much for all the benefit. And you don't even need this
> if you don't want to do fancy thinks like solve mapping probs...I certainly
> agree that packages ought to be optional.
My main rant against packages is that their behavior is different from
the default module ID-path resolution used in AMD loaders. Example:
If the dependency is 'underscore', then it is normally found via
baseUrl + 'underscore.js'. If 'underscore' was a package, there
*needs* to be a configuration stamp for 'underscore' so that it would
be found at baseUrl+underscore/main.js (in the simplest configuration
form).
I agree with the concept of a "this is a contained set of modules that
has a configuration specific to it" and the package.json for the one
time installation of dependencies into a project.
If a "package" was structured like so in source form (github repo):
packageName/
packageName.js (similar to 'main.js' or 'index.js' in trad packages)
packageName/
support.js (used by packageName.js)
Then installation into a project by placing packageName.js and
packageName/ in baseUrl means in the default, simple case there is no
configuration stamping going on.
This makes the use of a cpm/npm-type of tool very nice -- no source
modification for configuration on dependency install. There should be
a simple convention based project structure possible that does not
require a config object.
All that is just part of the argument that I want to use 'packageMap'
concept but outside a 'packages' config area, also in the 'paths'
config area, since I see 'paths' as a similar concept to 'packages',
but just that 'paths' uses the default ID-to-path logic.
## End Summary:
I like Dojo's packageMap concept for addressing the problem that
started this thread. I want to extend it to work for 'paths' config
too.
Since this aliasing can be used outside of the 'packages' config area,
it probably makes more sense to call this capability 'map' instead of
'packageMap'. I'm open to a different name though. 'alias' also works
for me, but something that does not have 'package' in the name.
So to hopefully start the end of this thread, I suggest the following
resolution:
Use Rawld's/Dojo's packageMap concept, but call it 'map'. Allow it as
part of 'packages' or 'paths' config.
For 'paths' config, if a map is needed, then instead of using a string
value for the path config, use an object, with a 'map' property for
the map, and a 'location' property for the path value. Use 'location'
to have overlap with package config.
{
paths: {
'a': {
//location is optional if 'a'
//can be found via baseUrl
location: 'some/other/a',
map: { 'c': 'c1'}
}
}
}
A 'map' config for a 'paths' entry applies to any module ID that is
under the path entry. So, for 'a/b', it would also use the map config
specified above.
How does that sound?
James
Frank Wang
Rawld
> Thanks for the info Rawld, a couple of comments on the packages thing,
> but then a summary at the end, hopefully to start the process of
>
> On Wed, Dec 14, 2011 at 12:24 AM, Rawld<rg...@altoviso.com> wrote:
>> Packages make this kind of reasoning very straightforward...both during
>> run-time and build-time.
>>
>> fwiw, I don't find the package "main" property particularly important (you
>> can certainly use packages without it). And, iiuc, we've all stopped
>> implementing support for the lib property. So to specify some package, say
>> "A", you need only write:
>>
>> packages[{
>> name:"A", location:"path-to-A-(perhaps-relative-to-baseUrl)"
>> }]
>>
>> That doesn't seem too much for all the benefit. And you don't even need this
>> if you don't want to do fancy thinks like solve mapping probs...I certainly
>> agree that packages ought to be optional.
> My main rant against packages is that their behavior is different from
> the default module ID-path resolution used in AMD loaders. Example:
>
> If the dependency is 'underscore', then it is normally found via
> baseUrl + 'underscore.js'. If 'underscore' was a package, there
> *needs* to be a configuration stamp for 'underscore' so that it would
> be found at baseUrl+underscore/main.js (in the simplest configuration
> form).
I don't know what you mean by "configuration stamp"
Also, "If 'underscore' was a package" doesn't make sense to me *unless*
you are using the package main property. As I understand it, if a module
is a member of a package, then it must have at least two path segments,
the first being the package name...unless it is the package "main"
module, in which case it can be referenced by just the package name.
Again, I'm neutral on the "main" property.
> I agree with the concept of a "this is a contained set of modules that
> has a configuration specific to it" and the package.json for the one
> time installation of dependencies into a project.
>
> If a "package" was structured like so in source form (github repo):
>
> packageName/
> packageName.js (similar to 'main.js' or 'index.js' in trad packages)
> packageName/
> support.js (used by packageName.js)
>
> Then installation into a project by placing packageName.js and
> packageName/ in baseUrl means in the default, simple case there is no
> configuration stamping going on.
So, iiuc, you are saying the packages myPackage and yourPackage would be
installed like this:
<baseUrl>/
myPackage.js
myPackage/
yourPackage.js
yourPackage/
If that is correct, then what if you want to install two different
packages (different versions or different completely) with the same
names? Encouraging a idiom that requires renaming two objects seems less
optimal than saying the entire package is completely under a single
tree...which would require renaming one object...the root of the tree.
> This makes the use of a cpm/npm-type of tool very nice -- no source
> modification for configuration on dependency install. There should be
> a simple convention based project structure possible that does not
> require a config object.
Hmmm, but you must rename a source file (rather than the location the
source is deposited). Maybe I don't understand your proposal here.
> All that is just part of the argument that I want to use 'packageMap'
> concept but outside a 'packages' config area, also in the 'paths'
> config area, since I see 'paths' as a similar concept to 'packages',
> but just that 'paths' uses the default ID-to-path logic.
I think paths are for mapping modules, and I don't want to use paths for
mapping packages because I think it's the wrong level of abstraction. I
think packages should be treated differently than individual modules so
I can solve a higher-level problem: I want to be able to say package A
references an external package *named* B, and Package X also references
an external package named B, but the two B's are different packages. Do
you have a soln for that problem using paths?
I'm not sure we're disagreeing.
> ## End Summary:
>
> I like Dojo's packageMap concept for addressing the problem that
> started this thread. I want to extend it to work for 'paths' config
> too.
>
> Since this aliasing can be used outside of the 'packages' config area,
> it probably makes more sense to call this capability 'map' instead of
> 'packageMap'. I'm open to a different name though. 'alias' also works
> for me, but something that does not have 'package' in the name.
map is fine with me.
> So to hopefully start the end of this thread, I suggest the following
> resolution:
>
> Use Rawld's/Dojo's packageMap concept, but call it 'map'. Allow it as
> part of 'packages' or 'paths' config.
>
> For 'paths' config, if a map is needed, then instead of using a string
> value for the path config, use an object, with a 'map' property for
> the map, and a 'location' property for the path value. Use 'location'
> to have overlap with package config.
>
> {
> paths: {
> 'a': {
> //location is optional if 'a'
> //can be found via baseUrl
> location: 'some/other/a',
> map: { 'c': 'c1'}
> }
> }
> }
>
> A 'map' config for a 'paths' entry applies to any module ID that is
> under the path entry. So, for 'a/b', it would also use the map config
> specified above.
>
> How does that sound?
So, map works like dojo packageMap in package config, correct?
And for a module that matches a path with a map config that refs another
module with that other module's first segment found in the map, then
apply the map.
For example, if the module "a" contained the following...
define(["c"], //...
then "c" would be resolved to "c1". Is that correct?
After it's resolved to "c1", is "c1" applied to paths again?
--Rawld
James Burke
Right, your alias: config is the same as the paths: config in other loaders.
This is important to have, since directly using versioned modules like
require('jquery/1.7.1/jquery') is not very scalable -- for things like
Backbone and jQuery plugins, it is best they just specify 'jquery' as
the dependency name, as they work with many versions of jQuery. The
version range of compatible dependencies are best reflected in a
package.json file that can be used on installation of the module into
a project.
James
James Burke
> On 12/14/2011 12:58 PM, James Burke wrote:
>> If the dependency is 'underscore', then it is normally found via
>> baseUrl + 'underscore.js'. If 'underscore' was a package, there
>> *needs* to be a configuration stamp for 'underscore' so that it would
>> be found at baseUrl+underscore/main.js (in the simplest configuration
>> form).
>
>
> I don't know what you mean by "configuration stamp"
>
> Also, "If 'underscore' was a package" doesn't make sense to me *unless* you
> are using the package main property. As I understand it, if a module is a
> member of a package, then it must have at least two path segments, the first
> being the package name...unless it is the package "main" module, in which
> case it can be referenced by just the package name.
Sorry, this is difficult to express -- I'm grappling with how to say
it succinctly since there are a few things wrapped up in this. The
following is not succinct, but trying to outline what the factors and
choices are, just so I have something I can refer back to later. I
have a summary at the end of this section though:
Packages, as defined via CommonJS-ish things, where all the code is in
a directory, is nice to easily deliver code -- that code could a few
modules inside it that interconnect. Having it all inside a directory
makes the use of relative paths for module IDs easy, and works even
after renaming the top level directory.
Module loading in the browser only gets one IO lookup chance per
module. So, how should a loader resolve a module called 'foo/bar'?
There needs to be an algorithm for it.
For me, it is very important that the algorithm is based on convention
and only convention can be used for most cases. Of course there needs
to be a way to do explicit configuration, but it should not be a
requirement to insert a configuration value for every module or
"package" that is used in the project.
Traditionally, scripts on the web are single file items. It is common
to grab a few scripts, put them in a scripts/ directory and you are
ready to go.
The convention used in most AMD loaders makes that an easy case to
handle -- if scripts/ is the baseUrl, then there is no need to put in
any other config values to find scripts.
However, if a CommonJS package thing is inserted into the project, it
has a different convention than simple one file scripts. Asking for
"foo" does not load baseUrl+foo.js, but baseUrl+foo+[main.js ||
package.json main config value].
However, given that the lookup rules for a package are different from
the simple baseUrl+foo.js convention in browser loaders, using these
packages mandates an insertion of a loader configuration value (what I
meant by a "configuration stamp") when they are installed.
That could be avoided if:
1) browser loaders always assumed that the first segment of an
absolute module ID is a package, and that "main.js" is normally used
for the main module. This means that using convention-based packages
do not require a configuration stamp on install. However, it then
requires some config stamp for single-file scripts that are common on
the web.
2) Encourage the construction of packages with the convention like the
following:
packageName/
packageName.js
packageName/
This allows the a package installer to insert those two inner values
at the baseUrl of the project and then no configuration stamping is
needed.
The difficulty with #2 is this aliasing/map feature being talked about
in this thread. While I do not think it is a big deal to rename two
file items instead of one, the ugly part comes in the need to adjust
any relative module ID references in packageName.js and also scanning
the inner packageName/ directory in case one of the modules has a
require('../packageName') mention. And there is a chance this renaming
work could fail if there are dynamic require([]) calls happening that
use constructed IDs that may assume 'packageName' in there.
#1 might be possible to do by relying on a package installer tool to
place single file scripts in a 'packageName/main.js' structure, but I
really dislike the need to rely on tools for this sort of thing, and
that it mangles the names. So instead of seeing jquery.js in the
debugger, I'll see jquery/main.js, and a bunch of other 'main.js'
files.
I strongly believe it should be possible to get working without
tooling, and keep the single file script distribution alive.
But I do not like the the #2 case because of the ID rewriting hazard
if it is used with the "map" approach in this thread. If there are two
versions of 'c' used in the project, one of them needs to be placed at
'c-2', but that means messing with the package's relative ID
internals.
Time for a #3 option:
======
Summary
======
Keep packages as they exist today, but instead of requiring a
configuration stamp when installed, on install create an adapter
module that is called packageName.js that is a sibling to packageName/
and have it just export the packageName/main.js value (the package
install tool can read the package.json and change that to be whatever
the main module really is). Example of adapter module's contents:
define(['packageName/main'], function (main) {
return main;
});
This is robust against renaming for 'map' configs -- this file is
created on package install, once the new name is known, and the
contents of the package are not touched.
It does create a bit more manual work if you want to manually rename
it after, but I do not think it is too bad, particularly since it
avoids loader config modifications. This will be a much rarer thing
to do than just being able to easily start with modules.
I think with that, it solves my concern about needing to touch the
loader config on each package install, and it works with the map
approach being discussed in this thread.
> I think paths are for mapping modules, and I don't want to use paths for
> mapping packages because I think it's the wrong level of abstraction. I
> think packages should be treated differently than individual modules so I
> can solve a higher-level problem: I want to be able to say package A
> references an external package *named* B, and Package X also references an
> external package named B, but the two B's are different packages. Do you
> have a soln for that problem using paths?
>
> I'm not sure we're disagreeing.
Right, for me, 'paths' are just 'packages' but with different path
resolution policies. I want to avoid mandating 'map' as only a
'packages' property since use of 'packages' *requires* stamping the
loader config on package install, since packages path resolution
policy is different from the default policy.
To put it another way: I see 'paths' just like 'packages', except
right now 'paths' just lets you set the 'location' property, and the
default path policy is a bit different (paths does not need 'main').
So in that light, I would like to use 'map' in 'paths' too. I would
still support it in a 'packages' config too, but I'm hoping over time,
with a package installer tool+packageName.js adapter creation, there
would be less use of 'packages' since the default path policy can be
used.
So that would mean 'paths' can have just a string that represents
'location', how it works today:
paths: {
a: 'some/path/to/a'
}
but it could be expanded to have a 'map' also, and if the location was
different than just baseUrl+a.js, a 'location' property:
paths: {
a: {
//location only needed if different
//than baseUrl location
location: 'some/path/to/a',
map: { c: 'c1' }
}
The 'map' values would apply for 'a/b' module IDs too, any sub-ID under 'a'.
One difference with 'paths' vs 'packages' is that 'paths' can operate
on a deeper name prefix:
paths: {
'a/b': {
map: { c: 'c2' }
}
> So, map works like dojo packageMap in package config, correct?
I believe so, but it is best to explore exactly what that means, like
your next comment does:
> And for a module that matches a path with a map config that refs another
> module with that other module's first segment found in the map, then apply
> the map.
>
> For example, if the module "a" contained the following...
>
> define(["c"], //...
>
> then "c" would be resolved to "c1". Is that correct?
>
> After it's resolved to "c1", is "c1" applied to paths again?
Yes. I would also expect that if 'c1' had a 'packages' config instead
of 'paths', that 'packages' config would be applied to 'c1' to find
it.
James