allow for alternate strategies for resolving pre-reqs

[Patrick Mueller]

I have to admit to not having much experience playing with some of the popular AMD implementations.  It's not clear from the spec how things like requirejs's config.path http://requirejs.org/docs/api.html#config fit into the big picture of resolving module references.  I guess there's some allowance in the AMD spec to allow implementations to have flexible means of resolving module references to actual modules.

I'd like to expand on this.  I am currently more or less familiar with node.js's module search algorithm http://nodejs.org/docs/v0.6.5/api/modules.html#loading_from_node_modules_Folders , which involves things like rooting through relevant node_module directories, looking for relevant package.json files, etc.  I would actually like to be able to use *exactly* this lookup scheme in AMD.  Specifically, I'd like to be able to reuse an existing non-trivial node "package" (with dependencies) using an AMD loader.

A little more; it needs to also work with "built" AMD runtimes like almond.  https://github.com/jrburke/almond

I don't necessarily want to "bake this into" an AMD implementation; I would prefer it to be a pluggable strategy that I can provide to AMD before it's gears start grinding away.

Not sure how closely related this is - it might point to the the answer: provide a hookable function like node's require.resolve() http://nodejs.org/docs/v0.6.5/api/modules.html#all_Together... - this is the only reference I could easily find to this function.  The basic idea is to allow someone else to provide the name/module lookup, presumably given the 1) module executing the `require()`, 2) the string value of the module being `require()`d.  I don't believe node's require.resolve() is hookable, am suggesting that maybe AMD could allow for such a thing to be hookable.

Perhaps there's some other way of accomplishing this, particular for "built" environments like almond, but I don't think so, given the richer semantics of node's search algorithm.

I'm willing to pay a price of creating some "fake" modules for directories, package.json file contents, etc, to make this happen.

Here's an example of a hand-built node-styled package hierarchy, where both the a and b modules load a "top level" module name c, which are actually different.  Also demonstrates the node_modules, package.json and index.js "patterns" that node makes use of.

https://github.com/pmuellr/sample-npmish-project

Note, I opened a bug at amdjs-api, because I couldn't think of a better place to ask this question, then I remembered this mailing list.  The issue here is, can be closed if folks feel strongly about it, but guess the conversation should take place here for now.  Or is there a better place?

https://github.com/amdjs/amdjs-api/issues/6

--
Patrick Mueller
http://muellerware.org

[Patrick Mueller]

Since my post was a little long-winded, wanted to clarify one thing:

I'm not looking for AMD to hard-code the node-type package/module resolution algorithm (yet!).  I'm looking for a way for me to plug some code into AMD so that I can implement it.

Good for a couple of reasons:

- some people may not want this (but those people are crazy :-) )

- who knows, next year we may find a better package/resolution story, and then all we need is a new resolver plugin

- the use of the hard-coded "node_modules" subdirectory grates on me, but is sadly legacy at this point; but perhaps a "built" version of these packages, used with an almond-style loader, could do something more aesthetically pleasing, at least as an internal detail.

[John Hann]

Hey Patrick,

IIUC, RequireJS allows its internals to be overridden to do things such as this.  curl.js 0.6 (in dev) also allows this.  Since there is no standard API for overriding the internals of AMD loaders and AMD builders, you'll have to create something specific to your favorite.  :)

I agree that there needs to be more synergy with node.js.  Maybe we can convince them to do things our way ... some day.  :)

-- J

[Patrick Mueller]

On Thu, Dec 8, 2011 at 11:27, John Hann <jo...@e-numera.com> wrote:

IIUC, RequireJS allows its internals to be overridden to do things such as this.  curl.js 0.6 (in dev) also allows this.  Since there is no standard API for overriding the internals of AMD loaders and AMD builders, you'll have to create something specific to your favorite.  :)

Right.  I think that's one of the things that bugs me about AMD.  It's like J2EE was - specified very high level APIs, but then to do anything real, you needed to do implementation-specific things.  It's not clear to me even where the whole "plugins" thing lives - standard?  implementation-specific?.  (and as an aside: I wish AMD implementations didn't have "plugins", but built additional facilities on top of the existing standard APIs to accomplish the same goals).

Hopefully this is just an issue of time; namely, over time, we'll have more standard ways of futzing with lower-level APIs.

One direction to start looking for WHERE to put such lower-level APIs, would be to create a new standard SPI module.  Call it "AMD-SPI".  The benefit to having this in a separate module is to keep the namespace pollution down - we got modules, damn it, let's fricken use them!

Presumably, code that used AMD-SPI would be a kind of of meta-code that you'd use as a prologue for the rest of your code; ie, every project could include some kind of setup logic that futzed with the AMD-SPI to set up module resolvers, config paths, etc.  You'd imagine there'd be a couple of flavors of this - one that handled requirejs configs, one that handled node.js module resolution, one that handled both (!), etc.  You'd just pull whichever one of these was appropriate for your project, into your project.  Perhaps we'd standardize that when an AMD runtime starts up, it looks for a special module (eg, "runtime-setup") which would contain this kind of code, so we'd have even less config, and more convention.  :-)

I agree that there needs to be more synergy with node.js.  Maybe we can convince them to do things our way ... some day.  :)

(aside and not relevant to the discussion but thought I'd chime in: I'd never bet real money that this would ever happen.  It doesn't seem productive to me to even consider it)

 

[James Burke]

On Thu, Dec 8, 2011 at 7:04 AM, Patrick Mueller <pmu...@gmail.com> wrote:
> I have to admit to not having much experience playing with some of the
> popular AMD implementations.  It's not clear from the spec how things like
> requirejs's config.path http://requirejs.org/docs/api.html#config fit into
> the big picture of resolving module references.  I guess there's some
> allowance in the AMD spec to allow implementations to have flexible means of
> resolving module references to actual modules.

Warning, a bit long:

Here are the (enumerated) questions I think are being asked, but feel
free to correct me:

###########
1) How best to translate code written for node to work in the browser?

---
A more general note/rant first: node's module lookup rules may work
well for a server, but they are not good for a browser. They are still
figuring out what is best for their environment and they have made a
point to explicitly not care what works best in a browser. They are
inwardly focused. That is fine, but I'm also not going to pick up any
of their baggage that makes it harder to program in the browser. Even
though node is hot, it still pales in comparison to all front-end
development -- there are all kinds of people using different kinds of
servers but still targeting the browser, and it is not possible to
write FE programs that are completely separate from the server (mobile
apps).
---

On to specifics of reusing node modules in an AMD loader, how to deal
with paths/module ID resolution:

I can see two paths:

a) When doing the module conversion to wrap the modules in define(),
update the dependency names to match how they will be organized. In
your sample-npmish example, in a.js, for its require('c') call,
rewrite it to require('a/c').

b) Allow setting a config that would help the loader do the
resolution. I think this is the path you want to take. I think this
requires some upgrades to the loaders first to really understand (see
#2 below).

It would be interesting to know what path browserify takes when it
bundles up node modules. Not as a "we should just do that" but just to
get a point of reference. If anyone has more experience with it, feel
free to share.

###########
2) How can modules that may need their own versions of the same module work?

Node solves this via nested node_modules lookups and has the IO
flexibility to try different paths. AMD loaders should avoid multiple
IO lookups to find a module, it does not work on the web.

So the configuration to find a module at exactly one location needs to
be set before the loader starts looking up modules.

There have been proposals on how that might work in the past. I have
found them to be a bit unwieldy and rely too much on paths, where I
would rather have a nested module ID-centric config that probably has
a new wrapping function in the define API. AMD is already more "module
ID"-centric than node's path-centric. But we should start a separate
thread on that (see summary below).

###########
3) Does there need to be a lower level AMD service API (SPI) to
accomplish 1 and 2?

I would rather tackle the node-specific bundling first, work with
concrete use cases and if something general pops up, consider it later
at that time.

Also, I do not think standardizing the SPI for method names would work
unless there was some general agreement on allowing multiple versions
of modules and how they would work and get serialized into a file. So
in other words trying to get some agreement or sketch on #2 above.

For the more more futuristic "there might be something better later",
I expect it will be sufficiently different that we will not get the
SPI interface set up correctly if we try to sort it out now. I think
it is more likely there will be specialized loaders to handle the new
thing and if it seemed generally useful, then consider it more in this
group. In other words, I would like to see concrete implementations
driving the API work.

###########
Summary of how I would like to proceed:

* Start a thread about how to handle multiple versions of the "same"
module being loaded. It has come up before way in the past, but I
think we may be in a better position to work something out now. I have
some ideas, but I will save them for the separate thread. I'll start
that thread later today.

* Start a different thread on how node modules that have a layout like
sample-npmish can be used today. I think it can be done as mentioned
before by doing dependency rewriting in the bundling tool. I'm open to
exploring this more in code too. If it makes sense to send some pull
requests to browserify for this so it outputs AMD that works for this
situation, even better.

It may be that just doing the dependency rewriting is the way to go
for now -- it will work with more loaders, and makes the loaders
simpler, which is good. Since the modules need to be converted to
define() anyway, there is likely greater leverage in doing the hard
work in one well-written converter than have lots of loaders have to
change.

That said, I would like to work out the "multiversion" issue for AMD
for the long term.

> Note, I opened a bug at amdjs-api, because I couldn't think of a better
> place to ask this question, then I remembered this mailing list.  The issue
> here is, can be closed if folks feel strongly about it, but guess the
> conversation should take place here for now.  Or is there a better place?
>
> https://github.com/amdjs/amdjs-api/issues/6

I went ahead and closed the issue with a link to this thread, since
this is more of a general topic thread, trying to discover what
problems should be solved. Once those are identified, we can add
issues as needed.

James

[Patrick Mueller]

On Thu, Dec 8, 2011 at 13:55, James Burke <jrb...@gmail.com> wrote:

 it is not possible to

write FE programs that are completely separate from the server (mobile
apps).

What's an FE program?

 

I can see two paths:

a) When doing the module conversion to wrap the modules in define(),
update the dependency names to match how they will be organized. In
your sample-npmish example, in a.js, for its require('c') call,
rewrite it to require('a/c').

Out of the question.  Inconceivable.

 

b) Allow setting a config that would help the loader do the
resolution. I think this is the path you want to take. I think this
requires some upgrades to the loaders first to really understand (see
#2 below).

yes, this is where I'm headed.

 

It would be interesting to know what path browserify takes when it
bundles up node modules. Not as a "we should just do that" but just to
get a point of reference. If anyone has more experience with it, feel
free to share.

I should have relooked at Browserify, I've only casually perused it in the past.

2) How can modules that may need their own versions of the same module work?

If you imagine you've solved the node module resolution problem, then this problem is a solved problem.  If you haven't solved (or enabled) the node resolution algorithm, then yeah, something needs to be done.  But I'll always be using the node module resolution algorithm so I don't care. 

I would rather tackle the node-specific bundling first, work with
concrete use cases and if something general pops up, consider it later
at that time.

Fair enough.

 

Also, I do not think standardizing the SPI for method names would work
unless there was some general agreement on allowing multiple versions
of modules and how they would work and get serialized into a file. So
in other words trying to get some agreement or sketch on #2 above.

I think the notion of an SPI is broader than just the multiple versions of a module problem.  But that's fine.

 

* Start a thread about how to handle multiple versions of the "same"
module being loaded. It has come up before way in the past, but I
think we may be in a better position to work something out now. I have
some ideas, but I will save them for the separate thread. I'll start
that thread later today.

Again, probably not interested myself, but carry on, I'll lurk.

 

* Start a different thread on how node modules that have a layout like
sample-npmish can be used today. I think it can be done as mentioned
before by doing dependency rewriting in the bundling tool. I'm open to
exploring this more in code too. If it makes sense to send some pull
requests to browserify for this so it outputs AMD that works for this
situation, even better.

Again, dependency rewriting - aka as parsing, analyzing, and rewriting your source - is out of the question.  It clearly couldn't work if I used variables for the require() parameter, for instance.  I'm willing to wrap CJS-authored modules, but not to do major surgery on them.

But thanks for the ref to Browserify.  On it.

 

I went ahead and closed the issue with a link to this thread, since

this is more of a general topic thread, trying to discover what
problems should be solved. Once those are identified, we can add
issues as needed.

 

np 

[James Burke]

On Thu, Dec 8, 2011 at 11:49 AM, Patrick Mueller <pmu...@gmail.com> wrote:
> On Thu, Dec 8, 2011 at 13:55, James Burke <jrb...@gmail.com> wrote:
>
> What's an FE program?

Sorry, Front End -- applies to browser-based web development. Server
code is Back End (BE).

>> 2) How can modules that may need their own versions of the same module
>> work?
>
> If you imagine you've solved the node module resolution problem, then this
> problem is a solved problem.  If you haven't solved (or enabled) the node
> resolution algorithm, then yeah, something needs to be done.  But I'll
> always be using the node module resolution algorithm so I don't care.

I believe what I proposed in the other thread on a scope API would
allow someone to code up a config that would mimic a nested
node_modules setup, as long as you know what path node chose to load
each module.

> Again, dependency rewriting - aka as parsing, analyzing, and rewriting your
> source - is out of the question.  It clearly couldn't work if I used
> variables for the require() parameter, for instance.  I'm willing to wrap
> CJS-authored modules, but not to do major surgery on them.

For a generic bundling tool, one that does not require the developer
to hand write the dependencies, it will need to parse out the require
calls to properly find the dependency tree and include those modules
in a bundled package. Since it already needs to parse the require
calls, it can do the renaming.

Variable-based require calls are not parseable by that solution, and
they would likely fail anyway in a browser env because they are
imperative require calls, and should be handled by a callback-style
require([], function(){}).

I agree that it is not a 100% solution, but for modules with
declarative string dependencies (most modules) it would be a workable
solution, particularly for modules that might actually be runnable in
the browser vs modules that depend on node's execution model.

But I understand if you are not interested in pursuing that approach
for your own tooling. I am going to focus my energy on the larger
multiversion issue, but it will likely require more coordination and
time than the "renaming tool" path.

James

[Patrick Mueller]

On Thu, Dec 8, 2011 at 13:55, James Burke <jrb...@gmail.com> wrote:

It would be interesting to know what path browserify takes when it

bundles up node modules. Not as a "we should just do that" but just to
get a point of reference. If anyone has more experience with it, feel
free to share.

 

I was able to use browserify to get the sample-npmish-project running in a browser; here's the wad of code it generated:

    https://github.com/pmuellr/sample-npmish-project/blob/master/using/browserify/sample.browserified.js

If it makes sense to send some pull
requests to browserify for this so it outputs AMD that works for this
situation, even better.

Not sure where you're going with this one.  Browserify doesn't rewrite my modules.  It works it magic by embedding the npm module resolution into define() and require().  I guess we could see if Browserify could also become AMD compliant, but that's not really my goal.  I would like AMD compliance to mean that it handles npm module resolution, perhaps optionally, or via configuration.  I'd like to have the universe of npm modules available to all users of AMD compliant runtimes.

-- 

Patrick Mueller
http://muellerware.org

[Patrick Mueller]

On Thu, Dec 8, 2011 at 17:59, James Burke <jrb...@gmail.com> wrote:

I believe what I proposed in the other thread on a scope API would

allow someone to code up a config that would mimic a nested
node_modules setup, as long as you know what path node chose to load
each module.

Are "configs" portable across AMD implementations?  Is there a spec for them?

 

For a generic bundling tool, one that does not require the developer

to hand write the dependencies, it will need to parse out the require
calls to properly find the dependency tree and include those modules
in a bundled package. Since it already needs to parse the require
calls, it can do the renaming.

I don't believe Browserify scans.  I've written "generic bundling tools" which do not scan.

 

Variable-based require calls are not parseable by that solution, and
they would likely fail anyway in a browser env because they are
imperative require calls, and should be handled by a callback-style
require([], function(){}).

I'm 100% sure I could use variable-based require calls in my code, in a browser, using almond, and they would never fail.

So, you can't parse, so you can't rewrite, it'll never work for all cases, don't bother.

[James Burke]

On Thu, Dec 8, 2011 at 3:11 PM, Patrick Mueller <pmu...@gmail.com> wrote:
> On Thu, Dec 8, 2011 at 13:55, James Burke <jrb...@gmail.com> wrote:
>>
>> It would be interesting to know what path browserify takes when it
>> bundles up node modules. Not as a "we should just do that" but just to
>> get a point of reference. If anyone has more experience with it, feel
>> free to share.
>
>
> I was able to use browserify to get the sample-npmish-project running in a
> browser; here's the wad of code it generated:
>
>
> https://github.com/pmuellr/sample-npmish-project/blob/master/using/browserify/sample.browserified.js

Neat, thanks for sharing. Browserify seems to just dump files in there
by their path name. I'll have to think about this more. On first look
though, I dislike the use of full paths for module IDs and the
package.json stuff in there.

>> If it makes sense to send some pull
>> requests to browserify for this so it outputs AMD that works for this
>> situation, even better.
>
>
> Not sure where you're going with this one.  Browserify doesn't rewrite my
> modules.  It works it magic by embedding the npm module resolution into
> define() and require().  I guess we could see if Browserify could also
> become AMD compliant, but that's not really my goal.  I would like AMD
> compliance to mean that it handles npm module resolution, perhaps
> optionally, or via configuration.  I'd like to have the universe of npm
> modules available to all users of AMD compliant runtimes.

I just meant that if browserify output something that was close to AMD
already and just required a bit of small massaging, then that would be
useful. However, it is encoding some other node-isms in there that may
not make that path realistic. Needs more thought though.

James

[James Burke]

On Thu, Dec 8, 2011 at 4:03 PM, Patrick Mueller <pmu...@gmail.com> wrote:
> Are "configs" portable across AMD implementations?  Is there a spec for
> them?

No, not yet, but there are enough implementations that I think share
at least baseUrl and paths that it may make sense to do that now.

> I don't believe Browserify scans.  I've written "generic bundling tools"
> which do not scan.

Besides dependency parsing, the only other way I have thought of doing
a generic tool is to actually run the modules and listen for what is
loaded. However, this does not work well once the modules that are
being run normally run in a different environment than the tool. For
instance, processing browser-based modules that may reference "window"
in node.

If there is another way of doing these kinds of tools though, please
let me know. Maybe I could use that path whenever I build my next
tool.

James

[Patrick Mueller]

On Thu, Dec 8, 2011 at 19:14, James Burke <jrb...@gmail.com> wrote:

Neat, thanks for sharing. Browserify seems to just dump files in there

by their path name. I'll have to think about this more. On first look
though, I dislike the use of full paths for module IDs and the
package.json stuff in there.

I figured all the module impls used a "full path" (relative to some page/project "root") for their moduleIDs.  Folks aren't using integers or something these days, are they?

The package.json stuff can't be that different from the plugin support that many AMD implementations seem to have, can it?  I always imagine it worked pretty much exactly like the Browserify version of the package.json files.  How would a typical AMD implementation using plugins store a .json file in a single-file cat'd build?

[Ben Hockey]

hi patrick,

to use dojo to load the structure in the npmish project (assuming the modules were wrapped in define), the configuration would be something like below (i'll keep https://gist.github.com/1450027 updated if i make changes).  i haven't actually made any attempt to wrap the modules and test this so it may need some tweaking but this shows the idea.

<script>
        var require = {
        // adjust to root of https://github.com/pmuellr/sample-npmish-project
        baseUrl: 'npmish',
        packages: [
            {
                name: 'a',
                location: 'node_modules/a',
                main: 'a.js',
                packageMap: {
                    c: 'c-a'
                }
            },
            {
                name: 'b',
                location: 'node_modules/b',
                main: 'b.js',
                packageMap: {
                    c: 'c-b'
                }
            }
        ],
        paths: {
            'c-a': 'node_modules/a/node_modules/c/c',
            'c-b': 'node_modules/b/node_modules/b/index'
        },
        deps: ['./sample']
    };
</script>
<script src="path/to/dojo/dojo.js"></script>

thanks,

ben...

[Patrick Mueller]

On Thu, Dec 8, 2011 at 22:42, Ben Hockey <neonst...@gmail.com> wrote:

to use dojo to load the structure in the npmish project (assuming the modules were wrapped in define), the configuration would be something like below (i'll keep https://gist.github.com/1450027 updated if i make changes).  i haven't actually made any attempt to wrap the modules and test this so it may need some tweaking but this shows the idea.

Thanks Ben.  You can see from the project, that I've added a recipe for how to do this with browserify, under the using directory:

    https://github.com/pmuellr/sample-npmish-project

I'd be happy to pull a request for a using/dojo subdirectory :-)

I guess the implication with what you've done is that, if I needed to do this generically, with a tool, I'd build a version of the packages and path elements of the require structure which included every module in a directory tree.

 

[Ben Hockey]

i saw the browserify sample and the only issue i had with doing the using/dojo was that i'm not sure of your criteria and end goal.

• can i copy the whole directory tree (with wrapped modules) into the using/dojo directory?
• do you want a "production" build (concatenated, minified, etc) or do you just want the bare minimum to get this to load?
• i thought i read a comment of yours recently that said you already have a tool for wrapping the modules so wasn't sure if that factored into this
• how automated does the process need to be?  if you add module 'd' do i need to automatically pick that up?

i'm not sure that i'm up to putting very much time into this anytime soon which is why i stopped at offering a config off the top of my head but if i had some more guidance it would help me focus any time i could put into this.

thanks,

ben...

[Patrick Mueller]

On Fri, Dec 9, 2011 at 22:14, Ben Hockey <neonst...@gmail.com> wrote:

i saw the browserify sample and the only issue i had with doing the using/dojo was that i'm not sure of your criteria and end goal.

• can i copy the whole directory tree (with wrapped modules) into the using/dojo directory?

Don't matter to me!  It's your directory!

• do you want a "production" build (concatenated, minified, etc) or do you just want the bare minimum to get this to load?

Bare minimum is fine, certainly don't minize, but concatenate is fine.  I'd expect you will put an .html file in the using/XYZ directory which will "run" the lamo sample. 

• i thought i read a comment of yours recently that said you already have a tool for wrapping the modules so wasn't sure if that factored into this

Assume no other tools.  You can "hand" wrap if you want to. 

• how automated does the process need to be?  if you add module 'd' do i need to automatically pick that up?

I expect a script &| manual instructions that result in getting the .html you provide to work.

 

i'm not sure that i'm up to putting very much time into this anytime soon which is why i stopped at offering a config off the top of my head but if i had some more guidance it would help me focus any time i could put into this.

ya, it's extra-credit, but it would be awesome to see more examples.  Especially (primarily) AMD ones.