A better require() extension system?

[Aseem Kishore]

Hey there,

I've never read any documentation on this, but from looking at CoffeeScript, Streamline.js and node-dev, it seems that the current standard way of extending/customizing require() is by assigning a handler function to require.extensions['.ext']:

• https://github.com/jashkenas/coffee-script/blob/master/src/coffee-script.coffee#L16-L22
  
  
• https://github.com/Sage/streamlinejs/blob/master/lib/compiler/register.js#L46-L49
  
  
• https://github.com/fgnass/node-dev/blob/master/wrapper.js#L47-L59

[For context: CoffeeScript is an alternate JS language; Streamline transforms synchronous-looking code to async; node-dev watches files you require and restarts your node process on those file changes.]

But the key point is that the signature of these handlers is void -- instead of returning anything, they perform a side-effect of telling Node directly what code to use. They seem to do this via module._compile(content, filename), where "content" is a string of JavaScript. (In this way, _compile() seems analogous to eval.)

This system seems to be kind of subpar. One library's handler can't reuse the logic in another library's handler, and order becomes fragile. For example:

• To support Streamlined CoffeeScript, Streamline has to both register its handler *after* CoffeeScript (so it doesn't get overridden), and it has to duplicate the logic in CoffeeScript for compiling files.
  
  
• node-dev doesn't work with Streamline, because node-dev registers its handler before requiring any other .js files, and Streamline overwrites that handler.
  
  
• I wrote a simple utility to cache compiled Streamline and/or CoffeeScript files, to improve startup time during development, and in this, I had to duplicate the logic in both Streamline and CoffeeScript, since I need to know what the compilation output is. Source: https://gist.github.com/1068606

What are the community's thoughts on improving this system? I would think that ideally, you would be able to wrap/chain these handlers easily, and have them return (transform) code instead of the handlers performing side effects.

Thanks in advance for the consideration, and let me know if I've yet to learn anything important here.

Cheers,

Aseem

[Aseem Kishore]

Anyone have any thoughts or responses? =D

Aseem

[Isaac Schlueter]

I think that this is one area where it's very easy to overdo things.

It sounds like the meat of your complaint is that streamline and
coffee-script both want to instrument .coffee files, and can clobber
each other. It seems easy enough for these two systems to detect if
there's already a require.extensions[".coffee"] and behave
appropriately.

As for the coffee-script repl, all that's needed is to set
`module.filename` and `module.paths` appropriately. Here's the two
lines that do this in node's lib/repl.js:

// hack for require.resolve("./relative") to work properly.
module.filename = process.cwd() + '/repl';

// hack for repl require to work properly with node_modules folders
module.paths = require('module')._nodeModulePaths(module.filename);

So, I'm not sure I buy that either of these problems are unsolvable
with the tools already exposed.

There have been a few requests to allow for a way to customize the
module lookup process. This *can* be done, though it's a bit
heavy-handed and brittle, by modifying
require("module")._resolveFilename or require("module")._findPath.

These APIs are undocumented and private, and any program that relies
on them should be considered a not-for-prime-time science experiment
(since unpublished APIs can change at any time, with no warning). But
if that's really a requirement for some use case, it might be good to
explore what kind of API would make sense there.

I cannot stress how important simplicity is in this area. There is a
fair amount of necessary complexity for the use cases we support, and
adding any unnecessary complexity makes it very easy to break every
node program, since *everyone* depends on modules loading properly.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

[Aseem Kishore]

Hey Isaac,

Thanks for the reply, but I don't get a sense that we're talking about the same thing. Let me try being more clear.

I'm not at all talking about the module lookup process, the CoffeeScript REPL, or anything like that. I'm talking only about: when you want to transform a source file when it's required, how do you do that in an extensible way?

Right now, you do it by attaching a handler to require.extensions for the extension you care about, and in that handler, you have to do *everything*: read the file, transform it, and tell Node to use it. Here's the default JS handler:

https://github.com/joyent/node/blob/master/lib/module.js#L442-L446

The key statement is that module._compile(). That's a function with a side effect. So handlers can have side effects, and worse, they're void, so they don't return the output of their transformations.

So if you're a library like Streamline, and you want to transform JS, you can't "chain" or "wrap" that default handler -- you have to reimplement and extend that logic entirely yourself.

If you're writing a utility to cache Streamline output, you can't simply call the Streamline handler, because it doesn't return the output. So you have to again reimplement all of Streamline's logic in order to extend it.

In other words, this system is just not extensible at all beyond one level. Multiple libraries that want to touch the same extension will have to either clobber each other, or reimplement each other.

To address what you wrote, then:

It seems easy enough for these two systems to detect if
there's already a require.extensions[".coffee"] and behave
appropriately.

My point is that there is nothing these systems can do; what does it mean to "behave appropriately"? They can't call an existing handler, because that handler will tell Node to compile something.

Hope this clears up my request. Thanks again for your response!

Aseem

[Isaac Schlueter]

I see. So you'd prefer, then, to have some sort of API where your
function gets a listing of source, and can return transformed source?

I think that could be done in a pretty backwards-compatible way.
Maybe something like this?

require("module").addTransform(".coffee", function (src, filename) {
if (src === null) src = fs.readFileSync(filename);
return someFancyStreamlineThingie(src, filename);
});

So, each thing like this pushes another transformer function onto a
queue. The first default item would be something like:

function (src, filename) { return fs.readFileSync(filename) }

That'd also be really handy for automatically adding code-coverage with burrito.

[Aria Stewart]

You're gonna have to return a tuple of [content, newtype] if it does transformation of type.

----
Aria Stewart

> > > On Fri, Jul 8, 2011 at 10:27, Aseem Kishore <aseem....@gmail.com (mailto:aseem....@gmail.com)>

> > > wrote:
> > > > Anyone have any thoughts or responses? =D
> > > > Aseem
> > > >

> > > > On Wed, Jul 6, 2011 at 7:41 PM, Aseem Kishore <aseem....@gmail.com (mailto:aseem....@gmail.com)>

> > > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > > To unsubscribe from this group, send email to

> > > > nodejs+un...@googlegroups.com (mailto:nodejs+un...@googlegroups.com)

> > > > For more options, visit this group at
> > > > http://groups.google.com/group/nodejs?hl=en?hl=en
> > >
> > > --
> > > Job Board: http://jobs.nodejs.org/
> > > Posting guidelines:
> > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > > You received this message because you are subscribed to the Google
> > > Groups "nodejs" group.

> > > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > > To unsubscribe from this group, send email to

> > > nodejs+un...@googlegroups.com (mailto:nodejs+un...@googlegroups.com)

> > > For more options, visit this group at
> > > http://groups.google.com/group/nodejs?hl=en?hl=en
> >
> > --
> > Job Board: http://jobs.nodejs.org/
> > Posting guidelines:
> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > You received this message because you are subscribed to the Google
> > Groups "nodejs" group.

> > To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> > To unsubscribe from this group, send email to

> > nodejs+un...@googlegroups.com (mailto:nodejs+un...@googlegroups.com)

> > For more options, visit this group at
> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.

> To post to this group, send email to nod...@googlegroups.com (mailto:nod...@googlegroups.com)

> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com (mailto:nodejs+un...@googlegroups.com)

[Aseem Kishore]

Exactly Isaac, thanks.

Aria, that's an interesting point, esp. considering that ultimately, everything needs to become JS before Node can use it.

Thinking out loud, here are the various use cases that come to mind:

- CoffeeScript wants to transform source Coffee code (which is usually found in files ending in '.coffee', but could theoretically come from another language that compiles to Coffee) to JS.

- Streamline wants to transform JS to JS. It just happens to be sync JS to async JS. The source JS could come from a file (ending in '_.js'), or it could come from another transform (like Coffee, which is what my team uses).

- A require watcher like node-dev wants to know whenever a file is required, so it can watch it and restart node on changes. So it could simply register an identity transform, but the key thing is it needs to know the source filename.

- A compilation cacher like the one I wrote wants to hook into the pipeline both at the beginning and at the end. At the beginning, to see if it can short-circuit all the transformations since the output is known and cached; at the end, to cache the output if it wasn't already. This is an interesting case, because its value comes from being able to override the entire pipeline.

- A code-coverage instrumenter like Burrito like Isaac mentioned. Haven't thought too deeply about the requirements of this one, but seems just like Coffee or Streamline.

It would be awesome if Node's API could support all these cases.

Aseem

[Shimon Doodkin]

few more ideas
https://github.com/shimondoodkin/nodejs-mongodb-app/wiki/convention-ideas

> >> On Fri, Jul 8, 2011 at 10:27, Aseem Kishore <aseem.kish...@gmail.com>

> >> wrote:
> >> > Anyone have any thoughts or responses? =D
> >> > Aseem
>

> >> > On Wed, Jul 6, 2011 at 7:41 PM, Aseem Kishore <aseem.kish...@gmail.com>

> >> > wrote:
>
> >> >> Hey there,
> >> >> I've never read any documentation on this, but from looking at
> >> >> CoffeeScript, Streamline.js and node-dev, it seems that the current
> >> >> standard
> >> >> way of extending/customizing require() is by assigning a handler
> >> >> function to
> >> >> require.extensions['.ext']:
>

> >> >>https://github.com/jashkenas/coffee-script/blob/master/src/coffee-scr...
>
> >> >>https://github.com/Sage/streamlinejs/blob/master/lib/compiler/registe...