brainstorm: preprocessor plugins and XML types

[Liam]

A recent discussion of E4X (http://groups.google.com/group/nodejs/
browse_thread/thread/713067d9d9db64e7) got me thinking...

If Node supported preprocessor plugins which run before compile, a lot
of possible functionality emerges -- XML literals, embedded SQL,
macros... Enabling such plugins should be pretty simple.

For XML, many find that XML literals are a big advantage, but folks
want different XML apis for different tasks.

A minimal (optional) XML preprocessor could simply validate and
transform XML literals into objects with a simple internal structure.
var a = <div style="stuff">things</div>
becomes
var a = new XML( { type:'el', tag:'div', attr:{style:'stuff'},
content:[ {type:'txt', text:'things'} ] } )

Any number of modules could then provide the XML type, so you can pick
your preferred api.

Thoughts?

[JeanHuguesRobert]

> If Node supported preprocessor plugins which run before compile, a lot
> of possible functionality emerges -- XML literals, embedded SQL,
> macros... Enabling such plugins should be pretty simple.

> Thoughts?

In some cases, you can leverage the fact that you have access to the
source file in process.argv[1]

If you detect that the file contains some preprocessing directives,
run the preprocessor first and then evaluate the result.

However, this works only if the source file's syntax is javascript
compliant, ie some stuff may need to be put in comments.

[Liam]

On Nov 20, 2:01 pm, JeanHuguesRobert <jeanhuguesrob...@gmail.com>
wrote:

> However, this works only if the source file's syntax is javascript
> compliant, ie some stuff may need to be put in comments.

Indeed, most such extended JS wouldn't parse. Node would have to find
the list of preprocessors in a header in the file, on the command
line, env variable, etc

[Mihai Călin Bazon]

Node already supports preprocessing modules.  AFAIK it's undocumented (and the API changed from 0.2 to 0.3).  I digged to learn about this in node.js.

Here's how you would do it in 0.3:

require.extensions[".my"] = function(module, filename) {
    var content = fs.readFileSync(filename);
    // now do whatever preprocessing you want, but return plain JS:
    content = preprocess(content);
    module._compile(content, filename);
};

require("./foo"); // if ./foo.my exists, it goes through your preprocessor now.

You are right that "enabling such plugins would be simple", but writing such plugins is quite hard. ;-)  I've spent 2-3 weekends trying to write a macro system on top of the parser I have in UglifyJS, but I gave up; the JavaScript syntax is way too complicated and it makes a (good) macro system almost impossible, or too complex anyway.

Cheers,
-Mihai

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

--
Mihai Bazon,
http://mihai.bazon.net/blog

[Floby]

I can add to this that once you've developed your preprocessor for,
say, coffeescript. you can also put this in a program, put it in your
path and then shebang properly your scripts. for example #!/usr/env/
bin coffeescript.

On Nov 21, 12:49 pm, Mihai Călin Bazon <mihai.ba...@gmail.com> wrote:
> Node already supports preprocessing modules.  AFAIK it's undocumented (and
> the API changed from 0.2 to 0.3).  I digged to learn about this in

> node.js<https://github.com/ry/node/blob/master/src/node.js>
> .

>
> Here's how you would do it in 0.3:
>
> require.extensions[".my"] = function(module, filename) {
>     var content = fs.readFileSync(filename);
>     // now do whatever preprocessing you want, but return plain JS:
>     content = preprocess(content);
>     module._compile(content, filename);
>
> };
>
> require("./foo"); // if ./foo.my exists, it goes through your preprocessor
> now.
>
> You are right that "enabling such plugins would be simple", but writing such
> plugins is quite hard. ;-)  I've spent 2-3 weekends trying to write a macro
> system on top of the parser I have in UglifyJS, but I gave up; the
> JavaScript syntax is way too complicated and it makes a (good) macro system
> almost impossible, or too complex anyway.
>
> Cheers,
> -Mihai
>
>
>
>
>
>
>
>
>
> On Sat, Nov 20, 2010 at 10:33 PM, Liam <networkimp...@gmail.com> wrote:
> > A recent discussion of E4X (http://groups.google.com/group/nodejs/

> > browse_thread/thread/713067d9d9db64e7<http://groups.google.com/group/nodejs/%0Abrowse_thread/thread/713067d...>)

> > got me thinking...
>
> > If Node supported preprocessor plugins which run before compile, a lot
> > of possible functionality emerges -- XML literals, embedded SQL,
> > macros... Enabling such plugins should be pretty simple.
>
> > For XML, many find that XML literals are a big advantage, but folks
> > want different XML apis for different tasks.
>
> > A minimal (optional) XML preprocessor could simply validate and
> > transform XML literals into objects with a simple internal structure.
> >   var a = <div style="stuff">things</div>
> > becomes
> >   var a = new XML( { type:'el', tag:'div', attr:{style:'stuff'},
> > content:[ {type:'txt', text:'things'} ] } )
>
> > Any number of modules could then provide the XML type, so you can pick
> > your preferred api.
>
> > Thoughts?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> > .

[Liam]

On Nov 21, 3:49 am, Mihai Călin Bazon <mihai.ba...@gmail.com> wrote:
> Node already supports preprocessing modules.

Oop, I simply searched the docs and groups :-)

So then, to the XML users, is there interest in a basic preprocessor
that renders XML literals into objects with a simple structure, which
various modules could then manipulate?

[jashkenas]

> I can add to this that once you've developed your preprocessor for,
> say, coffeescript.

FYI, doing a "require('coffee-script')" in your Node script enables
such pre-processing support, and from that point on, you can just
require CoffeeScript ".coffee" files, and everything works as if they
were already JavaScript. This has been the case since Tim Smart got
registerExtension included in Node, back in the Spring...

[Floby]

Yes it is true =) but I don't know if it would work that simply for
other preprocessors