[Ann] UglifyJS 2.0

[Mihai Călin Bazon]

Hi all,

I thought I'd drop this here for anyone using UglifyJS that's not
reachable through other channels: I started work on version 2 and it's
already usable, although not yet as good as v1 in terms of
compression. The goals of the rewrite are to have cleaner and more
maintainable code, a better AST structure (now using objects instead
of arrays), full location information in all nodes, add support for
generating source maps and keeping certain comments in the compressed
output.

More info at the project page: https://github.com/mishoo/UglifyJS2

Cheers,
--
Mihai Bazon,
http://mihai.bazon.net/blog

[Jimb Esser]

Good to hear!  We've been using UglifyJS and have been lamenting the lack of source maps when trying to decipher client crash reports, so I look forward to giving that a try!

  Jimb Esser

  Cloud Party

[Scott Taylor]

Very cool. What comments in the AST are you going to preserve?

Best

Scott

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Mihai Călin Bazon]

On Tue, Aug 28, 2012 at 5:33 AM, Scott Taylor <sc...@railsnewbie.com> wrote:
> Very cool. What comments in the AST are you going to preserve?

The new AST is able to store all comments, and the compressor and code
generator will be able to keep most of them. However, I suspect that
in general people will only need to store copyright notices, and those
usually start with some special marker like "/*!". It'll be easy to
add a configuration option to keep such comments, as long as they're
not in code that's going to be dropped (for example dead code, like,
code that follows a return, throw, break or continue statement).

[Scott Taylor]

Wonderful!  I've been working on a project that is sort of like parenscript - but much more of a straight javascript in lisp/scheme clothes with a define-syntax macro system.

https://github.com/smtlaissezfaire/loop

I've been hacking on the 1x source of uglify to translate javascript into a lispy type system (and back) - but inline comments have been a cause of concern.  Where is the 2.x source at this point?

Cheers,

Scott

[Mihai Călin Bazon]

Well, the code generator doesn't yet have an option to keep comments,
but I can add it easily; the harder part was having them in the AST,
and that's done.

What exactly are you trying to achieve? My understanding is that you
compile Lisp to JS (cool!), do you want to be able to do the reverse
transform? If so, perhaps a better idea is to generate a source map.
(not sure what you need to do though, just guessing)

Cheers,
-Mihai

[Scott Taylor]

Yeah, I'm compiling both ways (JavaScript to Loop) and vise versa. I'll have to look at the new 2.x source to figure out what fits best

Thanks for the heads up!

Scott

[Felipe Gasper]

I personally would love to use the new UglifyJS as a code *beautifier*,
which becomes feasible now if it’s able to keep comments intact.

-FG

[Marcel Laverdet]

Just curious why you need the comments in the AST at all? If you've got the start position & length of every token in the AST (much easier to do) you implicitly have the comments as well. The "fiber" engine in Streamline (https://github.com/Sage/streamlinejs/blob/master/lib/fibers/transform.js) does this with really good results.

[Scott Taylor]

Obviously it depends what you are trying to do.  Usually compiled programming languages strip them out in the tokenizer; for source to source translations sometimes you want to keep them in (for instance a header with copyright notice when compressing or *all* of them in a case like mine)

Best

Scott

[Marcel Laverdet]

Yeah I understand the purpose of maintaining comments; I've actually written a lot of source-to-source transformations myself. The reason I ask is that I always found it cumbersome to stick non-AST information on the AST itself. For instance, "for /* this is my loop */ (/* this is my initializer */ var ii = 0 /* there it is! */;;)". I tortured myself for a long time messing with maintaining comments and whitespace when I found that the solution is actually implicit in the token start/end positions.

If you're rendering the AST to string and you've got the source tokens attached to each node, you can just look at the original source string to find the comments and whitespace. Like I said, you should check out the fibers transformation in Streamline. It does a source-to-source transformation while maintaining *all* non-semantic source data (whitespace & comments).

The other solution I've seen is to maintain a "lastDocBlock" variable in your tokenizer state. Then when the parser parses a function, you just attach `lastDocBlock` to the node that way. That's a pretty simple and easy to implement solution and it handles concerns about maintaining copyright information, but it won't maintain whitespace or esoteric comments which is desirable in many situations.

[Mihai Călin Bazon]

Well, the good news is that I'm maintaining both start/end tokens in
all AST nodes, and inside tokens I maintain line/col/pos/endpos too.
And on top of this, there's a comments_before array in all tokens
which tell what comments were found before that token (though that's
not so useful for perfect reconstitution of whitespace).

Cheers,
-M.

[Dan Milon]

Out of topic, but you gave me the clues.
Do you know where i can read up about AST trees for javascript?

Thanks a ton,
danmilon.

> <mailto:nod...@googlegroups.com>

> > To unsubscribe from this group, send email to
> > nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>

> > For more options, visit this group at
> > http://groups.google.com/group/nodejs?hl=en?hl=en
> >
> >
> > --
> > Job Board: http://jobs.nodejs.org/
> > Posting guidelines:
> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> > You received this message because you are subscribed to the Google
> > Groups "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com

> <mailto:nod...@googlegroups.com>

> > To unsubscribe from this group, send email to
> > nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>

> > For more options, visit this group at
> > http://groups.google.com/group/nodejs?hl=en?hl=en
>
>
>
> --
> Mihai Bazon,
> http://mihai.bazon.net/blog
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com

> <mailto:nod...@googlegroups.com>

> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com

> <mailto:nodejs%2Bunsu...@googlegroups.com>

[Marcel Laverdet]

Not sure where a good place to start is. They're no different than ASTs for any other language. uglify-js dumps out a big nested array which is a pretty good start if you want to see about what is going on.

I got started in meta-programming with Bison which is a C parser and Flex which is a C scanner. (I mean, they don't parse and scan C, they are general purpose parsers which happen to be written in C). Basically you just take a source file and run it through a scanner which turns it into a bunch of "tokens" which are just one step up from raw source (parse something like "var hello = 1" into [t_VAR', 'hello', t_EQ, 1]), and then the parser takes those tokens and turns them into a meaningful tree.

    <mailto:nodejs@googlegroups.com>

    > To unsubscribe from this group, send email to

    > nodejs+unsubscribe@googlegroups.com

    <mailto:nodejs%2Bunsubscribe@googlegroups.com>

    > For more options, visit this group at
    > http://groups.google.com/group/nodejs?hl=en?hl=en
    >
    >
    > --
    > Job Board: http://jobs.nodejs.org/
    > Posting guidelines:
    > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    > You received this message because you are subscribed to the Google
    > Groups "nodejs" group.
    > To post to this group, send email to nod...@googlegroups.com

    <mailto:nodejs@googlegroups.com>

    > To unsubscribe from this group, send email to

    > nodejs+unsubscribe@googlegroups.com

    <mailto:nodejs%2Bunsubscribe@googlegroups.com>

    > For more options, visit this group at
    > http://groups.google.com/group/nodejs?hl=en?hl=en



    --
    Mihai Bazon,
    http://mihai.bazon.net/blog

    --
    Job Board: http://jobs.nodejs.org/
    Posting guidelines:
    https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
    You received this message because you are subscribed to the Google
    Groups "nodejs" group.
    To post to this group, send email to nod...@googlegroups.com

    <mailto:nodejs@googlegroups.com>

    To unsubscribe from this group, send email to

    nodejs+unsubscribe@googlegroups.com

    <mailto:nodejs%2Bunsubscribe@googlegroups.com>

    For more options, visit this group at
    http://groups.google.com/group/nodejs?hl=en?hl=en


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

[Scott Taylor]

If you are curious about uglify's parse trees, I've got a wrapper around uglify that spits out the tree for any given JavaScript in the loop project (https://github.com/smtlaissezfaire/loop/blob/master/script/uglify-syntax-tree-json)

Best

Scott