A preview of estr (Ecmascript traversals)

[Claus Reinke]

Estr (Ecmascript traversals & transformations) is a commandline
tool for working with Javascript code, using nodejs and esprima:

https://github.com/clausreinke/estr

It is work in progress but its current features already make it
useful, and I would like some real-world testing of the renaming
feature. Hence this preview for fellow developers on the esprima
list, before I announce it more widely. The error messages are
also still somewhat rough.

Currently available features (see repo for details):

- generating scope-aware tags
(this will mostly be interesting to Vim users, for whom I have
a plugin that uses these tags for scope-aware source navigation)

- variable renaming
(this is the feature I've been working on recently; it passes some
basic functional tests, but I'd like to get some real world feedback
before I declare it done; to keep testers out of trouble in this
preview,
it will write the modified source to stdout instead of replacing the
source file; you can pipe the output into another file and do a diff
to check whether you are happy with the result)

- finding variable occurrences: for a given variable occurrence,
find its scope, its declaration, and other occurrences with the
same name scope (this isn't documented yet; the functionality
needed to be implemented anyway, but I need to think about
how to present the information; the idea is that non-Vim
editors/IDEs might use this for scope-aware source navigation)

The feature of interest in this preview is renaming: it is mostly
done - the last bug fixes were related to language scoping edge
cases that no sane coder is going to use in practice, but I want
to get to a state where the tool will complain instead of silently
fail, even on such edge cases.

Javascript makes it impossible to guarantee that renaming is
a safe refactoring - we will always have to check/test that the
names themselves are not of significance to other source
processing tools (which might look for capitalization, or for
_test suffixes, and the like). I neither can nor want to check
against all naming conventions used by others.

So the question is: is the current state sufficient in practice or are
more checks needed? For which I'd like your input/bug reports.

Beyond the tool itself, readers here might find the approach
to AST traversals interesting, or the augmentation of the AST
with scope-related information.

Enjoy!-)
Claus

[Ariya Hidayat]

Hi Claus,

I looked at the code, it's quite impressive :) For another take on the
scope resolution, there is also
https://github.com/DamonOehlman/periscope and I'm sure Orion's Content
Assist is doing related stuff.

My spare time is currently rather going on the negative side, I really
like to complete my attempt on identifier highlight (see
http://code.google.com/p/esprima/issues/detail?id=98). Possiblly even
extending it to a simple refactoring example. If I still struggle for
this example, would you be willing to donate your binding scope
resolving routine for Esprima example?

Thank you!

Best regards,

--
Ariya Hidayat, http://ariya.ofilabs.com
http://twitter.com/ariyahidayat

[Claus Reinke]

Hi Ariya,

> I looked at the code, it's quite impressive :)

thanks. It has become rather more complex than I'd like for
something as fundamental as scoping.

> For another take on the scope resolution, there is also
> https://github.com/DamonOehlman/periscope

I've had a quick look. Oddly, it has BlockStatement as scope starter,
and is missing CatchClause. So it doesn't look complete yet, but
could probably be completed and still be simpler than the code I
use for renaming in scope_utils.js. Indeed, even my own code for
scope tracking in tags.js was much simpler.

But there are oddities in JS scoping, like the interaction of var
and function declarations with function scope and catch scope
(function and catch scope differ, leading to further interactions,
some of which are best dealt with by ruling out otherwise
syntactically valid code).

And renaming deals with two scoped ASTs: before and after
renaming. One has to check that the name change does not
change the scope structure - renamed variable occurrences
must not be captured by existing binders, and the renamed
binder must not capture existing variable occurrences.

To support such validity checks, I had to track rather more
information than I'd like.

> and I'm sure Orion's Content Assist is doing related stuff.

Is the author on this list? I'd like to hear more about status and plans.

> My spare time is currently rather going on the negative side, I really
> like to complete my attempt on identifier highlight (see
> http://code.google.com/p/esprima/issues/detail?id=98). Possiblly even
> extending it to a simple refactoring example. If I still struggle for
> this example, would you be willing to donate your binding scope
> resolving routine for Esprima example?

Interesting. I am planning to make estr usable as a library for JS
code, including browser code (the fs dependency is already factored
out, but I have to decide on the module format), and that would be a
good example. I want to get some practice with codemirror, too. Do
you want to import estr into esprima, or do you want to wait until I
get round to hack up such an example in estr?

If the former, just let me know what format you prefer for the
information. Currently, scope_utils.js findVar logs to stdout (estr's
undocumented 'findVar file.js var line column' command). It could
return a JS object representing the info. That should suffice for
identifier highlighting. Or you could adapt scope_utils.js rename
for the refactoring. I only ask that bug reports and fixes get
forwarded to estr;-)

Meanwhile, I guess I will have to look for testers more widely,
probably on js-tools.

Claus

[Claus Reinke]

>> My spare time is currently rather going on the negative side, I really
>> like to complete my attempt on identifier highlight (see
>> http://code.google.com/p/esprima/issues/detail?id=98). Possiblly even
>> extending it to a simple refactoring example. If I still struggle for
>> this example, would you be willing to donate your binding scope
>> resolving routine for Esprima example?
>
> Interesting. I am planning to make estr usable as a library for JS
> code, including browser code (the fs dependency is already factored
> out, but I have to decide on the module format), and that would be a
> good example. I want to get some practice with codemirror, too.

There is now a minimal demo (highlighting/renaming) online at

http://clausreinke.github.com/estr

still needs improvements, but shows the potential.

Claus

[Ariya Hidayat]

Looks good, indeed!

I've cleaned up the highlight example
(http://code.google.com/p/esprima/issues/detail?id=98), try it online
at http://esprima.org/demo/highlight.html.

Time to plugging in that scope analysis feature :)


Regards,

[Ariya Hidayat]

I've wrapped Yusuke escope (https://github.com/Constellation/escope)
in a small helper library, https://github.com/ariya/esrefactor. This
gives a proper scoping for the identifier, as well as the definition
location.

Give it a try with the new highlight demo:
http://esprima.org/demo/highlight.html.


Thanks!

Regards,



--
Ariya Hidayat, http://ariya.ofilabs.com
http://twitter.com/ariyahidayat

http://gplus.to/ariyahidayat