Shaping operators to function calls, an example/tutorial
Olov Lassus
examples/add-op-to-call/add-op-to-call.js:
"use strict"; "use restrict";
var Shaper = Shaper || require("shaper.js") || Shaper;
Shaper("add-op-to-call", function(root) {
var template = Shaper.parseExpression("$ + $");
return Shaper.traverseTree(root, {pre: function(node, ref) {
if (Shaper.match(template, node)) { // if (node.type === tkn.PLUS) {
var call = Shaper.parseExpression("add($, $)");
Shaper.replace(call, node.children[0], node.children[1]);
Shaper.cloneComments(call, node);
return ref.set(call);
}
}});
});
Here's a walk-through of the plugin, line by line:
3: When the Shaper pipeline reaches "add-op-to-call", here's the function to invoke.
4: We' create a template that matches binary plus. $ is a wildcard that matches any node.
5: We traverse (walk) the tree and provide a callback function to be called on each node, before traversing it's descendants.
6: In the callback we the node matches the template, i.e. is whether it's a plus node or not. We could have skipped the template and just checked the type directly for this simple example.
7: We parse a snippet that becomes the call stub which we want to swap in. Note that call is just another tree, just like template.
8: We replace the placeholders ($ identifiers) in the call tree with left and right plus operand. We don't alter the operands so any formatting and comments belonging to them are kept intact.
9: We clone comments belonging directly to the plus node into the call node. It's the same as running call.leadingComment = node.leadingComment, call.trailingComment = node.trailingComment.
10: Finally, we hook the call node into the tree (replacing the old plus node). We return the call node (ref.set returns its argument) so that the tree traversal will continue on it instead of the now-removed plus node.
The example leverages a lot of Shaper's functionality. It shows how to use declarative matching instead of manually inspecting properties, often recursively. It also shows how to create sub-trees declaratively, instead of manually creating a bunch of nodes and wiring them together.
Here's an example input program, examples/add-op-to-call/tests/simple.js:
1 + (/*mul*/ 2 *
/*function*/ f(/*plus*/ 3 + /*number*/ 4)) + 5;
Let's run it:
node run-shaper.js -- examples/add-op-to-call/tests/simple.js examples/add-op-to-call/add-op-to-call.js --source
It becomes:
add(add(1, (/*mul*/ 2 *
/*function*/ f(/*plus*/ add(3, /*number*/ 4)))), 5);
All binary + operators have been replaced with add calls. The formatting and all annotations/comments are preserved. We did this in 13 lines of code, generously counted.
/Olov
Michael
feature, in my opinion. I've wanted a way to match nodes in a similar
style as a regex matches characters for a long time. Can you expand a
little on how flexible the expression can be? For example, if I wanted
to match, not a fixed number of nodes, but an arbitrarily long
collection of nodes, such as this:
Convert:
var x = anyvalue,
y = anyvalue;
Into:
var x = anyvalue;
var y = anyvalue;
Where there may be any number of variables involved.
Juho Vepsäläinen
Olov Lassus
> The Shaper.parseExpression("$ + $"); method is an incredibly useful
> feature, in my opinion. I've wanted a way to match nodes in a similar
> style as a regex matches characters for a long time. Can you expand a
> little on how flexible the expression can be? For example, if I wanted
> to match, not a fixed number of nodes, but an arbitrarily long
> collection of nodes, such as this:
>
> Convert:
> var x = anyvalue,
> y = anyvalue;
>
> Into:
> var x = anyvalue;
> var y = anyvalue;
>
> Where there may be any number of variables involved.
Shaper.parseExpression just parses any valid JavaScript snippet and returns a tree. As far as parseExpression is concerned, $ is just another valid identifier. Let's check it out:
~/projects/js/jsshaper/src % d8 --shell shaper.js
V8 version 3.3.1 [console: readline]
d8> Shaper.parseExpression("$ + $").printTree()
PLUS: @ + @ < root
IDENTIFIER: $ < PLUS.children[0]
IDENTIFIER: $ < PLUS.children[1]
Shaper.match(template, node) tells you whether a template and a node matches. match considers the identifiers $ and $$ to be special (wildcards). $ matches any node and its descendants, and $$ matches any remaining nodes (of a list) and their descendants.
Shaper.match will grow further capabilities, see an earlier discussion with Jared. It will be possible to specify the wildcards, and it will also be possible to inspect the matches, turning the declarative knob to 11. It may or may not be possible to provide conditional wildcard functions, I haven't decided whether the API tax is worth it yet.
For your example try Shaper.match(Shaper.parseExpression("var $$"), Shaper.parseExpression("var x = 1, y")).
Similarly, Shaper.replace considers $ nodes placeholders and swaps in other nodes for them.
/Olov
Olov Lassus
> I threw together a brief blog post showing how to use this approach to implement operator overloading for JavaScript. You can find it here, http://nixtu.blogspot.com/2011/05/using-jsshaper-to-provide-operator.html .
That's great! Here's a minor suggestion for improving the code. Consider parsing and storing the four templates once and for all before Shaper.traverseTree is invoked, to follow best practice. Right now you're re-parsing them over and over again as many times as there are nodes in the tree. Alternatively you may want to use node.type (a number) instead of match now that you want to select between a bunch of different types. It would make the example simple, although less declarative. That's how the restricter plugin does it.
> I didn't quite grok how to use restricter yet (var a += 3; bit). Even without that it seems pretty cool.
It's not that you should use restricter, rather that you'll end up reimplementing or copying the code that's already in the restricter plugin for the same purpose. Restricter is a plugin that transforms a lot of operators (+ and += included) into funcalls, for restrict mode checking.
The operator-to-funcall transformation from a + b is easy as you've discovered: add(a, b). Doesn't matter what kind of expressions a or b are.
The += transformations are harder if we want to retain the semantics (never re-evaluate the lvalue). Specifically:
identifier += v becomes identifier = add(identifier, v)
expression.identifier += v becomes set(add, expression, "identifier", v)
expression1[expression2] += v becomes set(add, expression1, String(expression2), v)
Where set is implemented as
function set(fn, base, name, v) {
return base[name] = fn(base[name], v);
}
It doesn't make sense to force all plugins to duplicate this code whenever they want to do an operator-to-funcall transformation. Tells me this code should be factored out of restricter into its own plugin. Once that is in place your op-overloading.js example should become even smaller, and merely describe which operators you want to have transformed to which funcalls.
/Olov
Juho Vepsäläinen
maanantaina 16. toukokuuta 2011 23.25.13 UTC+3 Olov Lassus kirjoitti:
16 maj 2011 kl. 13.40 skrev Juho Vepsäläinen:> I threw together a brief blog post showing how to use this approach to implement operator overloading for JavaScript. You can find it here, http://nixtu.blogspot.com/2011/05/using-jsshaper-to-provide-operator.html .
That's great! Here's a minor suggestion for improving the code. Consider parsing and storing the four templates once and for all before Shaper.traverseTree is invoked, to follow best practice. Right now you're re-parsing them over and over again as many times as there are nodes in the tree. Alternatively you may want to use node.type (a number) instead of match now that you want to select between a bunch of different types. It would make the example simple, although less declarative. That's how the restricter plugin does it.
> I didn't quite grok how to use restricter yet (var a += 3; bit). Even without that it seems pretty cool.
It's not that you should use restricter, rather that you'll end up reimplementing or copying the code that's already in the restricter plugin for the same purpose. Restricter is a plugin that transforms a lot of operators (+ and += included) into funcalls, for restrict mode checking.
The operator-to-funcall transformation from a + b is easy as you've discovered: add(a, b). Doesn't matter what kind of expressions a or b are.
The += transformations are harder if we want to retain the semantics (never re-evaluate the lvalue). Specifically:
identifier += v becomes identifier = add(identifier, v)
expression.identifier += v becomes set(add, expression, "identifier", v)
expression1[expression2] += v becomes set(add, expression1, String(expression2), v)Where set is implemented as
function set(fn, base, name, v) {
return base[name] = fn(base[name], v);
}It doesn't make sense to force all plugins to duplicate this code whenever they want to do an operator-to-funcall transformation. Tells me this code should be factored out of restricter into its own plugin. Once that is in place your op-overloading.js example should become even smaller, and merely describe which operators you want to have transformed to which funcalls.
Olov Lassus
> It could be interesting to implement caching behavior on API level. This way actual plugins could remain relatively simple. At least there could be some standard mechanisms to make this kind of work easier.
I've considered that - it's about getting the API right. Applying internal caching to Shaper.parseExpression doesn't yield great benefits because it would still need to clone the cached tree at each invocation, because the caller may want to mutate it.
It would be possible to add caching to Shaper.match by allowing it to take the template in string form. The API becomes more complex and with caches come cache invalidation, but the increased benefit may be worth it. I'll let that sink in a while.
> It's definitely sensible to move this transformation into a plugin of its own. Do you want to do that (I'm the noob here :) )? It's not like I'm in hurry or anything. It just would be cool to know if the idea made it onto your TODO list. :)
It did, watch out for the optocaller plugin.
> In the meantime I probably have to try out some other transformations and see how that goes...
Please do and keep us posted. Growing an API and framework into something beautiful and meaningful requires real-world usage and experimenting. We're still at the stage where breaking the API is allowed. That's a luxury too good to neglect.
/Olov